BlackMagic Integration

[matttidridge]

Is there a best practice or suggestion for integrating BlackMagic input into MediaPipe4U?

We currently have other parts of the system utilizing Render Targets. Is it possible to also send these Render Target Texture frames to MediaPipe4U or should we use a different Image Source?

Thank you! Great plugin.

[matttidridge]

The UE BlackMagic plugin produces a MediaBundle with a MediaPlayer & RenderTarget. I can use Blueprints to downscale the RenderTarget, flip it, crop it, etc. Can I feed that RenderTarget into MediaPipe4U?

https://docs.unrealengine.com/4.26/en-US/WorkingWithMedia/ProVideoIO/BlackmagicQuickStart/

[endink]

Theoretically this is possible, you need to implement an Image Resource yourself（Inherit from interface）, you must use C++, because some API blueprints are not accessible, refer to the .h file of GstreamerImageSource, but I haven't done much help code for this requirement, maybe it will be hard , the key thing is that you can't copy images in memory, which will introduce latency.

[endink]

MediaPlayer can be used as an image source, I don't know if that will help you.

Refer：

https://opensource-labijie-com.translate.goog/Mediapipe4u-plugin/mocap/features/image_source.html?_x_tr_sl=zh-CN&_x_tr_tl=en&_x_tr_hl=zh-CN&_x_tr_pto=wapp#mediaplayerimagesourcecomponent

[matttidridge]

Yes I did see that. I just had concerns because I need the HD footage to display on screen, but the MediaPlayer Image Source does not support horizontal inversion & resolution limits.

GStreamer seems to support BlackMagic decklink. Think that would be difficult to implement?

https://gstreamer.freedesktop.org/documentation/decklink/decklinkvideosrc.html?gi-language=c

[endink]

If gstreamer supports it, then this would be easy to implement, but it need gstramer custom command feature, I can support gstreamer pipeline command yourself in the next version.

The only thing you need to know is how to write the gs pipiline command, like this:

decklinkvideosrc device-number=0 connection=sdi mode=1080p25 ！ autovideosink

[endink]

The M4U static image source is very special, it is half-synchronous processing (poll mix observer) internally, and the other sources are observer for all, which will bring a lot of latency, and close some "stream" feature of mediapipe. So dont use static image source as stream.

The latest M4U can work at 25 fps (AMD 3600, 720p video), 12 fps is too slow.

[matttidridge]

Understood. As I've been working I came across two use cases for 'gstreamer pipeline command'. One for the BlackMagic and one for Image Sequence (I tried using Media Player but it results in a consistent crash).

https://gstreamer.freedesktop.org/documentation/multifile/imagesequencesrc.html?gi-language=c

I believe my colleague has been talking to you about licensing for our projects. Is it possible this feature could be added some time this week so we can make our deadline?

Thank you!

[endink]

It's not clear to me that MediaPlayer "crash" you mean the M4U causing it to crash? If so, is it possible for you to give me a simple reproduction step?

Media issues get tricky in UnrealEngine because Epic doesn't support access D3D textures, which are in the GPU and the only way in UE is copy them to the CPU, it is will bring a lot of performance cost, which is why I don't recommend using mediaplayer, GStreamer is the most convenient solution for working with media at this time, it has a good pipeline design that supports accessing textures directly from the GPU or CPU instead of copying them.

BTW, are you using C++? Blueprint may not be able to complete this job.

[matttidridge]

I am using a combination of C++ & Blueprints.

I think GStreamer will solve my problem if I can use it to play both Image Sequences and DeckLink input. Can I use the "Start GStreamer Launch" node to do that?

Here are some screenshots of the crash when using Media Player:

[endink]

Sorry, I can‘t promise the release time of the next version, because there are still many bugs to be fixed, there is a lot of code in plugin, it usually takes several hours to fully compile. So, frequent release versions become very difficult.

Fortunately, I just found out the latest version already supports custom GStreamer pipeline command, using the example of opening a file:

XXX ! videoconvert ! video/x-raw,format=(string)RGBA {transform} ! appsink name=mediapipe4u_sink

You can found the full command string in log.

And your can call GStreamerImageSourceComponent::StartGStreamerLaunch in blueprint, It accepts a pipline command string as a parameter.

Two things：

1. appsink name=mediapipe4u_sink : This is the m4u gstreamer sink, it is required.
2. videoscale2 width={width} height={height}: This is an M4U implemented image scaler that you can use to scale frames.

[endink]

This may require you to be familiar with GStreamer usage, but I don't think it's that difficult, good luck ! 😄 😃

[endink]

Forgot, the final frame conversion in RGBA format is also required.

video/x-raw,format=(string)RGBA

[matttidridge]

That's great, thank you! I will try out GStreamerImageSourceComponent::StartGStreamerLaunch tomorrow and see how it goes 🚀

Is it possible to run that Async as well so the game does not freeze while it is trying to start?

Sorry for so many requests 😅

[endink]

use StartImageSourceAsync (instead StartImageSource), gstreamer is async, GStreamerImageSourceComponent is run at asyn mode, so StartGStreamerLaunch will not block your game thread, dont worry abot that.

[endink]

Lastly, when you use the StartGStreamerLaunch function, it means that you get the steering wheel of media processing, you need to control frame process yourself, like frame scale, flip etc. And the image source ( resolution limitation, flip ) configuration on the HolisticComponent will be ignore. All GStreamer features are supported.

[matttidridge]

Ah OK so I call StartGStreamerLaunch first, then StartImageSourceAsync to run it async.

I'll try my best to learn some of the GStreamer commands. Do you happen to have one handy for what happens when you run Start Camera with the mirror (flip) and resolution limitation? That would really help get me started.

Afterwards I'm happy to post the commands I come up with for Image Sequences & BlackMagic so you can incorporate them as features if you like.

[endink]

StartGStreamerLaunch and StartImageSource do not have strict order requirements, StartGStreamerLaunch means to start GS and another means to start mediapipe, but it is generally recommended to start mediapipe (Start Image Source) first.

[endink]

Do you happen to have one handy for what happens when you run Start Camera with the mirror (flip) and resolution limitation?

I dont know your means clearly, flip (for fix frame if camera graph is not correct) and resolution limit (for improve mediapipe fps,the smaller the image, the faster the mediapipe ) can be configured on ImageSource propperty of MediapipeHolisticComponent (you can see details panel in UE editor).

[matttidridge]

I was just wondering if you had an example gstreamer command for opening the webcam with a flip and set resolution since that node does not output a log.

[endink]

Run demo project and start video , you will see the log in UE editor

[matttidridge]

@endink GStreamerImageSourceComponent::StartGStreamerLaunch worked great for my purposes! 😸 👍 💯

One issue I am noticing however is when I use MediaPipeHolisticComponent::OnVideoTextureCreated, the frames I am recieving in Unreal are more washed out than what I get when I run the same command in GStreamer from command line.

Unreal vs GStreamer->AutoVideoSink

I am taking the texture from MediaPipe and drawing it to a Render Target (my application displays the users camera feed as part of the game).

[endink]

I'm not an artist, I don't know much about materials and shaders, I believe this can be fix in Material Editor, and maybe you should ask a technical artist for help.

Sorry, I don't have enough ability to solve the problems about the shader.

[matttidridge]

No problem. As a temporary fix I can write a shader to adjust the gamma/contrast, but I just wanted to see if it was coming to me from MediaPipe that way or if it was a shader issue. My materials are set to UI (unlit) and looks the same in flat-shaded mode so I don't think it's the Material.

[endink]

On the HolisticComponent you can change your display mode property, when using "Raw", it will come directly from ImageSource, when set to "Annotation", it will go through mediapipe, mediapipe processing of video is very simple, using openCV to overlay layers on each frame, you can watch the original video (from GS) with "Raw".

[matttidridge]

OK I understand. I am using RAW so as long as I am getting the raw frames from GStreamer the issue must be in Unreal. 👍

Is there any easy way to split the stream into two?

I would be nice to use 1080p video for the texture and 720p video for MediaPipe

[endink]

It's not easy, two way:

1. Wrap the GstreamerImageSource as a new image source and splite it , but, its hard, you need to clearly known the details about your ImageaSource.

2. Use an custom mediapipe graph, scale image in mediapipe graph. You have to be very familiar with google mediapipe.

[matttidridge]

I fixed this Color Space issue as well. You need to run the texture returned from MediaPipe through Apply Color Space Transform to set the source and output to sRGB. Now the colors look the same as they do from the camera!

[endink]

Cool , maybe I will add this feature to next version, let we can easy config color space.

[endink]

It's a RHI issue?

https://forums.unrealengine.com/t/utexture2d-createtransient-and-srgb/580979

[matttidridge]

Possibly. I don't see an easy way to change the Texture Sampler in the material between RAW & sRGB.

The color space issue can be seen in the example project as well with your default setup (using an MP4).