Browser vs Pion and Ffmpeg stream

[silverark]

Not usre where exactly yo ask this, but I'm looking for some advise on processing a stream. Firstly, thank you for Pion, it's awesome!

We have it in a project that connects in to a call and monitors the users by doing some facial recognition and adult content detection. Everything is working really well, apart from when a user has a bad wifi connection.

We started building from the janus-gateway example and to get the image so we can process it in gocv we use the example in gocv-receive.

This uses ffmpeg with an input and output pipe. An ivfWriter sends it to the input, and we read it from the 'output' as per the example. We then open it as an opencv mat.

The problem:
When viewing the stream in a browser, when it get's choppy due to a bad wifi signal the browser drops the frames rather than showing garbled data. With pion -> ffmpeg -> gocv we get garbled frames as shown in the image below.

I've tried various ffmpeg commands to see if we can detect these bad fames/packets: "ffmpeg", "-err_detect", "buffer", "-fflags", "discardcorrupt" but whatever I try, when we simulate a bad connection it destroys the image.

Is there a way of detecting when the image is bad? Is there a way of stopping or dropping these frames? I'm not 100% sure where the issue is so even pointing me in the right direction would be greatly appreciated!

What did you expect?

When the signal is bad, for bad frames to be dropped.

What happened?

I'm seeing garbled frames when the connection is bad, or just before it disconnects.

[iammeizu]

met the same problem, any suggestion?

[iammeizu]

hi, @silverark I solved this problem. ivfwriter is not reliable, I recommand samplebuilder instead of ivfwriter because samplebuilder will sort rtp packet by sequence number. Check the expamle named "Save-to-webm", and use ffmpeg pipe as videowriter. Later, I will commit a new example, Good Luck!

[silverark]

Thank you @iammeizu

I will give it a go tomorrow! I was also contacted by Sean DuBois toio who said he was workign on Interceptors for V3 too: pion/webrtc-v3-design#34

[silverark]

@iammeizu - I've got this working with sampleBuilder but strangely found the image to be worse using this method! What value did you have for the maxLate? When testing with a simulted bad network, values of 10 - 20 almost didn't work at all and produced the image below. A value of 60 worked quite well but turned to snow sometimes. Values over 200 introduce quite a delay.

I'm wondering if it's my ffmpeg command?
"ffmpeg", "-i", "pipe:0", "-s", strconv.Itoa(width)+"x"+strconv.Itoa(height), "-pix_fmt", colourFormat, "-f", "rawvideo", "pipe:1"

I tried using the example from the twilio example, but outputting to pipe1 instead, but I found that the stream never started!

[iammeizu]

hi @silverark , my Maxlate is 50, and you can use send PLI by rtcp, which is used to control keyframe internal. I set 1 PLI per second, which means server get a complete frame per second. so the snow phenomenon can be relief by this way.

[Sean-Der]

Sorry for the slow response on this! @silverark emailed me directly, copying my respond here

So the issue is that Pion doesn't provide error concealment by default. We are fixing that in V3 so you get it by default.

For now what I would suggest is sending a PLI every couple seconds. You can see an example of that here https://github.com/pion/webrtc/blob/master/examples/save-to-disk/main.go#L95-L104

This requests that the encoder sends a new frame, so even if you have packet loss it will recover much faster. Thanks for sharing all your comments/feedback on the GitHub issue so far.

I am also only Slack https://pion.ly/slack and happy to answer questions there as well. I hope you keep using Pion, and will try my best to make this all work :) 

[marlocarlo]

I am having the exact same issue with rtp-forwarder. But its happening always, not only when the signal is bad. To get this ready for production, i need to somehow solve this.

I would appreciate any guidance

Thanks