transcoding with rpi produces missing h264 headers

[arkanoid87]

Describe what you are trying to accomplish and why in non technical terms
transcode from mjpeg to h264 using hardware acceleration (v4l2m2m) on rpi4 via go2rtc

Describe the solution you'd like
generated h264 stream is created but cannot be decoded due to missing h264 headers

[h264 @ 0x756c8c0061c0] decode_slice_header error
[h264 @ 0x756c8c0061c0] no frame!
[h264 @ 0x756c8c0061c0] non-existing PPS 0 referenced    0B f=0/0 

encoder params are not properly set in ffmpeg v4l2m2m https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/v4l2_m2m_enc.c

v4l2-ctl -L -d /dev/video11 

Codec Controls

                 video_b_frames 0x009909ca (int)    : min=0 max=0 step=1 default=0 value=0 flags=update
                 video_gop_size 0x009909cb (int)    : min=0 max=2147483647 step=1 default=60 value=60
             video_bitrate_mode 0x009909ce (menu)   : min=0 max=1 default=0 value=0 (Variable Bitrate) flags=update
                                0: Variable Bitrate
                                1: Constant Bitrate
                  video_bitrate 0x009909cf (int)    : min=25000 max=25000000 step=25000 default=10000000 value=10000000
           sequence_header_mode 0x009909d8 (menu)   : min=0 max=1 default=1 value=1 (Joined With 1st Frame)
                                0: Separate Buffer
                                1: Joined With 1st Frame
         repeat_sequence_header 0x009909e2 (bool)   : default=0 value=0
                force_key_frame 0x009909e5 (button) : value=0 flags=write-only, execute-on-write
          h264_minimum_qp_value 0x00990a61 (int)    : min=0 max=51 step=1 default=20 value=20
          h264_maximum_qp_value 0x00990a62 (int)    : min=0 max=51 step=1 default=51 value=51
            h264_i_frame_period 0x00990a66 (int)    : min=0 max=2147483647 step=1 default=60 value=60
                     h264_level 0x00990a67 (menu)   : min=0 max=15 default=11 value=11 (4)
                                0: 1
                                1: 1b
                                2: 1.1
                                3: 1.2
                                4: 1.3
                                5: 2
                                6: 2.1
                                7: 2.2
                                8: 3
                                9: 3.1
                                10: 3.2
                                11: 4
                                12: 4.1
                                13: 4.2
                                14: 5
                                15: 5.1
                   h264_profile 0x00990a6b (menu)   : min=0 max=4 default=4 value=4 (High)
                                0: Baseline
                                1: Constrained Baseline
                                2: Main
                                4: High

Describe alternatives you've considered
patching ffmpeg
https://lists.ffmpeg.org/pipermail/ffmpeg-user/2020-December/051067.html

[NickM-27]

Which build of frigate are you using?

[arkanoid87]

docker image: ghcr.io/blakeblackshear/frigate:stable

cat version.py
VERSION = "0.13.2-6476f8a"

[NickM-27]

this build uses a specific build targeted at the rpi that ensures hwaccel works, the standard ffmpeg build has broken hwacel for the rpi numerous times

#2030
#3780

I don't foresee the build changing as that has led to a lot of work on our side in the past and this is working well for decoding. Frigate supports using a custom build of ffmpeg so that's likely the route that you'll want to take. https://docs.frigate.video/configuration/advanced#custom-ffmpeg-build

[arkanoid87]

thanks for the answer

I see another possible route: exec gstreamer pipeline instead of ffmpeg
gstreamer let you set custom v4l2 device settings on the encoder device, avoiding copy just as m2m. It should be possible to publish to go2rtc rtsp server

[marcusb]

@arkanoid87 I made a patch to ffmpeg similar to the one you linked, this fixes the missing headers. Confirmed working on my Raspberry Pi 4B.