Update FFmpeg presets for Rockchip boards

[MarcA711]

see #979

• This is based on this ffmpeg fork from nyanmisaka. If you use other ffmpeg forks with rockchip hardware support, these presets will likely not work.
• I provide up to date static builds. You can just download the binary and execute is using ./ffmpeg .... No need to install any libraries or dependencies. You need to run linux with up to date kernel drivers though. I recommend ubuntu-rockchip if your board is supported.
• supports transcoding and rescaling
• I added :force_original_aspect_ratio=0 to the scale command to allow rescaling. No other preset does this afaik, but I know that some frigate users want to change their aspect ratio
• Rockchip hardware supports lots of other filters. I hope to add them as well if I have some time.

[MarcA711]

@AlexxIT I know you are not able to test these changes. Is there some way we can still implement them?

[AlexxIT]

I'm busy for the next two weeks

[bugui2007]

Hello all,
@MarcA711, Im using a Orange Pi 5 Plus, and I set the following configuration over go2rtc-ffmpeg config section.

go2rtc:
  ffmpeg:
    input: -fflags nobuffer -flags low_delay -timeout 5000000 -i {input}
    h264/rk: -vf scale_rkrga=w={XXX}:h={XXX}:force_original_aspect_ratio=0 -c:v h264_rkmpp -g 50 -bf 0 -profile:v high -level:v 4.1 -an -vf "transpose=1,transpose=1" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}`
    h265/rk: -vf scale_rkrga=w={XXX}:h={XXX}:force_original_aspect_ratio=0 -c:v hevc_rkmpp -g 50 -bf 0 -profile:v main -level:v 5.1 -an -vf "transpose=1,transpose=1" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}`
    h264/rkmpp: -c:v h264_rkmpp -g 50 -bf 0 -profile:v high -level:v 4.1
    h265/rkmpp: -c:v hevc_rkmpp -g 50 -bf 0 -profile:v main -level:v 5.1

Even with this I didnt note any inprovement on decoding or encoding on hw for go2rtc...but frigate is using rkrga without problems.

Every 1.0s: cat /sys/kernel/debug/rkrga/load                                                                                                                                                                 osiris: Mon Jul 29 18:50:58 2024

num of scheduler = 3
================= load ==================
scheduler[0]: rga3
         load = 0%
-----------------------------------
scheduler[1]: rga3
         load = 0%
-----------------------------------
scheduler[2]: rga2
         load = 2%
-----------------------------------
         process 8: pid = 36993, name: ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel rkmpp -hwaccel_output_format drm_prime -user_agent FFmpeg Frigate/0.14.0-a4eb435 -rtsp_transport tcp -timeout 5000000 -i rtsp://127.0.0.1:85
54/FrenteIzquierda_sub -r 5 -vf scale_rkrga=w=640:h=480:format=yuv420p:force_original_aspect_ratio=0,hwmap=mode=read,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:
         process 9: pid = 37058, name: ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel rkmpp -hwaccel_output_format drm_prime -user_agent FFmpeg Frigate/0.14.0-a4eb435 -rtsp_transport tcp -timeout 5000000 -i rtsp://127.0.0.1:85
54/Cocina_sub -r 5 -vf scale_rkrga=w=640:h=480:format=yuv420p:force_original_aspect_ratio=0,hwmap=mode=read,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:
         process 10: pid = 37048, name: ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel rkmpp -hwaccel_output_format drm_prime -user_agent FFmpeg Frigate/0.14.0-a4eb435 -rtsp_transport tcp -timeout 5000000 -i rtsp://127.0.0.1:8
554/Altillo_sub -r 5 -vf scale_rkrga=w=640:h=480:format=yuv420p:force_original_aspect_ratio=0,hwmap=mode=read,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:
         process 12: pid = 36999, name: ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel rkmpp -hwaccel_output_format drm_prime -user_agent FFmpeg Frigate/0.14.0-a4eb435 -rtsp_transport tcp -timeout 5000000 -i rtsp://127.0.0.1:8
554/Frente_sub -r 5 -vf scale_rkrga=w=640:h=480:format=yuv420p:force_original_aspect_ratio=0,hwmap=mode=read,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:
         process 13: pid = 37035, name: ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel rkmpp -hwaccel_output_format drm_prime -user_agent FFmpeg Frigate/0.14.0-a4eb435 -rtsp_transport tcp -timeout 5000000 -i rtsp://127.0.0.1:8
554/Living_sub -r 5 -vf scale_rkrga=w=640:h=480:format=yuv420p:force_original_aspect_ratio=0,hwmap=mode=read,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:
         process 14: pid = 37010, name: ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel rkmpp -hwaccel_output_format drm_prime -user_agent FFmpeg Frigate/0.14.0-a4eb435 -rtsp_transport tcp -timeout 5000000 -i rtsp://127.0.0.1:8
554/FrenteDerecha_sub -r 5 -vf scale_rkrga=w=640:h=480:format=yuv420p:force_original_aspect_ratio=0,hwmap=mode=read,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:
         process 15: pid = 344434, name: ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel rkmpp -hwaccel_output_format drm_prime -user_agent FFmpeg Frigate/0.14.0-a4eb435 -rtsp_transport tcp -timeout 5000000 -i rtsp://127.0.0.1:
8554/Patio_sub -r 5 -vf scale_rkrga=w=640:h=480:format=yuv420p:force_original_aspect_ratio=0,hwmap=mode=read,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:

Is there any way to force Go2RTC to use HW for decoding and encoding ? How can I debug it?

Another comment:
On the commit #1154 file internal/ffmpeg/hardware/hardware_unix.go @skrashevichskrashevich is using "h264_rkmpp_encoder" and "hevc_rkmpp_encoder" instead "h264_rkmpp" and "hevc_rkmpp", that you propose in your commit in this issue.

Best regards,
Thanks
Hernan

[MarcA711]

Hey @bugui2007,

do you have a section like:

go2rtc:
  streams:

You are just defining some presets. Do you use them when defining your streams? Do you see any video from go2rtc?

If you do this correctly, you force go2rtc to use hw accel. It won't fallback to software. Either it works and uses hardware or it will crash. You can use ps -ef | grep ffmpeg. This will show all ffmpeg processes and the options. For example this process uses hw accel (can be seen from the bold parts):

ffmpeg -hide_banner -allowed_media_types video+audio -fflags nobuffer -flags low_delay -timeout 5000000 -rtsp_flags prefer_tcp -hwaccel rkmpp -hwaccel_output_format drm_prime -afbc rga -i rtsp://xxx -c:v h264_rkmpp -g 50 -bf 0 -an -rtsp_transport tcp -f rtsp rtsp://xxx

h264_rkmpp_encoder and hevc_rkmpp_encoder where old optione from this ffmpeg fork: https://github.com/hbiyik/FFmpeg. This newer fork: https://github.com/nyanmisaka/ffmpeg-rockchip uses h264_rkmpp and hevc_rkmpp. The newer fork is the one that frigate uses.

Btw. you can use the exec command to start ffmpeg with custom commands. You don't need to mess with the presets:

go2rtc:
  streams:
    MyCam: exec:ffmpeg -hide_banner -allowed_media_types video+audio -fflags
      nobuffer -flags low_delay -timeout 5000000 -rtsp_flags prefer_tcp -hwaccel rkmpp
      -hwaccel_output_format drm_prime -afbc rga -i rtsp://xxx
      -c:v h264_rkmpp -g 50 -bf 0 -an -rtsp_transport tcp -f rtsp {{output}}

[bugui2007]

Hello Marca711,
I was using this exec command on each camera (dahua and imou ip cameras) but I cant get audio from this.

go2rtc:
  streams:
     Frente:
      - exec:ffmpeg -hide_banner -allowed_media_types video+audio -fflags nobuffer
        -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp
        -hwaccel rkmpp -hwaccel_output_format drm_prime -afbc rga -i  -c:v hevc_rkmpp
        rtsp://admin:[email protected]/cam/realmonitor?channel=1&subtype=0&unicast=true&proto=Onvif
        -c:v h264_rkmpp -g 50 -bf 0 -an -rtsp_transport tcp -f rtsp {{output}}

Then I switched back to the standar rtsp command like

 - rtsp://admin:[email protected]/cam/realmonitor?channel=1&subtype=0&unicast=true&proto=Onvif

And audio is working now.

Do you have some recommendations to inprove the exec:ffmpeg command in order to get h265 to h264 transcoding on the fly as well can use audio?
Thanks in advace,

Bugui

[MarcA711]

Hey @bugui2007

Try to remove the -an parameter. It removes audio.

You can use -c:a copy copy the audio from the camera or you can transcode the audio to another format if you need to.

[derzahla]

Thanks for your work on this @MarcA711. Currently I only use go2rtc to rotate a camera stream 90 degrees. It was previous working on the latest frigate -rk dev container using:

go2rtc:
  streams:
    rtsp_doorbell_90_ch0:
      - ffmpeg:rtsp://xxxxx:[email protected]:554/ch0#video=h264#video_size=1536x2048#rotate=90#hardware

But that wasnt actually using hardware acceleration from what I gather was the point of your PR? I had previously thought it was (any good way to test?)

But anyway, the rotated stream ends up with a diagonal line and green stripe on the bottom artifacts like below and I was hoping maybe that would be fixed by using the custom ffmpeg cmdline:

I finally got it working using an amalgam of your suggested commands and those at https://github.com/nyanmisaka/ffmpeg-rockchip/wiki/Filter, resulting in:

      - exec:ffmpeg -hide_banner -allowed_media_types video+audio -init_hw_device rkmpp=hw -filter_hw_device hw -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp -i rtsp://xxxxxxx:[email protected]:554/ch0 -vf hwupload,vpp_rkrga=w=2048:h=1536:transpose=clock -c:v h264_rkmpp -c:a copy -f rtsp {{output}}

But the result has the same artifacts in the stream unfortunately.

Can you confirm that this should enable hardware acceleration the stream tranposition?

Also, I would prefer to keep the rotated stream h265 like the source stream is but I cannot get frigate to connect to it as a camera unless i convert to h264 - maybe that's a limitation of 'preset-rtsp-restream'?

Let me know if I am doing something wrong and thanks again for you and everyone elses work on improving rockchip support

[MarcA711]

AFAIK, there is no version of frigate that supports hw acceleration using #hardware in go2rtc's ffmpeg command. There was hardware support for rockchip boards in go2rtc, but this was for an old FFmpeg version. Frigate uses a newer FFmpeg version with rockchip support that is not compatible to the way hw accerlation is done in go2rtc. The point of this PR is to update hw accel to support the newest version of FFmpeg for rockchip.

any good way to test?

You can use ps -ef | grep ffmpeg. There you can see what options go2rtc passed to ffmpeg. If it contains some rockchip specific options like -c:v h264_rkmpp, it will use hw acceleration.

But the result has the same artifacts in the stream unfortunately.

There were multiple issues with artifacts in hw accelerated streams. This was mostly due to odd weird resolution. You apperently have 1536x2048, can you try if it works for a standard resolution like 1920x1080? Maybe save a clip from your camera, use software ffmpeg to crop it to 1920x1080 and try if the artifacts still appear.

Can you confirm that this should enable hardware acceleration the stream tranposition?

yes, the command that you provided uses hw acceleration.

Two things to consider:

1. Are you sure your camera can't provide a rotated stream? Most cameras have some settings to rotate the stream correctly.
2. I am not sure if the rotation is very resource intensive. If you rotate it in software, how high is your CPU usage? If it is low, maybe don't use hw accel for this.

Also, I would prefer to keep the rotated stream h265 like the source stream is but I cannot get frigate to connect to it as a camera unless i convert to h264 - maybe that's a limitation of 'preset-rtsp-restream'?

Frigate should be able to process the h265 stream. Did you select the correct hardware acceleration preset (hwaccel_args: preset-rk-h264 or preset-rk-h265)? Are you sure frigate fails to connect to the h265 stream? Because most web browsers don't support h265 since it is not free. If frigate shows no error but your web browser shows some error saying that it cannot play the video, your web browser might not be able to play h265.

[nyanmisaka]

@derzahla @MarcA711 This artifact has been fixed recently. Updating ffmpeg-rockchip should fix the problem.

- exec:ffmpeg -hide_banner -allowed_media_types video+audio -init_hw_device rkmpp=hw -filter_hw_device hw -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp -i rtsp://xxxxxxx:[email protected]:554/ch0 -vf hwupload,vpp_rkrga=w=2048:h=1536:transpose=clock -c:v h264_rkmpp -c:a copy -f rtsp {{output}}

Also, hardware decoding is not used. It is SwDec -> Hw{Scale,Transpose} -> HwEnc.

[MarcA711]

@nyanmisaka Thank you for this info.

@derzahla I try to build a new FFmpeg Rockchip binary with the fix. I will write here, when I am done.

[MarcA711]

@derzahla I uploaded a new version that should contain the fix. You can download it here: https://github.com/MarcA711/Rockchip-FFmpeg-Builds/releases/tag/6.1-6
I haven't tested it yet. It would be great if you can confirm that it is working for you.

[derzahla]

Thanks for the detailed response @MarcA711 and @nyanmisaka for the good news about the fix!

Unfortunately my opi5+ died(or maybe just the SSD) and I am failed over to an x86 frigate VM with coral passed through for now. I havent has a chance to dig into the orangepi yet but I was having stability issues(OS lockups and occasional storage errors where my root fs would set itself RO) running the latest frigate on the rk3588 ubuntu build so that is why I was considering moving away from it until it 'stabilized' but now I suspect maybe it was just due to my hardware.

@nyanmisaka So youre saying my config and frigate version was at least using HW transpose? But Marc's preset changes should allow HW decode now?

Two things to consider:

1. Are you sure your camera can't provide a rotated stream? Most cameras have some settings to rotate the stream correctly.

Yes. I use the open source thingino firmware primarily and though the hardware supports rotation the streamer that it uses isnt working. I have an open bug report

2. I am not sure if the rotation is very resource intensive. If you rotate it in software, how high is your CPU usage? If it is low, maybe don't use hw accel for this.

are you sure rotating is even possible without decode/re-encoding? I was trying to find a definitive source on that, but since my cam is outputing h265 and frigate wont connect to the rotated go2rtc stream unless I convert it to h264 I am still unsure.

Frigate should be able to process the h265 stream. Did you select the correct hardware acceleration preset (hwaccel_args: preset-rk-h264 or preset-rk-h265)? Are you sure frigate fails to connect to the h265 stream? Because most web browsers don't support h265 since it is not free. If frigate shows no error but your web browser shows some error saying that it cannot play the video, your web browser might not be able to play h265.

I have preset-rk-h265 set but I am also confused about how that works because i connect frigate directly to multiple cams, some outputting h265 and some only capable of h264. That all works. Its only on the doorbell cam that I need to rotate the stream so that is the only one i am using go2rtc for. My observations were not based on any web browser though but the frigate error log itself which would complain about 'garbage data' or something if i tried to keep just rotate the stream and not also convert to h264.

[derzahla]

@MarcA711 @nyanmisaka

I got my opi5+ back online running frigate. I also setup go2rtc streams for all my camera streams, not just the one I need rotated, since that is supposed to be best practice and offer better quality with hass integration. Unfortunately(running the latest dev frigate -rk containers and using the full "-exec" ffmpeg options on go2rtc as posted above I am at 100% cpu and nothing works. When I go back to the simple config I hover around 6% CPU and streams "mostly" work (h264 outputting cameras work fine, h265 outputting cameras can be spotty and timeout frequently). I did try copying the latest ffmpeg binaries you linked to, to the frigate container and "stop/starting it" but I had the same issue with 100% cpu usage. I know none of this is particularly helpful, but I am guessing my config has multiple problems due to my own misunderstandings and the bleeding-edge status of frigate-rk. I am particularly unclear on where or if I should even be setting hwaccel_args presets since the docs seem to suggest setting preset-rk-h264 or preset-rk-h265 at a global level, yet I have cameras that output both. I will post my current config below in case you have the time to do a once-over and notice anything that warrants fixing.

curious, do you have more up-to-date containers for rockchip fixes than the standard frigate -rk dev builds?

config:

cat scrubbed-config.yaml 
mqtt:
  enabled: true
  host: mqtt.local
  user: frigate
  password: frfsdrgtrgsdrgf

ffmpeg:
  hwaccel_args: preset-rk-h265

detectors:
  rknn:
    type: rknn
    num_cores: 3

model: 
  path: deci-fp16-yolonas_s
  width: 320
  height: 320
  input_pixel_format: bgr
  input_tensor: nhwc
  labelmap_path: /labelmap/coco-80.txt

record:
  enabled: True
  retain:
    days: 3
    mode: motion

go2rtc:
  streams:
    wvdb1:
      - ffmpeg:rtsp://thingino:[email protected]:554/ch0#rotate=90#video=h264#hardware
    wvdb1_sub:
      - ffmpeg:rtsp://thingino:[email protected]:554/ch1#rotate=90#video=h264#hardware
    litokam1:
      - rtsp://thingino:[email protected]:554/ch0
      - "ffmpeg:litokam1#video=h264#hardware"
    litokam1_sub:
      - rtsp://thingino:[email protected]:554/ch1
      - "ffmpeg:litokam1_sub#video=h264#hardware"
    ginzzu1:
      - rtsp://thingino:[email protected]:554/ch0
      - "ffmpeg:ginzzu1#video=h264#hardware"
    ginzzu1_sub:
      - rtsp://thingino:[email protected]:554/ch1
      - "fmpeg:ginzzu1_sub#video=h264#hardware"
    ginzzu2:
      - rtsp://thingino:[email protected]:554/ch0
      - "ffmpeg:ginzzu2#video=h264#hardware"
    ginzzu2_sub:
      - rtsp://thingino:[email protected]:554/ch1
      - "fmpeg:ginzzu2_sub#video=h264#hardware"
    cinnado-d1:
      - rtsp://thingino:[email protected]:554/ch0
    cinnado-d1_sub:
      - rtsp://thingino:[email protected]:554/ch1
    cinnado-d2:
      - rtsp://thingino:[email protected]:554/ch0
    cinnado-d2_sub:
      - rtsp://thingino:[email protected]:554/ch1

telemetry:
  # Optional: Enabled network interfaces for bandwidth stats monitoring (default: empty list, let nethogs search all)
  network_interfaces:
    - eth0
  # Optional: Configure system stats
  stats:
    # Enable network bandwidth stats monitoring for camera ffmpeg processes, go2rtc, and object detectors. (default: shown below)
    # NOTE: The container must either be privileged or have cap_net_admin, cap_net_raw capabilities enabled.
    network_bandwidth: True
  # Optional: Enable the latest version outbound check (default: shown below)
  # NOTE: If you use the HomeAssistant integration, disabling this will prevent it from reporting new versions
  version_check: True

cameras:
  litokam1:
    enabled: false
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/litokam1
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/litokam1_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
            - audio
    onvif:
      host: litokam1.iot.local
      port: 80
      user: thingino
      password: 4$3cr37
      autotracking:
        enabled: true
        calibrate_on_startup: false
        zooming: disabled
        # A lower value will keep more of the scene in view around a tracked object.
        # A higher value will zoom in more on a tracked object, but Frigate may lose tracking more quickly.
        # The value should be between 0.1 and 0.75
        zoom_factor: 0.3
        required_zones:
          - default
        movement_weights: 0.0, 1.0, 0.07234835624694824, 0.0699901862811017, -0.0017159346611269107
    detect:
      enabled: true
    audio:
      enabled: true
    record:
      enabled: true
    motion:
      threshold: 30
      contour_area: 10
    zones:
      default:
        coordinates: 0,0.541,1,0.572,1,1,0,1
        loitering_time: 0
        inertia: 2
  cinnado-d1:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/cinnado-d1
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/cinnado-d1_sub
          input_args: preset-rtsp-restream
          roles:
            - audio
            - detect
    onvif:
      host: cinnado-d1.iot.local
      port: 80
      user: thingino
      password: 4$3cr37
      autotracking:
        enabled: true
        calibrate_on_startup: false
        zooming: disabled
        zoom_factor: 0.3
        required_zones:
          - default
        movement_weights: 0.0, 1.0, 0.06849527359008789, 0.06990983716903196, 0.0032007671171619023
    detect:
      enabled: true
    audio:
      enabled: true
    record:
      enabled: true
    motion:
      threshold: 30
      contour_area: 10
      improve_contrast: 'true'
    zones:
      default:
        coordinates: 0,0.541,1,0.572,1,1,0,1
        loitering_time: 0
        inertia: 2
  cinnado-d2:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/cinnado-d2
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/cinnado-d2_sub
          input_args: preset-rtsp-restream
          roles:
            - audio
            - detect
    onvif:
      host: cinnado-d2.iot.local
      port: 80
      user: thingino
      password: 4$3cr37
      autotracking:
        enabled: true
        calibrate_on_startup: false
        zooming: disabled
        zoom_factor: 0.3
        return_preset: Home
        required_zones:
          - default
        movement_weights: 0.0, 1.0, 0.07111072540283203, 0.0018210358209467651, 0.05707824273819155
    detect:
      enabled: true
    audio:
      enabled: true
    record:
      enabled: true
    motion:
      threshold: 30
      contour_area: 10
      improve_contrast: 'true'
    zones:
      default:
        coordinates: 0,1,1,0,1,1
        loitering_time: 0
        inertia: 3
  cinnado-d3:
    enabled: false
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/cinnado-d3
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/cinnado-d3_sub
          input_args: preset-rtsp-restream
          roles:
            - audio
            - detect
    onvif:
      host: cinnado-d3.iot.local
      port: 80
      user: thingino
      password: 4$3cr37
      autotracking:
        enabled: true
        calibrate_on_startup: false
        zooming: disabled
        zoom_factor: 0.3
        return_preset: Home
        required_zones:
          - default
        movement_weights: 0.0, 1.0, 0.06697750091552734, 0.07079035492353541, 0.003817764405281324
    detect:
      enabled: true
    audio:
      enabled: true
    record:
      enabled: true
    motion:
      threshold: 30
      contour_area: 10
      improve_contrast: 'true'
    zones:
      default:
        coordinates: 0.299,0,0.427,0.959,0.841,0.847,0.924,0
        loitering_time: 0
  wvdb1:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/wvdb1
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/wvdb1_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
            - audio
    detect:
      enabled: true
    audio:
      enabled: true
    record:
      enabled: true
    motion:
      threshold: 30
      contour_area: 10
      improve_contrast: 'true'
    zones:
      Default:
        coordinates: 
          0.662,0.813,0.562,0.69,0.73,0.623,0.845,0.554,0.509,0.568,0.702,0.575,0.484,0.582,0.351,0.595,0.165,0.663,0.104,0.868,0.004,0.922,0,1,1,1,1,0.792,0.805,0.731
        loitering_time: 0
  ginzzu1:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/ginzzu1
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/ginzzu1_sub
          input_args: preset-rtsp-restream
          roles:
            - audio
            - detect
    onvif:
      host: ginzzu1.iot.local
      port: 80
      user: thingino
      password: 4$3cr37
      autotracking:
        enabled: true
        calibrate_on_startup: false
        zooming: disabled
        zoom_factor: 0.3
        return_preset: Home
        required_zones:
          - default
        movement_weights: 0.0, 1.0, 0.07828545570373535, 0.12285056021615348, 0.01581922659625892
    detect:
      enabled: true
    audio:
      enabled: true
    record:
      enabled: true
    motion:
      threshold: 30
      contour_area: 10
      improve_contrast: 'true'
    zones:
      default:
        coordinates: 0.99,0.984,0.867,0.348,0.237,0,0,0.024,0.003,0.995
        loitering_time: 0
  ginzzu2:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/ginzzu2
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/ginzzu2_sub
          input_args: preset-rtsp-restream
          roles:
            - audio
            - detect
    onvif:
      host: ginzzu2.iot.local
      port: 80
      user: thingino
      password: 4$3cr37
      autotracking:
        enabled: true
        calibrate_on_startup: false
        zooming: disabled
        zoom_factor: 0.3
        return_preset: Home
        required_zones:
          - default
        movement_weights: 0.0, 1.0, 0.07828545570373535, 0.12285056021615348, 0.01581922659625892
    detect:
      enabled: true
    audio:
      enabled: true
    record:
      enabled: true
    motion:
      threshold: 30
      contour_area: 10
      improve_contrast: 'true'
    zones:
      default:
        coordinates: 0.99,0.984,0.867,0.348,0.237,0,0,0.024,0.003,0.995
        loitering_time: 0
version: 0.15-0
camera_groups:
  Home:
    order: 1
    icon: LuHome
    cameras:
      - birdseye
      - litokam1
      - cinnado-d1
      - cinnado-d2
      - cinnado-d3
      - wvdb1
      - ginzzu1
      - ginzzu2

Also, here is the config I was trying with the hard-coded ffmpeg options in go2rtc section and the new build off ffmpeg you provided. Let me know if there is a better way for me to help you test things.

cat scrubbed-config-rktest.yaml 
mqtt:
  enabled: true
  host: mqtt.local

ffmpeg:
  hwaccel_args: preset-rk-h265

detectors: # required
  rknn: # required
    type: rknn # required
    # number of NPU cores to use
    # 0 means choose automatically
    # increase for better performance if you have a multicore NPU e.g. set to 3 on rk3588
    num_cores: 3

model: # required
  # name of model (will be automatically downloaded) or path to your own .rknn model file
  # possible values are:
  # - deci-fp16-yolonas_s
  # - deci-fp16-yolonas_m
  # - deci-fp16-yolonas_l
  # - /config/model_cache/your_custom_model.rknn
  path: deci-fp16-yolonas_s
  # width and height of detection frames
  width: 320
  height: 320
  # pixel format of detection frame
  # default value is rgb but yolo models usually use bgr format
  input_pixel_format: bgr # required
  # shape of detection frame
  input_tensor: nhwc
  # needs to be adjusted to model, see below
  labelmap_path: /labelmap/coco-80.txt

record:
  enabled: True
  retain:
    days: 3
    mode: motion

go2rtc:
  streams:
    wvdb1:
      - exec:ffmpeg -hide_banner -allowed_media_types video+audio -init_hw_device rkmpp=hw -filter_hw_device hw -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp -i rtsp://thingino:p@[email protected]:554/ch0 -vf hwupload,vpp_rkrga=w=2048:h=1536:transpose=clock -c:v h264_rkmpp -c:a copy -f rtsp {{output}}
    wvdb1_sub:
      - exec:ffmpeg -hide_banner -allowed_media_types video+audio -init_hw_device rkmpp=hw -filter_hw_device hw -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp -i rtsp://thingino:p@[email protected]:554/ch1 -vf hwupload,vpp_rkrga=w=2048:h=1536:transpose=clock -c:v h264_rkmpp -c:a copy -f rtsp {{output}}
    litokam1:
      #- rtsp://thingino:p@[email protected]:554/ch1
      - exec:ffmpeg -hide_banner -allowed_media_types video+audio -init_hw_device rkmpp=hw -filter_hw_device hw -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp -i rtsp://thingino:p@[email protected]:554/ch0 -c:v h264_rkmpp -c:a copy -f rtsp {{output}}
    litokam1_sub:
      # - rtsp://thingino:p@[email protected]:554/ch1
      - exec:ffmpeg -hide_banner -allowed_media_types video+audio -init_hw_device rkmpp=hw -filter_hw_device hw -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp -i rtsp://thingino:p@[email protected]:554/ch1 -c:v h264_rkmpp -c:a copy -f rtsp {{output}}
    ginzzu1:
      #- rtsp://thingino:p@[email protected]:554/ch0
      - exec:ffmpeg -hide_banner -allowed_media_types video+audio -init_hw_device rkmpp=hw -filter_hw_device hw -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp -i rtsp://thingino:p@[email protected]:554/ch0 -c:v h264_rkmpp -c:a copy -f rtsp {{output}}
    ginzzu1_sub:
      - rtsp://thingino:p@[email protected]:554/ch1
      - exec:ffmpeg -hide_banner -allowed_media_types video+audio -init_hw_device rkmpp=hw -filter_hw_device hw -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp -i rtsp://thingino:p@[email protected]:554/ch1 -c:v h264_rkmpp -c:a copy -f rtsp {{output}}
    cinnado-d1:
      - rtsp://thingino:p@[email protected]:554/ch0
    cinnado-d1_sub:
      - rtsp://thingino:p@[email protected]:554/ch1
    cinnado-d2:
      - rtsp://thingino:p@[email protected]:554/ch0
    cinnado-d2_sub:
      - rtsp://thingino:p@[email protected]:554/ch1

telemetry:
  # Optional: Enabled network interfaces for bandwidth stats monitoring (default: empty list, let nethogs search all)
  network_interfaces:
    - eth0
  # Optional: Configure system stats
  stats:
    # Enable network bandwidth stats monitoring for camera ffmpeg processes, go2rtc, and object detectors. (default: shown below)
    # NOTE: The container must either be privileged or have cap_net_admin, cap_net_raw capabilities enabled.
    network_bandwidth: True
  # Optional: Enable the latest version outbound check (default: shown below)
  # NOTE: If you use the HomeAssistant integration, disabling this will prevent it from reporting new versions
  version_check: True

cameras:
  litokam1:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/litokam1
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/litokam1_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
            - audio
    onvif:
      host: litokam1.iot.local
      port: 80
      user: thingino
      password: p@$$yp4$$
      autotracking:
        enabled: true
        calibrate_on_startup: false
        zooming: disabled
        # A lower value will keep more of the scene in view around a tracked object.
        # A higher value will zoom in more on a tracked object, but Frigate may lose tracking more quickly.
        # The value should be between 0.1 and 0.75
        zoom_factor: 0.3
        required_zones:
          - default
        movement_weights: 0.0, 1.0, 0.07234835624694824, 0.0699901862811017, -0.0017159346611269107
    detect:
      enabled: true
    audio:
      enabled: true
    record:
      enabled: true
    motion:
      threshold: 30
      contour_area: 10
    zones:
      default:
        coordinates: 0,0.541,1,0.572,1,1,0,1
        loitering_time: 0
        inertia: 2
  cinnado-d1:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/cinnado-d1
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/cinnado-d1_sub
          input_args: preset-rtsp-restream
          roles:
            - audio
            - detect
    onvif:
      host: cinnado-d1.iot.local
      port: 80
      user: thingino
      password: p@$$yp4$$
      autotracking:
        enabled: true
        calibrate_on_startup: false
        zooming: disabled
        zoom_factor: 0.3
        required_zones:
          - default
        movement_weights: 0.0, 1.0, 0.06849527359008789, 0.06990983716903196, 0.0032007671171619023
    detect:
      enabled: true
    audio:
      enabled: true
    record:
      enabled: true
    motion:
      threshold: 30
      contour_area: 10
      improve_contrast: 'true'
    zones:
      default:
        coordinates: 0,0.541,1,0.572,1,1,0,1
        loitering_time: 0
        inertia: 2
  cinnado-d2:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/cinnado-d2
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/cinnado-d2_sub
          input_args: preset-rtsp-restream
          roles:
            - audio
            - detect
    onvif:
      host: cinnado-d2.iot.local
      port: 80
      user: thingino
      password: p@$$yp4$$
      autotracking:
        enabled: true
        calibrate_on_startup: false
        zooming: disabled
        zoom_factor: 0.3
        return_preset: Home
        required_zones:
          - default
        movement_weights: 0.0, 1.0, 0.07111072540283203, 0.0018210358209467651, 0.05707824273819155
    detect:
      enabled: true
    audio:
      enabled: true
    record:
      enabled: true
    motion:
      threshold: 30
      contour_area: 10
      improve_contrast: 'true'
    zones:
      default:
        coordinates: 0,1,1,0,1,1
        loitering_time: 0
        inertia: 3
  cinnado-d3:
    enabled: false
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/cinnado-d3
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/cinnado-d3_sub
          input_args: preset-rtsp-restream
          roles:
            - audio
            - detect
    onvif:
      host: cinnado-d3.iot.local
      port: 80
      user: thingino
      password: p@$$yp4$$
      autotracking:
        enabled: true
        calibrate_on_startup: false
        zooming: disabled
        zoom_factor: 0.3
        return_preset: Home
        required_zones:
          - default
        movement_weights: 0.0, 1.0, 0.06697750091552734, 0.07079035492353541, 0.003817764405281324
    detect:
      enabled: true
    audio:
      enabled: true
    record:
      enabled: true
    motion:
      threshold: 30
      contour_area: 10
      improve_contrast: 'true'
    zones:
      default:
        coordinates: 0.299,0,0.427,0.959,0.841,0.847,0.924,0
        loitering_time: 0
  wvdb1:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/wvdb1
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/wvdb1_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
            - audio
    detect:
      enabled: true
    audio:
      enabled: true
    record:
      enabled: true
    motion:
      threshold: 30
      contour_area: 10
      improve_contrast: 'true'
    zones:
      Default:
        coordinates: 
          0.662,0.813,0.562,0.69,0.73,0.623,0.845,0.554,0.509,0.568,0.702,0.575,0.484,0.582,0.351,0.595,0.165,0.663,0.104,0.868,0.004,0.922,0,1,1,1,1,0.792,0.805,0.731
        loitering_time: 0
  ginzzu1:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/ginzzu1
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/ginzzu1_sub
          input_args: preset-rtsp-restream
          roles:
            - audio
            - detect
    onvif:
      host: ginzzu1.iot.local
      port: 80
      user: thingino
      password: p@$$yp4$$
      autotracking:
        enabled: true
        calibrate_on_startup: false
        zooming: disabled
        zoom_factor: 0.3
        return_preset: Home
        required_zones:
          - default
        movement_weights: 0.0, 1.0, 0.07828545570373535, 0.12285056021615348, 0.01581922659625892
    detect:
      enabled: true
    audio:
      enabled: true
    record:
      enabled: true
    motion:
      threshold: 30
      contour_area: 10
      improve_contrast: 'true'
    zones:
      default:
        coordinates: 0.99,0.984,0.867,0.348,0.237,0,0,0.024,0.003,0.995
version: 0.15-0
camera_groups:
  Home:
    order: 1
    icon: LuHome
    cameras:
      - birdseye
      - litokam1
      - cinnado-d1
      - cinnado-d2
      - cinnado-d3
      - wvdb1
      - ginzzu1

[MarcA711]

Hey, since this is rather frigate related, we can discuss this in further detail here: blakeblackshear/frigate#11502. This way more people can see this. For this request it is fine, but if you have any more questions, we should move to the other thread.

I am particularly unclear on where or if I should even be setting hwaccel_args presets since the docs seem to suggest setting preset-rk-h264 or preset-rk-h265 at a global level, yet I have cameras that output both.

Setting the preset on the camera level is fine and should be done in your case, where your cameras output different codecs.

Rockchip hardware video acceleration has some limits on how much FPS it can achieve. So you want to transcode as few frames as possible per second.

You should only use go2rtc for your main stream. Also, for a security camera lower frame rates might be enough. 10 fps will still give a good video (but noticeably laggy) and will reduce the load that the hardware needs to handle. As a good side effect, the video need less disk space.

I wrote a quite detailed answer to a similar request here, if you are interested: blakeblackshear/frigate#14395 (reply in thread)