GStreamer libcamerasrc sensor-mode support

[kraln]

I find myself wanting to override the automatic sensor-mode selection performed by libcamera, as it seems to always pick the not-right one ;)

I found this patchwork from 2023, but it doesn't seem to have made its way into the project: https://patchwork.libcamera.org/cover/18458/

any idea why, or if there is a timeline or better way to accomplish this functionality?

[naushir]

Indeed this can be quite troublesome. Pinging @kbingham to see if this patch can be progressed.

[kbingham]

Taking a look!

[kbingham]

So it looks like it never made it's way in because no one continued the discussion or the implementation. It's a tricky subject because not all of our pipelines support RAW roles and a processed role at the same time - so the 'mechanism' wasn't possible back then.

Since then we added a SensorConfiguration structure however which I think could better describe this and be tied into the gstreamer component - but it's still an area with rough edges that needs someone to look at in more detail I fear.

[kraln]

What I am trying to accomplish:

I have an imx290 (variant...) which offers a few different modes--1920x1080 and 1280x720, 12 and 10bit. The 1280x720 is unfortunately a crop. I'd like to arrange a stream where the input from the camera is 1920x1080x12, and the output is 960x540 (half)--with this resizing being done inside the pi isp. I can arrange this with rpicam-vid (or hello) fairly trivially, but as soon as I ask for a lower resolution with libcamerasrc, it starts to pick the smaller camera mode and introduces cropping. I can of course ask for the full resolution and then resize it with videoconvertscale, but that then uses cpu resources. Especially as I ultimately want two 960x540 streams, one YUV one GRAY, I'd like to avoid the CPU until the last possible moment for reasons of resource management.

$ libcamera-vid --list-cameras
Available cameras
-----------------
0 : imx290 [1920x1080 12-bit RGGB] (/base/axi/pcie@120000/rp1/i2c@88000/imx290@1a)
    Modes: 'SRGGB10_CSI2P' : 1280x720 [60.00 fps - (320, 180)/1280x720 crop]
                             1920x1080 [60.00 fps - (0, 0)/1920x1080 crop]
           'SRGGB12_CSI2P' : 1280x720 [60.00 fps - (320, 180)/1280x720 crop]
                             1920x1080 [60.00 fps - (0, 0)/1920x1080 crop]

1 : imx290 [1920x1080 12-bit RGGB] (/base/axi/pcie@120000/rp1/i2c@80000/imx290@1a)
    Modes: 'SRGGB10_CSI2P' : 1280x720 [60.00 fps - (320, 180)/1280x720 crop]
                             1920x1080 [60.00 fps - (0, 0)/1920x1080 crop]
           'SRGGB12_CSI2P' : 1280x720 [60.00 fps - (320, 180)/1280x720 crop]
                             1920x1080 [60.00 fps - (0, 0)/1920x1080 crop]

$ gst-launch-1.0 libcamerasrc name=frame_input camera-name=/base/axi/pcie@120000/rp1/i2c@88000/imx290@1a ! video/x-raw,format=NV12,width=1920,height=1080,framerate=20/1  ! videoflip video-direction=identity   ! queue ! videoconvertscale name=frame_output ! ximagesink
Setting pipeline to PAUSED ...
[1:47:49.425519479] [10231]  INFO Camera camera_manager.cpp:327 libcamera v0.3.2+218-b7ba311b
[1:47:49.438612510] [10233]  INFO RPI pisp.cpp:720 libpisp version v1.0.7 28196ed6edcf 29-08-2024 (16:33:32)
[1:47:49.443989627] [10233]  INFO RPI pisp.cpp:1179 Registered camera /base/axi/pcie@120000/rp1/i2c@88000/imx290@1a to CFE device /dev/media0 and ISP device /dev/media3 using PiSP variant BCM2712_C0
[1:47:49.444048053] [10233]  INFO RPI pisp.cpp:720 libpisp version v1.0.7 28196ed6edcf 29-08-2024 (16:33:32)
[1:47:49.448012927] [10233]  INFO RPI pisp.cpp:1179 Registered camera /base/axi/pcie@120000/rp1/i2c@80000/imx290@1a to CFE device /dev/media2 and ISP device /dev/media4 using PiSP variant BCM2712_C0
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
[1:47:49.449632484] [10239]  INFO Camera camera.cpp:1197 configuring streams: (0) 1920x1080-NV12
[1:47:49.449767614] [10233]  INFO RPI pisp.cpp:1484 Sensor: /base/axi/pcie@120000/rp1/i2c@88000/imx290@1a - Selected sensor format: 1920x1080-SRGGB12_1X12 - Selected CFE format: 1920x1080-PC1R

$ gst-launch-1.0 libcamerasrc name=frame_input camera-name=/base/axi/pcie@120000/rp1/i2c@88000/imx290@1a ! video/x-raw,format=NV12,width=960,height=540,framerate=20/1  ! videoflip video-direction=identity   ! queue ! videoconvertscale name=frame_
output ! ximagesink
Setting pipeline to PAUSED ...
[1:55:01.244673003] [10265]  INFO Camera camera_manager.cpp:327 libcamera v0.3.2+218-b7ba311b
[1:55:01.259771721] [10267]  INFO RPI pisp.cpp:720 libpisp version v1.0.7 28196ed6edcf 29-08-2024 (16:33:32)
[1:55:01.266202968] [10267]  INFO RPI pisp.cpp:1179 Registered camera /base/axi/pcie@120000/rp1/i2c@88000/imx290@1a to CFE device /dev/media0 and ISP device /dev/media3 using PiSP variant BCM2712_C0
[1:55:01.266269097] [10267]  INFO RPI pisp.cpp:720 libpisp version v1.0.7 28196ed6edcf 29-08-2024 (16:33:32)
[1:55:01.271238880] [10267]  INFO RPI pisp.cpp:1179 Registered camera /base/axi/pcie@120000/rp1/i2c@80000/imx290@1a to CFE device /dev/media2 and ISP device /dev/media4 using PiSP variant BCM2712_C0
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
[1:55:01.273363715] [10273]  INFO Camera camera.cpp:1197 configuring streams: (0) 960x540-NV12
[1:55:01.273534030] [10267]  INFO RPI pisp.cpp:1484 Sensor: /base/axi/pcie@120000/rp1/i2c@88000/imx290@1a - Selected sensor format: 1280x720-SRGGB12_1X12 - Selected CFE format: 1280x720-PC1R

Until this makes its way through the development process, is there an alternative suggestion on how to set up something like what I want?

Alternatively, is this an area where I could somehow support?

[kraln]

p.s. on the pi5, it seems that v4l2transform got thrown out with the h264 stuff, so there doesn't seem to be a hardware-accelerated way to downscale unless you use the ISP. This renders computer vision applications nearly completely unusable with higher-resolution sensors if the first step is a cpu-resize.

[kraln]

Confirming, when I patch libcamera to always choose the larger source format rather than the smaller one, and let the ISP do the resizing, the CPU usage is approximately 1/3. (from 15% of one core to 5% of one core)