Fails to build against kernel 5.11.x #35
Comments
|
Im having the same problem and looks like h264 definitions in the kernel have been updated for example https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v5.11.21&id=2287c5e65cbcc99633c412dbfe1d39bd9f7bf1ce I did try to hack in some changes too see if I could sync up the api changes and use the headers from the 5.11 kernel but getting too deep for me to understand. At runtime I was left with:
Edit: The tiled_to_planar issue seems resolved in this pull request #32 for aarch64 but still vaInitialize errors that I don't have the expertise to understand:
|
|
I can confirm that I cannot compile it for kernel 5.12. In the past, I was able to get it working for kernel 5.7, making small changes to the kernel source to revert the new h264 definitions that @bjharper mentioned, see #31. Now the kernel sources are much more complicated to follow what changed in the h264 def. and I could not revert it, although it would be much better to modify the libva-v4l2-request to follow the kernel modifications. Unfortunately, I have no idea how to do it and already spent many hours on it. It is a pity that a lot of work was done to get Cedrus working in the past, and we cannot use it after 2 years that the development stopped. I'm afraid that Cedrus will never go from stagging and may be dropped from the kernel sources entirely. |
|
There is no intention to drop Cedrus. In fact, there is a talk to promote it to stable in near future, together with Hantro. This library is not maintained though. Easy way to use it would be 5.12 kernel and gstreamer build from latest development sources. Another possibility would be ffmpeg with additional patches... |
|
@jernejsk That is very good news! I'm already using kernel 5.12. Can you point me to the GStreamer version that would work? I heard about this patched ffmpeg version, is this patch from LibreELEC you are referring to? Should I use it the same way as in:
would this work? I'll try it fast :) Thank you! |
|
As I said, use latest gstreamer development sources from git. Latest stable version doesn't include necessary improvements yet. And I don't know what command would be needed to actually use Cedrus. Check #linux-sunxi IRC log. Regarding ffmpeg, that patchset is for kernel 5.10 and it presumes that additional kernel patches are applied for codecs other than H264. And no, that line won't work. Those patches have nothing to do with VAAPI. Something like this may work: Both approaches were already discussed on #linux-sunxi IRC and Armbian forum, so I suggest you check there for further info. |
|
That won't help with Firefox, which is using VA-API for hardware video decoding. Or Chromium with the unofficial patch that most distributions are using. We really need a working VA-API driver for the most common use case here. Regarding the FFmpeg patch, I have seen no response from its author to initial review on the developer mailing list, so I don't think it's going to be included any time soon. |
|
Yes I'm looking for VA-API solution for amlogic platforms (particularly for firefox / chromium). . I might have another go at resolving some of the issues interfacing with the newer kernel but it will be a big learning curve for me. ffmpeg with v4l2 decoder is working ok on my platform. |
|
@rathann The thing is: development is driven by developer needs or contracts. In my case, I only care about Cedrus and ffmpeg (I'm co-author of ffmpeg patches) since I need them for Kodi on LibreELEC. Collabora mostly works only on Hantro and gstreamer. You can still check status of forks of this repo - at least one updated some codecs in the past. Regarding ffmpeg patches - there is still plan to send updated version but some parts need redesign (apart from addressing comments). Since V4L2 request API is still evolving and we don't have much spare time for this, it will take a while longer. @bjharper Amlogic V4L2 codecs don't use request API, so this is useless. They expect to receive full frames, while VA-API and V4L2 request API codecs expect to get parsed frames (extracted information from frame header). This is not a driver thing - it's how HW is designed. Technically it would be possible to restore full frames from VA-API data, but it's complicated. |
|
Looking for more information, I found this video on YouTube from Collabora, they explain the actual status of the decoder. At 31:11, Nicolas Dufresne (one of the developers behind Cedrus and Gstreamer Cedrus support) said they stopped developing this VA-API driver since FFmpeg, GStreamer and Chromium already have support. Any further development on this VA-API driver needs to come from the community. I believe they assume that there is no need for the VA-API if the application can directly access the decoder with the recent kernel development. This means that it will take a while for everything to get patched (or maybe never will). It is a pity because Firefox took so long to support VA-API, and now that it finally supports hardware video decoding on Linux, Cedrus doesn't support it anymore. I know Firefox uses FFmpeg for some tasks, but I don't know if it uses (or can use) for video decoding. @jernejsk Thank you for your work on the FFmpeg patches :). I can also confirm that Gstreamer works with hardware acceleration. I needed to compile it (version 1.19.0.1), and I used the following to test:
It works even on top of Wayland, very nice. However, I had some problems with 60fps 1080p videos, everything lower than this works without problems (maybe I need to add more CMA). To test with software decoding, I used |
|
Hi, I got the libva-v4l2-request to compile and run on Linux 5.12. It turns out we need to free the capture buffers. Apart from that, there are a couple of structs that changed, and some definition names as well, for instance: the control I also tried to follow what the people from GStreamer did to get it to work on the merge-request that added cedrus support. when I run:
Everything seems fine, I don't get any driver errors, and the decoding speed is around 10x with a small load on the CPU, in line with what I would expect for hardware decoding. With software decoding for the same video, I get around 2x with all the cores at 100%. when I run cvlc or ffmpeg with output to fbdev, like in:
The video image is not correct. It seems that we still need to set the video output format, or there is still something missing in the initialization. Maybe it is now that the problem really starts :-) The noisy image is dynamic, and it seems that the hardware decoding is going on but in the wrong video format.
Since I'm not a developer by any means, and I know nothing about video decoding/encoding, v4l2 implementation... And I don't get any error to debug. I don't really know where to look to find the cause of this problem. It would be very good if someone can point me in a direction and indicate an example to follow. Any help is welcome. @jernejsk do you have any idea where I could look to fix it? I think libva on Cedrus is very important because it would make the driver compatible with much more software already developed. You can see the changes I made to get to this stage on my fork, on this commit. I adapted a few changes from pH5 on #29 and the Tile format fix from Kwizart on #32. Thank you all, |
|
Not really, no. Most probably some field in some control is not set correctly. Now comes the hard part - dumping register values (on kernel side) or dumping control values (on userspace side) and comparing them... |
|
@jernejsk thank you. The problem also happens with MPEG2 videos. Most of the changes in the kernel were for h264... So, maybe it is something before we send the slices. I noticed that on the GStreamer patch that they set a single control ( On the MPEG2 decoding, we set only two controls: |
|
@wolfallein Thanks for testing it on gstreamer. I'm trying the same where I've built the gstreamer master since there seems to be some changes related to v4l2codes/h264 has been merged recently. It is not playing while I've used the below pipeline, one you mentioned on the same thread. However it is playing while I have passed the videoconvert in between decoder and autovideosink but it is playing slow even for 720p video. These are my systems details: Please let me know if I miss anything? May be sharing your root can cross-check if anything missing on my gst configuration? Jagan. |
|
Hi @jernejsk I want to try to get v4l2-request working with ffmpeg under Armbian buster on my T95 Max H6 TV box as the stepping stone to mpv. Which version or commit of ffmpeg is your patch for? It doesn't seem to apply cleanly to ffmpeg 5.0 nor the latest git version and it seems you only updated it 10 days ago. Thanks |
|
I only update patches for LibreELEC/Kodi use. In order to use them, you need ffmpeg 4.4 and kernel >= 5.14 (no other combination currently supported nor planned). HEVC support won't work without additional kernel patches. mpv can currently use ffmpeg v4l2 request api acceleration only in gbm mode (no desktop running). |
|
Thanks @jernejsk |
|
I've tried a couple of times now to build your patched ffmpeg 4.4 under Armbian bullseye with a 5.15 kernel and headers on my H6 TV box. Your patch applied cleanly and the last ffmpeg configure command I tried was: I installed all of the ffmpeg build deps by running As you can see I've tried using What configure options do you use to build ffmpeg for LE? |
|
|
|
Thanks @jernejsk ! I left ffmpeg building last night and I'm happy to say it did build successfully but I've not had time to test it yet, I will do that tonight. I don't need VP9 support and I couldn't be bothered to rebuild my kernel so, for those following along at home, the missing configure option I required for a successful build was Under mpv I use the option |
|
Note that this really works with GBM mode only (no desktop environment), so full screen is kinda assumed. H6 doesn't have rotation core. Even if it had, there is no ffmpeg filter which would take advantage of it. Some Cedrus variants support rotation for some codecs, but it's not implemented. In short, no HW accelerated rotation on Allwinner SoCs for now. |
|
Yes I know about the GBM only bit and I put 'full screen' in quotes because what I meant is not playing a landscape video in portrait mode which leads to big borders and so its not 'full screen'. Shame to hear about the lack of hw accelerated rotation in the cedrus driver. Have you got any plans to work on adding hw accel rotation? |
|
No and even if it would be added, I'm not sure it would be easy to expose it in ffmpeg API. Even if that was done, I'm not sure mpv would automatically work with it. This is pretty unexplored territory, at least from my side, since I don't need it. |
|
After considerable effort, I think I should have all my ducks in a row now for v4l2-request playback yet instead I'm getting this error:
When trying to play a UHD h264 on my T95 MAX H6 TV box running file under Armbian bullseye w/ kernel 5.15 and your patched ffmpeg. I have been trying this command, based on a command posted in an earlier comment: I only have a /dev/video0 and a /dev/media0, no /dev/video1 device here. I found this page so I also tried exporting these: What command should I be using to play a h264 file with ffmpeg or ffplay from the console using v4l2-request hwaccel? |
|
After building and installing your patched ffmpeg, I built and installed the latest git version of mpv and tried using the command used in the link in my previous post but I get this error:
|
|
I guess you misunderstand something. LE ffmpeg patches have nothing to do with VA-API. They are meant to directly talk to Cedrus driver, without intermediate lib like libva-v4l2-request. This lib is not maintained, so it doesn't work with latest uAPI. |
|
I did try to build libva-v4l2-request but I didn't think it was necessary or used by your patched ffmpeg. It failed to build. It would seem from the comments in this and the other ticket that people have got both ffmpeg and mpv working outside of LE with their Allwinner devices so I suppose I need to know:
Thanks v.much for your help Jernej! I will document (and package) this on the Armbian forums if I get it working so you can send someone to my link in the future, hopefully. If I get mpv working I'm going to write a simple |
|
I work only with built-in drivers due to my dev setup, so I can't really tell you which modules should be loaded. Anyway, if you have cedrus among them, it should be good. No EVs are necessary for ffmpeg nor mpv, just make sure ffmpeg libs are installed system wide and mpv was compiled for same ffmpeg version (mpv will complain otherwise). you can test with ffmpeg commands: I forgot mpv command, but if you enable debug output, it should be pretty obvious, if request api is used. I guess you can set hwdec to auto and see if that works. Note, gstreamer 1.20 was just released, which supports request api decoding of MPEG2, VP8 and H264 out of the box (maybe even VP9), without patching anything. However, I have very little experience there, so I can't help you with that. |
|
I was miles off with my ffmpeg command so I'm glad I asked someone who knows whats going on here! fbdev output is enabled by default in ffmpeg and it would seem my kernel has fbdev support enabled because I have a /dev/fb0 but your ffmpeg command hasn't worked with my test files. Maybe I'd need to transcode them to use bgra instead because I noticed the swscaler warning about no accelerated colorspace conversion. Is that likely to make videos unwatchable ie less than 24 fps when they should be ~30 fps, if I could get one to play? If I have to transcode videos, what are the optimal h264 or h265 encoding options for playback on cedrus? These videos play fine with ffmpeg and mpv on the Jetson Nano and under amd64 Intel/Nvidia Linux etc. |
|
You can't avoid swscaler, as I already told you. Cedrus decodes to NV12 format, which is form of YUV 4:2:0, fbdev needs bgra, which is form of 8-bit RGB with alpha channel. Only DRM plane rendering can directly show that and even then HW will internally convert it to RGB (at least with current drivers). I suggest you use h264 video for testing purposes. uAPI is stable for some time now and it should be able to decode any H264 video out there. HEVC needs patching on kernel side to properly work. I'm not aware of any H264 video that can't be properly decoded, assuming it conforms to specs (many DVB sources don't always conform). While in theory you could put together HEVC video which couldn't be decoded on Cedrus, you would actually need to know how to do that. Practically, all HEVC videos work. Also make sure CMA size is set to at least 256 MB, 384 MB if you want 10-bit 4K HEVC. Also disable H6 IOMMU driver, because currently doesn't work well with Cedrus. If that doesn't help, I don't know. |
|
I'm more interested in mpv than gstreamer because I've not found any gstreamer based video player that offers the amount of playback options that mpv has BUT Will this latest gstreamer release allow for 4K h26X playback using v4l2-request under X or Wayland or both? That would be very cool if that works, even if only in full screen mode. Will update you on the CMA testing etc later. |
|
As I said, I have not much experience with gstreamer, I only tested VP9 without any desktop environment. But HEVC won't work since gstreamer is using only current uAPI, which doesn't cover everything. If you applied HEVC kernel patches, gstreamer won't work at all for HEVC. |
|
Note, I don't care about desktop environments on ARM, so I don't have much experience with video decoding there. But I know that mpv would need to be extended to work with request API on desktop environment. mpv renders such buffers only with DRM planes, but there is no reason why they couldn't be rendered via GPU. Somebody just needs to add code which imports dmabuf buffers to OpenGL ES. |
|
I have tried again but with a IOMMU is enabled in the kernel I'm using so I will try to rebuild the kernel without IOMMU. This is how my current ffmpeg build is configured: |
|
I installed a different kernel and ffmpeg isn't giving any errors now but I've not worked out the correct command to get a picture out of it yet. Under mpv, which I've only got to work with software decoding so far (I presume mpv must need libva-v4l2-request for HW decoding) I have to use the option Whilst I'm talking about HDMI pictures, if I disconnect the HDMI from my H6 TV box (running Linux 5.15) the picture doesn't return under X when I re-connect the HDMI cable. The non-X console doesn't have this issue but it picks one of two resolutions to use seemingly at random so sometimes I have to unplug and re-plug it in a few times to get the 4K res back. This has been the case for all of the kernels and distros I've used on it over the last year or so. Is this a known bug with AW H6? |
No, mpv works fine for me with modified ffmpeg libraries installed system wide. I tested this about 2 months ago, but I already forgot exact command used for that. You can try command from Kwiboo/mpv@9c39a05 commit message (changes are already present in latest mpv).
That's not unusual and it depends on driver loading order for GPU and display. One way to guarantee often assumed order (card0 display and card1 GPU) is to build panfrost as a module and sun4i-drm as built in. Regarding 4K resolution, make sure you have this patch: https://github.com/LibreELEC/LibreELEC.tv/blob/master/projects/Allwinner/patches/linux/0065-wip-fix-H6-4k-60.patch Other than that, I don't know. |
When building with latest OE-Core master. Running into bellow issue. After further research issue appears to be similar to bootlin/libva-v4l2-request#35 | src/libavcodec/hevc-ctrls-v2.h:59:8: error: redefinition of 'struct v4l2_ctrl_hevc_sps' | 59 | struct v4l2_ctrl_hevc_sps { | | ^~~~~~~~~~~~~~~~~~ | In file included from /../../../recipe-sysroot/usr/include/linux/videodev2.h:65, | from src/libavcodec/hevc-ctrls-v2.h:14: Disable ffmpeg libva-v4l2-request so that master builds and until better investigation can be done to fix issue. Signed-off-by: Vincent Davis Jr <[email protected]>
When building with latest OE-Core master. Running into bellow issue. After further research issue appears to be similar to bootlin/libva-v4l2-request#35 | src/libavcodec/hevc-ctrls-v2.h:59:8: error: redefinition of 'struct v4l2_ctrl_hevc_sps' | 59 | struct v4l2_ctrl_hevc_sps { | | ^~~~~~~~~~~~~~~~~~ | In file included from /../../../recipe-sysroot/usr/include/linux/videodev2.h:65, | from src/libavcodec/hevc-ctrls-v2.h:14: Disable ffmpeg libva-v4l2-request so that master builds and until better investigation can be done to fix issue. Signed-off-by: Vincent Davis Jr <[email protected]>
When building with latest OE-Core master. Running into bellow issue. After further research issue appears to be similar to bootlin/libva-v4l2-request#35 | src/libavcodec/hevc-ctrls-v2.h:59:8: error: redefinition of 'struct v4l2_ctrl_hevc_sps' | 59 | struct v4l2_ctrl_hevc_sps { | | ^~~~~~~~~~~~~~~~~~ | In file included from /../../../recipe-sysroot/usr/include/linux/videodev2.h:65, | from src/libavcodec/hevc-ctrls-v2.h:14: Disable ffmpeg libva-v4l2-request so that master builds and until better investigation can be done to fix issue. Signed-off-by: Vincent Davis Jr <[email protected]>
It looks like nearly all of the structs defined in
include/h264-ctrls.hare defined in/usr/include/linux/v4l2-controls.halready:If I patch out all the struct definitions, I get this error:
The text was updated successfully, but these errors were encountered: