Fix YUV444 10 and 12 bits

[thesword53]

NVIDIA driver uses 16 bits surfaces for YUV 444 10 and 12 bits decoding.
VA_FOURCC_Q410 (yuv444p10), VA_FOURCC_Q412 (yuv444p12) formats lead to invalid video output, so I only kept VA_FOURCC_Q416 (yuv444p16).

To test these formats, I patched ffmpeg and libva with following files:

ffmpeg 6.0:

diff --git a/libavcodec/vaapi_decode.c b/libavcodec/vaapi_decode.c
index 134f10eca5..af95b94caa 100644
--- a/libavcodec/vaapi_decode.c
+++ b/libavcodec/vaapi_decode.c
@@ -289,6 +289,9 @@ static const struct {
 #ifdef VA_FOURCC_I010
     MAP(I010, YUV420P10),
 #endif
+#ifdef VA_FOURCC_Q416
+    MAP(Q416, YUV444P16),
+#endif
 #undef MAP
 };

diff --git a/libavutil/hwcontext_vaapi.c b/libavutil/hwcontext_vaapi.c
index 938bd5447d..ec70775556 100644
--- a/libavutil/hwcontext_vaapi.c
+++ b/libavutil/hwcontext_vaapi.c
@@ -161,6 +161,10 @@ static const VAAPIFormatDescriptor vaapi_format_map[] = {
     // support for Y412, so we must fudge the mapping here.
     MAP(Y412, YUV444_12,  XV36, 0),
 #endif
+#ifdef VA_FOURCC_Q416
+    MAP(Q416, YUV444_10,  YUV444P16, 0),
+    MAP(Q416, YUV444_12,  YUV444P16, 0),
+#endif
 };
 #undef MAP

libva 2.19.0

diff --git a/va/va.h b/va/va.h
index 5b98658..bb44539 100644
--- a/va/va.h
+++ b/va/va.h
@@ -4574,6 +4574,11 @@ VAStatus vaSyncBuffer(
  * Four bytes per pixel: X, Y, U, V.
  */
 #define VA_FOURCC_XYUV          0x56555958
+/** Q416: three-plane 16-bit YUV 4:4:4.
+ *
+ * The three planes contain Y, U and V respectively.
+ */
+#define VA_FOURCC_Q416          0x36313451

 /* byte order */
 #define VA_LSB_FIRST        1

I will also pull request these patches to libva and FFmpeg repo.

[elFarto]

Thanks for the patch. It's probably worth waiting until the libva patch is accepted before merging this in.

[thesword53]

The pull request for VA_FOURCC_Q416 has been merged to libva 2.20.0 intel/libva#735