H264Libav: Handle colour space conversion

[any1]

The scaler is now informed of the colour space encoded into the stream so that it may do the proper conversion.

[cgm999]

@any1 For those that are not knowing much about ffmpeg what are the benefits of the patch?

[any1]

The H.264 Annex B bit stream contains information about the YCbCr conversion coefficients, range, colour primaries and transfer characteristics. This passes that information on to the scaler so that it can convert the input colours so that they use the same colour profile as the RGB buffer into which they're rendered.

The parameter that is most important here and will be most obvious when it's wrong is colour range. H.264 is most often encoded using limited colour range, which means that values range from 16 to 235. This is done for no good reason other than "that's how it's always been done", but let's not dwell on it.

If the input has limited range, but you don't map it to the full 0 - 255 range when rendering it, the colours will not be quite right. The same applies, but in a different way if you do map something that's already full range to full range.

[cgm999]

@any1 Interesting, TYVM for your detalied explanation .

[cgm999]

Hi @any1, since you are already doing changes related to sws_scaling, can you also set that flag to SWS_BICUBIC :
sws = sws_getCachedContext(sws, frame->width, frame->height, avctx->pix_fmt,
frame->width, frame->height, AV_PIX_FMT_RGB32,
SWS_BICUBIC, nullptr, nullptr, nullptr);
without it it will not cache the sws object.. and return a new one every time (easy to see ,add below line after sws_getCachedContext )
vlog.error("sws=%p\n",sws);

I did tested this PR on my pikvm .. can not see issues with it .

[any1]

Sure, I can put a commit in there to fix this, but I'd use SWS_POINT as there is no actual scaling taking place. Using any form of interpolation is just a waste of resources.

[cgm999]

Agreed, I tested with SWS_POINT and it works fine.

[any1]

Added the commit for the sampling method.

[cgm999]

Thank you @any1 .
My testing (with pikvm as H264 source and 2 diff systems connected to it, changed resolutions ) did not show any issues with both commits applied.