hello_xr colors/non-linearity seems to be ill-defined and inconsistent, especially across different swapchain formats and headsets

[shinyquagsire23]

Big-picture issue:

I recently ran into a curious discrepancy between all of the headsets I have, as part of my attempts to add HDR support to the ALVR project, verifying my changes has no regressions:

These screenshots are of hello_xr depicted as they appeared to my own eyes in-headset, the key feature being the background darkness. To obtain them, I checked out the current main branch as of this issue, and forced the list of app-supported formats to just the described formats, only GL_RGBA16F and then only GL_SRGB8_ALPHA8. The headsets all say they support both of these swapchain formats. These headsets are all OpenXR certified, that I'm aware at least.

XrSwapchain seems to be too ambiguous, and the hello_xr conformance testing has not picked up on these different interpretations of XrSwapchain. That I can tell, there are 3 different assumptions I can see being made here:

• Meta: "Images submitted in sRGB color space must be created using an API-specific sRGB format" refers to the more-tightly defined specification of ITU-R BT.709, not non-linear RGB color. DXGI_FORMAT_R8G8B8A8_UNORM is not able to represent the full sRGB color space well. The compositor should only read linear colors, and gamma correction is done exclusively by the compositor. The underlying data is irrelevant, sRGB textures are just a compression format.
• Valve: "Images submitted in sRGB color space must be created using an API-specific sRGB format" means that if an application is submitting gamma-corrected color, it is signaled by creating a swapchain with an sRGB format. The application may be writing linear colors to an sRGB render target, or gamma-corrected colors to an RGB render target. The underlying data is relevant, and sRGB texture formats change how the frame is displayed.
• Pico: "Images submitted in sRGB color space must be created using an API-specific sRGB format" means that linear RGB colors must be corrected to be non-linear RGB (??)

hello_xr colors as linear RGB:

in src/common/xr_linear.h, there are the following definitions for grey:

static const XrColor4f XrColorLightGrey = {0.7f, 0.7f, 0.7f, 1.0f};
static const XrColor4f XrColorDarkGrey = {0.3f, 0.3f, 0.3f, 1.0f};

in web non-linear sRGB color, these would be #959595 and #dadada. Which, to me personally, is more of a light gray and a white in-headset. But, then again, Magenta is also labeled as XrColorPurple. The OpenXR spec has a very specific definition for XrColor4f, which is that it only holds linear RGB unless otherwise specified.

• #959595
• #dadada

There's also this SlateGrey color in hello_xr/options.h:

static const std::array<float, 4> SlateGrey{0.184313729f, 0.309803933f, 0.309803933f, 1.0f};

in web non-linear sRGB color, it's #779797. This is also a very light grey-blue, instead of a dark grey-blue.

• #779797

To provide an extra opinion on what dark grey, light grey and slate gray look like, here are GitHub's LaTeX interpretations:

$${\color{DarkGrey}Dark Grey █ }$$

$${\color{LightGrey}Light Grey █ }$$

$${\color{LightGrey}Slate Grey █ }$$

hello_xr colors as non-linear RGB:

Interpreting all of those same colors as gamma-corrected colors seems to yield colors closer to what they are labeled. Dark grey would be #4d4d4d, light gray #b2b2b2, slate gray would be #2f4f4f.

• #4d4d4d
• #b2b2b2
• #2f4f4f

Additionally, at some point in the past, the same slate gray was once described as DarkSlateGray. This name is still used by the current Pico 4 SDK sample, interestingly.

Takeaways:

• CTS should include testing all runtime-supported texture formats
• If Valve's OpenXR implementation is correct, hello_xr should gamma correct non-linear texture formats.
• If Pico's OpenXR implementation is correct, hello_xr should gamma correct linear texture formats.

[shinyquagsire23]

Did an additional test on Quest 2 and Quest3: They are perceptually identical to the Quest Pro for GL_RGBA16F and GL_SRGB8_ALPHA8.

[rpavlik]

Thanks for the thorough report.

Yeah I always think of it as being a weird Windows 95 desktop green, but I usually get funny looks when I say that 😅 I have no idea in whose universe "dark slate grey" is a sickly green, but fortunately naming colors is not my job.

Yeah I think with "linear unless otherwise specified", it tends to be mostly non-linear, because we do say "do not use linear 8bit per channel unless absolutely required".

We do test all the texture formats but we don't attempt to interactively render in all of them for perceptual comparison. (I suppose this would be easy enough to add, though making the test more usable and clear for pass vs failure might be tricky). I did think we had some linear vs gamma/srgb-eotf tests in the composition but presumably if we do, they miss this. I do know there are headsets out there where I see different interactive tests look different, which I have tracked internally as a conformance failure, but I didn't realize it was as widespread of a problem as you're showing.

(Note that hello_xr is not actually the conformance test, but the conformance tests do tend to look a lot like it in the interactive portions, including sharing the same color definitions.)

[rpavlik-bot]

An issue (number 2238) has been filed to correspond to this issue in the internal Khronos GitLab (Khronos members only: KHR:openxr/openxr#2238 ), to facilitate working group processes.

This GitHub issue will continue to be the main site of discussion.

[shinyquagsire23]

Thanks for the thorough report.

Yeah I always think of it as being a weird Windows 95 desktop green, but I usually get funny looks when I say that 😅 I have no idea in whose universe "dark slate grey" is a sickly green, but fortunately naming colors is not my job.

Yeah I think with "linear unless otherwise specified", it tends to be mostly non-linear, because we do say "do not use linear 8bit per channel unless absolutely required".

We do test all the texture formats but we don't attempt to interactively render in all of them for perceptual comparison. (I suppose this would be easy enough to add, though making the test more usable and clear for pass vs failure might be tricky). I did think we had some linear vs gamma/srgb-eotf tests in the composition but presumably if we do, they miss this. I do know there are headsets out there where I see different interactive tests look different, which I have tracked internally as a conformance failure, but I didn't realize it was as widespread of a problem as you're showing.

(Note that hello_xr is not actually the conformance test, but the conformance tests do tend to look a lot like it in the interactive portions, including sharing the same color definitions.)

Super appreciated, and yeah the issue writeup was kinda broad only because I wasn't sure if I was looking at a hello_xr-specific issue, a CTS issue, or a spec specificity issue, and the more I looked the more confused I got 😅. I figured the perceptual screenshots would help at least kickstart some kind of conversation needed to resolve it in any case.

[SafariMonkey]

Currently looking into this.

KhronosGroup/OpenXR-CTS#20 may be relevant. Looks like it was noticed that sRGB was incorrect on the CTS on OpenGL, so sRGB formats were excluded from selection. This lends credence to the idea that hello_xr is at least partially at fault here, as the CTS rendering code is (seemingly) derived from hello_xr.