Access to browser-inferred color space of an HTMLImageElement

[donmccurdy]

three.js recently shipped experimental support for wide-gamut Display P3 color spaces in WebGL.¹

An issue that has been difficult, in terms of providing a friendly API surface to three.js users, has been that three.js does know what color space an image uses unless the user explicitly tags each image:

const loader = new THREE.TextureLoader();
const texture = await loader.loadAsync('path/to/albedo_map.png');
texture.colorSpace = THREE.SRGBColorSpace;

Textures embedded in 3D models (e.g. .glb) have context that is reliable, but for loose textures we can't infer.

WebGL can read ICC profiles and unpack accordingly, but we only want to enable that option in certain cases – data textures like normal, ambient occlusion, roughness, and metalness textures are frequently tagged with sRGB ICC profiles while they are in fact non-color data, and shouldn't be decoded with the sRGB EOTF.

For that reason, we only enable WebGL's unpack option (to detect and decode color in images) if the user tags the image with a color space that requires decoding, and we need the user to tag every color texture.

In my view, we could offer a better experience if HTMLImageElement provided insight into what the browser believes an image's color space to be, which (presumably?) is what WebGL will use when unpacking. While I don't generally trust ICC profiles claiming an image is "srgb", an ICC profile claiming an image is "srgb-linear" or "display-p3" is very likely to be correct.

tl;dr – would exposing a property like HTMLImageElement#colorSpace, after an image has loaded, be a possibility in future color APIs?

Footnotes

1. https://threejs.org/examples/?q=wide#webgl_test_wide_gamut works in Chrome, and worked in Safari prior to a regression. ↩

[palemieux]

Yes, I think we need to explore two related topics:

• how to expose metadata of images loaded into a Canvas
• specify metadata for images saved from a Canvas

@ccameron-chromium has some ideas if I am not mistaken

[ccameron-chromium]

I think that this level of information would be best accessed via WebCodecs.

There is already something similar in WebCodecs' ImageDecoder API. That API will produce a VideoFrame, which has a VideoColorSpace, which gives the CICP values for the image (including the primaries and transfer function).

My sense is that it would be useful for WebCodecs to provide:

• the CICP data (as it does now)
  • it might be nice to know if this CICP was inferred from ICC vs being "the real thing"
• a raw data blob for the ICC profile (if present, and nothing if not present)
• the rest of the HDR metadata that can be attached (content color volume, content light level info, mastering display color volume, nominal diffuse white level).

[donmccurdy]

Currently three.js recommends using createImageBitmap to decode images, to move work off the main thread. But we default to HTMLImageElement for backward compatibility, browser support, and to avoid CORS restrictions.

If using the ImageDecoder API, can we still expect that decoding stays off the main thread?