Summary of the wide gamut support on Canvas

[fserb]

This is an IDL summary of the new proposed interface to support wide gamut on Canvas (half-float + extended SRGB)

2D Context

enum CanvasPixelFormat {
  "uint8", // default
  "float16",
};

dictionary CanvasRenderingContext2DSettings {    
  boolean alpha = true;    
  CanvasPixelFormat pixelFormat = "uint8";
};

dictionary CanvasContextCreationAttributesModule {    
  // Canvas 2D attributes    
  boolean alpha = true;  // Also used for WebGL.    
  CanvasPixelFormat pixelFormat = "uint8";
};

ImageData createImageData(unsigned long sw, unsigned long sh, 
    ImageDataColorSettings imageDataColorSettings);

ImageData createImageData(ImageDataArray data, unsigned long sw, unsigned long sh, 
    optional ImageDataColorSettings imageDataColorSettings);

ImageData

enum ImageDataStorageType {
  "uint8", // default
  "uint16",
  "float32",
};

typedef (Uint8ClampedArray or Uint16Array or Float32Array) ImageDataArray;

dictionary ImageDataColorSettings {
  ImageDataStorageType storageType = "uint8";
};

[
  Constructor(unsigned long sw, unsigned long sh, 
    optional ImageDataColorSettings imageDataColorSettings),
  Constructor(ImageDataArray data, unsigned long sw, optional unsigned long sh, 
    optional ImageDataColorSettings imageDataColorSettings),
  Exposed=(Window,Worker)] interface ImageData {
  readonly attribute unsigned long width;
  readonly attribute unsigned long height;
  readonly attribute ImageDataArray data;
  
  ImageDataColorSettings getColorSettings();
};

ImageBitmap

enum ImageBitmapPixelFormat { "default", "uint8" };

dictionary ImageBitmapOptions {
  ImageBitmapPixelFormat imagePixelFormat = "default";
};

convertToBlob

enum ImagePixelFormat {
    "uint8", // default
    "uint16",
};

dictionary ImageEncodeOptions {
    DOMString type = "image/png";
    unrestricted double quality = 1.0;
    ImagePixelFormat pixelFormat = "uint8";
};
                                                                                
[HTMLConstructor]
interface HTMLCanvasElement : HTMLElement
{
  Promise<Blob> convertToBlob(optional ImageEncodeOptions options);
};

[fserb]

cc @whatwg/canvas

[fserb]

Some clarification notes, that came up while talking to @annevk:

• when a float16 canvas is created, we default to extended-SRGB color space
• we need a defined behavior for putImageData of float16 to uint8: clamp from 0 to 1.
• there's no known primitive in web platform to specificy this float16 backing storage. We are up to suggestions of better alternatives.

[jrmuizel]

I haven't really followed all of the previous discussion about this kind of extension. Can you explain how this fits into the context of w3ctag/design-reviews#315, #299 and https://github.com/WICG/canvas-color-space/blob/master/CanvasColorSpaceProposal.md?

[fserb]

This is a summary of the IDL changes proposed by w3ctag/design-reviews#315.

This is also a subset of the original proposal, that included color spaces specification and that was discussed on #299. So those two are a superset of what's being proposed here.

[jrmuizel]

Can you provide the use cases that this solves along with the rationale for the decisions made relative to other options?

[fserb]

The high level goal is to allow canvases to take advantage of full color gamut available on display devices.
Example use cases:

• support applications that are wide gamut sensitive like medical imaging
• add a first step of support towards better photo editing / graphics on the web

A fully formed color space spec would include this (wide gamut) plus color spaces support (P3, rec2020, etc... and maybe a SRGB mode that doesn't do linear blending of non-linear values).

That said, it is our understanding that even that fully formed spec would need a wide-gamut mode that is backward compatible with the current SRGB support. This spec is this minimal color space support (extended SRGB) and wide gamut support on canvas and canvas export API.

[jrmuizel]

Do you envision the entire canvas pipeline running at float16 when CanvasPixelFormat is set to "float16"?

What is the functionality that those applications need from canvas? Naively, I can't think of much overlap between the canvas api and the needs of a photo editor.

[fserb]

I'm not 100% sure what you mean by "the entire canvas pipeline", but if you mean the backing storage for both 2D and WebGL and the drawing commands for 2D, then yes.

The main 2D commands that are impacted by this, as it stands now, are drawImage of wide gamut content (video, images, etc) and rendering of gradients.

In general, for any type of drawing (and some game) application color fidelity (and representing colors outside of sRGB) is a big issue.

Granted, the case strictly for 2D is a bit reduced, although we do have cases of people doing WebGL -> 2D canvas for interface, in which we would like to preserve the quality of the rendered image.

[jrmuizel]

CoreGraphics doesn't seem to support a 16bit float pixel format. Do you know what Apple's feedback on this proposal is?

[ccameron-chromium]

There was general consensus about this being the way forward at 1-year-ago's TPAC. There is definitely support in macOS for half-float IOSurfaces through the whole pipeline (I haven't locally tried in CoreGraphics myself)

[fserb]

(cc @smfr, I think)

[smfr]

CoreGraphics doesn't seem to support a 16bit float pixel format. Do you know what Apple's feedback on this proposal is?

CG does support half-float pixel formats on both its GPU-accelerated and non-accelerated code paths, so we support this proposal.

[annevk]

But what kind of backing store would you use that can be exposed to JavaScript? JavaScript doesn't have float16 so you'd have to do conversion at the boundary, no?

[kenrussell]

@annevk per the IDL above, Float32Array will be used to represent the pixels during getImageData and putImageData operations when the backing store is float16. These aren't the fastest way to use the canvas APIs anyway, so a conversion is fine. WebGL already uses Float32Array for uploading to float16 format textures and reading back float16 format framebuffers.

[svgeesus]

@fserb wrote:

when a float16 canvas is created, we default to extended-SRGB color space

Where is the extended sRGB space defined? What is the transfer function? What is the range of the function? In other words, what do values less than 0 or greater than 1.0 (255/255, in 8bits) mean?

[annevk]

@kenrussell what is the fastest way? And why don't we add Float16Array if we have several use cases?

[annevk]

By the way, related to @svgeesus's question, are we ensuring consistency with CSS and SVG somehow?

[kenrussell]

@annevk the fastest Canvas 2D APIs are the drawing operations (drawImage, rectangle/line/text drawing), not per-pixel manipulation (getImageData/putImageData).

Adding Float16Array to JS would require a large amount of hand-tuned assembly language to be added to every JS engine in existence. float16 is a native data type on GPUs but not on today's CPUs, and is handled with a library in every other language. I think it should be handled with a library in JS, too, and such libraries have already been written: https://github.com/petamoriken/float16 .

[svgeesus]

@annevk

By the way, related to @svgeesus's question, are we ensuring consistency with CSS and SVG somehow?

I plan to, yes; primarily by following/commenting on this and making sure that CSS Color 4 and canvas stay aligned wrt Wide Color Gamut and High Dynamic Range.

[annevk]

@kenrussell thanks, could you perhaps also address #4167 (comment)? And how would out-of-range values be handled, e.g., when going from float32 to float16?

@svgeesus that's great to hear. CSS Color 4 also seems like the ideal place to describe the WCG/HDR abstract model (the mapping of float16 values to colors, if you will).

[kenrussell]

@fserb is the best person to describe the extended sRGB color space; that's outside my domain.

float32 values can be defined in the spec as being converted to float16 internally with similar rules to how double (float64) is converted to float (float32) in C/C++: https://stackoverflow.com/questions/16737615/how-is-floating-point-conversion-actually-done-in-cdouble-to-float-or-float .

[annevk]

Let's close this in favor of #8708.