Allow 2dcontexts to use deeper color buffers

[grorg]

It is now common to come across displays that support more than 8 bits per channel. HTML Image elements can use these wider-gamut displays by embedding a color profile in the resource. The CSS Working Group is adding a way to specify the profile used in a color definition, and ensuring that colors are not clipped to 0-255.

That leaves Canvas objects. We need at least a few things in order to use better colors in canvas:

1. When calling getContext("2d", options) we need to specify in the options that we want a deeper backing store. This could be done a few ways (a boolean indicating 10bpp, explicitly saying the depth, or some keywords).
2. A way to ask an existing canvas object (or the context) how deep its backing store is.
3. Alternative forms of getImageData and putImageData that are not restricted to 0-255 bytes. For example, we might need a ImageDataFloat that uses floats in the 0-1 range, and get/setImageDataFloat methods.

All existing methods that take images or ImageData would still keep the existing behaviour. That is, if you create a deep canvas and putImageData into it, that data is assumed to be in sRGB.

[grorg]

If we did make an ImageDataFloat, it would be nice to use 16-bit floating point rather than 32. For this type of data we don't need 4 bytes.

[annevk]

We could perhaps overload ImageData rather than introduce a new object. And that if the user agent supports the "10bpp" option, .data returns a different view.

[heycam]

Am I right that a 16-bit floating point value has just enough precision for 10bpp images, but not higher depths? I wonder if instead we should just be using integers with a greater range. If there is concern that authors would forget to check for the specific channel depth, and only expect values in the range [0,1023] say, we could normalise the values to be always in the range [0,65535] or something.

Are there image formats where colours are really floating point values, where it would definitely make sense to use a floating point type to expose them?

[grorg]

I'm (we're) still not sure which is better:

A. Expose 32-bit floats for each channel and allow the implementation to squash them into however much precision the backing store provides. Pros: same code "works" on all backing stores. Cons: performance since everything needs to be converted + copied, memory use in the ImageData, and colors would not match on different profiles.

B. Same as A but with 16-bits. Cons: Same except slightly better memory wastage. Also, no real system support for half-float types.

C. Expose 32 or 16-bit integers for each channel and a way to detect how many bits are available (and thus the maximum value you can put in the data, 2^8 for 8bpc, 2^10 for 10bpc). Pros: Allows exact precision. Cons: still needs a copy + conversion when coming from a 10bpc backing store.

D. As Cameron suggests, 32 or 16-bit integers that are effectively normalized into a 0-1 range. Pros and Cons are similar to A and B.

E. Depending on the backing store type, expose the buffer as directly as possible. For example, if you have RGB101010 then each pixel is a 32-bit unsigned integer. Pros: Copying is easy. No conversion needed. Cons: Requires ugly bit math to use the data.

F. Something else.

I kind-of think option A, B or D are the nicest to use on the developer side. However, it will likely be very slow and is definitely wasteful. E is efficient but probably horrible to use.

[zcorpan]

Is it possible/practical to implement A/B/D on top of E in a JS lib?

What is the solution to specify wide gamut colors in CSS?

[grorg]

Exposing A/B/C/D is all possible on top of E. And E has the benefit of being the most performant, so I'm tempted to suggest that.

We'd still need some extra information on the result, such as the layout. For example, a wide gamut opaque buffer might be 10-10-10-2, and a non-opaque buffer might be the same with a chunk of 8s appended for the alpha.

As for CSS colors, see the thread http://www.w3.org/mid/5B15F03C-533C-4CBD-B266-4A499E230C18@apple.com

[annevk]

@grorg what is the backing store that OS X has?

JavaScript doesn't have 16-bit floating point arrays, so I think that's out. We don't want to introduce another TypedArray object unless TC39 is interested in that.

It sounds like 16-bit integers is a poor mapping for what is going on underneath?

E sounds tempting, if it's likely that another OS would have the same representation. Otherwise it seems like we might want to have at least one layer of abstraction.

[ojhunt]

So my opinion (as a former canvas implementer/spec writer-type person), and as an engineer on a javascript engine.

Don't bother with floats unless you're thinking in terms of dynamic range (and very important thing to think about these days). If we do believe floats are the way to go, don't mess around with halfs, CPU don't support them, so they will be slow -- all operations will require half->double expansion which does not have hardware support, followed by a subsequent double->half conversion on store, in addition to all the clamping you'd be expecting,

For integral sized backing stores, efficiency means running in multiples of 8bit. Also, I have encountered cameras that produce 12bpp RAW, so defining things in terms of 10bpp is just asking for many subsequent variants on the spec for 11,12,23... bit.

My recommendation is to provide an option for a 16bpp backing store, we can trivially support it, there are already defined Uint16Array types, there's no increase memory cost vs 8>bpp<16, graphics operations can be performed efficiently internally as CPUs have hardware to handle 16bit integers directly. It's also somewhat future proof vs. ever increasing channel depth.

The only time you need to worry about 16bpp vs 10bpp (etc) is when extracting the canvas content as an actual image, where it may be desirable to set the destination channel depth if the image format supports higher channel depths.

[kornelski]

I see it as two separate issues:

1. Supporting gamut wider than sRGB
2. Supporting precision higher than 8 bits

For 1 you technically don't need > 8bbp. Most image formats use a color profile and "stretch" the 8 bits of image data to the destination gamut. That's simple and convenient.

For the gamut case I'd suggest API that takes a color profile from an image: createContext('2d', {useSameColorProfileAsThisImage: 'http://example.com/whatever-image-with-my-profile-embedded.jpeg'});. I suggest getting profile out of an image to make it simple for authors to create such file. Exporting a raw ICC profile is tricker. edit: actually that wouldn't fly, since getContext is synchronous and it'd be too weird to have profile change dynamically

Pre-defined names like "adobe rgb 1998" might work too, although then it'd be impossible to support custom profles from digital cameras.
As for 2:

• For sanity, pixels must be multiples of byte size, so please don't do actual 10-bit per gun pixels.
• 30-bit RGB pixels padded to 32bit are fine, but slightly inconvenient as they require bit-fiddling.
• 4x 32-bit float pixels with intensities from 0...1.0 are convenient, especially when using SIMD. Make sure to define gamma and how out-of-range values are handled (I'd prefer clamped).

[ojhunt]

I consider both of those good reasons for float (I had previously considered the possibility of float per channel). Just note that it will be substantially slower than integer paths.

[grorg]

@ojhunt said:

there's no increase memory cost vs 8>bpp<16

That's not quite true. We'll probably expose a buffer that is 10bpp, which can fit into a 32-bit integer when opaque. i.e. won't be any additional memory cost over 8bpp, but will be less than 16bpp.

[grorg]

@pornel said:

Supporting gamut higher than sRGB

For 1 you technically don't need > 8bbp. Most image formats use a color profile and "stretch" the 8 bits of image data to the destination gamut. That's simple and convenient.

Yep. This is what we already do with our wider gamut hardware. We just can't create a canvas with a profile.

For the gamut case I'd suggest API that takes a color profile from an image: createContext('2d', {useSameColorProfileAsThisImage: 'http://example.com/whatever-image-with-my-profile-embedded.jpeg'});. I suggest getting profile out of an image to make it simple for authors to create such file. Exporting a raw ICC profile is tricker. Pre-defined names like "adobe rgb 1998" might work too, although then it'd be impossible to support custom profles from digital cameras.

Yes, this is what I was proposing, although I think we can start with just well-known names. On the CSS discussion (linked above) I suggested P3 and Rec.2020, although maybe just Rec.2020 is enough.

However, I think that you are going to want the larger backing store, which I'll reply to separately.

[grorg]

@pornel said:

For sanity, pixels must be multiples of byte size, so please don't do actual 10-bit per gun pixels.

Whose sanity? The developer? I sort-of agree, but there is a big advantage to exposing the same format as the backing store is that the get and put operations don't require any conversion and will be fast. Yes, you'll have to do some ugly bitmasking and shifting, but the users of these APIs are probably already doing some nasty things for every pixel.

30-bit RGB pixels padded to 32bit are fine, but slightly inconvenient as they require bit-fiddling.

Yep.

4x 32-bit float pixels with intensities from 0...1.0 are convenient, especially when using SIMD. Make sure to define gamma and how out-of-range values are handled (I'd prefer clamped).

The issue here is that you're probably exploding the memory use by a factor of 3 or 4, and requiring a conversion for every channel of every pixel when reading and writing, and losing some precision in the process.

On the positive side, it means you can write code that will work with any depth of backing store, including the current RGBA8.

Another positive is that this form can be polyfilled from the direct-access method. i.e. if I gave you the raw 30-bit RGB + 8-bit Alpha buffers, you could provide a 32-bit floating access in JS.

Note about the lack of half float: while the CPU doesn't support it, we could use unsigned short and normalize values from 0-2^16 into 0-1. There would still be some memory wastage (if the backing store is 10bpc) and conversion.

[grorg]

@grorg said:

We'll probably expose a buffer that is 10bpp

I meant 10bpc (channel). That is a 32-bit value that is 10bits of R, G and B.