10 bit YUV formats / accessing pixel data

[paulygreen]

Hi,

I have been experimenting with webcodecs to decode h264 video and generally things have been easy. One issue I have run into is when decoding 10bit videos, the returned VideoFrame has a null format field (I'm assuming that's because all the video formats listed are 8bit). I can pass the decoded videoFrame to texImage2D() and render in GL but the Yuv to RGB color conversion (which was done magically by texImage2D to my surprise) seems to be a little off (like its assuming the wrong bt601 color space instead of bt709, or maybe full range data, not really sure). Unfortunately I can't figure out how to access the raw YUV bytes directly from the VideoFrame since CopyTo() doesnt work if format is null.. Is there some way to get access to the underlying yuv bytes so I can do the color conversion in a shader myself? or any thought on adding more formats for 10bit ? or ways to supply the colorspace info for proper yuv2rgb conversion?

thanks!
Paul

[dalecurtis]

This is #384 (comment) - adding high bit depth (HBD) pixel types is a work in progress. Once we have a PR for adding the pixel types (just needs to be sent) it shouldn't be hard to add for software decoded cases. Hardware decoded cases will take some time since we don't have HBD readback paths and HBD canvas is a work in progress: whatwg/html#299

[paulygreen]

Ok thanks.. So to be clear.. there is currently no way for me to access the pixel buffers right now (ie while format is returned null)? (I'm assuming copyTo will work fine once 384 is done and format is no longer null.. but until then I just have to live with slightly wrong color conversion?). I'm asking because teximage2d etc seem to be able to read the data, so I was wondering if Im just not seeing some API call, or not supplying exactly the correct params to a call, etc..

[dalecurtis]

Kind of. In Chrome, once #384 is resolved you'll be able to use copyTo() with all software decoded cases. However hardware decoded cases may take some time or will undergo a conversion to 8-bit.

Are you configuring the VideoColorSpace on the decoder? If so, the color conversion issue seems like a bug, so please file at https://crbug.com/new with a reproduction case and we can take a look on the Chrome side.

[paulygreen]

yeah I added a colorSpace to the VideoDecoderConfig, but it seems to have no effect on the final rgb image (I have tried changing the primaries, transfer func, matrix, etc bjut the resulting image is always the same). I attributed that to the fact that the decoded VideoFrame always has a null colorspace. I can try to file an issue if that sounds like a bug.

[dalecurtis]

It could be a bug, please file one with the reproduction details and I'll take a closer look.

[jchen10]

You could also try WebGPU rather than WebGL. It supports 0-copy import and color space conversion. For now you still need the "--enable-unsafe-webgpu --disable-dawn-features=disallow_unsafe_apis" cmdline options to enable the WebGPU support. Chrome will officially release WebGPU soon. You can also read the sample for reference.

[paulygreen]

Thanks for the suggestion. Im not too familiar with WebGPU and I have some other WebGL shaders that I need to run, so I'll have to look into it more.

[dalecurtis]

@jchen10 does WebGPU support 10-bit textures already?

[jchen10]

I think it should be working now, although there might be one more copies required. The 0-copy import path is not fully supported yet. But it won't be too hard.

[kedanielwu]

I encountered a similar issue, but with H.265 video with yuv420p10le pixel format. I tried WebGPU approach with VideoFrame as external texture, but I don't think current setup provides what I need.

however, in my case, what I need is not canvas/webgpu correctly render the image, rather I want unclampped 10 bit rgb or original YUV data and let me do the YUVtoRGB conversion and SDR tone-mapping myself, since the HDR video includes dynamic metadata like HDR10+ and DolbyVision, and it not likely be supported natively by the user-agent.

[kedanielwu]

since it is H.265 video and I'm using Chrome, i guess it falls into what you guys call HW decoded case?

[BryanChrisBrown]

anyone know if there's a way to encode YUV 4:2:2? Is it just a specific HEVC encoder level I'd need to use?

[dalecurtis]

At least in Chrome, I think only 4:4:4 encoding w/ software VP9 or AV1 is possible:

• https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/web_tests/external/wpt/webcodecs/full-cycle-test.https.any.js;l=51;drc=e568e644898401cdefe6d3901a2f8fab8ef533af
• https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/web_tests/external/wpt/webcodecs/full-cycle-test.https.any.js;l=28;drc=e568e644898401cdefe6d3901a2f8fab8ef533af

[BryanChrisBrown]

thanks for the response here, I see those tests are mostly working with software encoding, I was hopeful that if using hardware accelerated encoding there'd be a way to pass it a proper config, but figuring out exactly which hevc encoder encodes 4:2:2 has been quite a difficult topic to search for.