KHR_displayencoding

[rsahlin]

AKA - "The calibration opto-electrical transfer function" or "The certification display encoding"

The purpose of this extension is to enable consistent material representation under highly varying lighting conditions.
At the same time retaining color representation (hue) - in usecases that mix models and light setup from multiple glTF sources.
Priority is to retain the spectral properties of the material and incoming lights - output shall be neutral, unfiltered and raw.
Meaning that filtering, tonemapping, colorgrading or other type of post-processing effects are not part of this extension.
Neither is any kind of operations or processing with the goal of simulating how humans interpret the visual input, the goal is to output physically correct raw or unadjusted color values.

The following requirements shall be met:

• Display output shall be unadjusted,unfiltered and raw - there shall be no visual or artistic intent or interpretation applied.
  Output value shall retain the spectral properties of the material and those of the lights.
• Enable interoperability between multiple glTF assets, where either may contain one or more lights.
• Enable consistent light setup values, provide a common understanding of what X lumens / m2 means.
• Display output shall be physically correct when light values are dynamically changed, this includes adding, removing or changing the intensity of lights. There shall be no colorshift or white-out.
• Globally enable extension by including extension name in asset - no parameters shall be used.
• Shall work on 'light setup' assets, ie assets that only includes light information and no geometry.
• Utilize precision on 8 - 12 bit displays.
• Can be used in glTF assets as well as glXF assets.

It is assumed that the light contribution in any given glTF scene can vary from low values to very high values.
These, internal, values must then be matched to a range that is acceptable by displays.
This is usually in the [0.0 - 1.0] range at a precision of 8 bits, with HDR displays increasing the precision to 10 or 12 bits.

This extension is compatible with both HDR and SDR displays.
The intended usecases for this extension is any usecase where the light contribution values will go above 1.0, for instance by using KHR_lights_punctual, KHR_emissive_strength or KHR_environment_lights.

It defines the output range - which is unspecified in the glTF spec but generally treated as: 1.0 equals a fully exposed pixel
The output is now defined as a display, with target of interactive framrates but compatible with pathtracers, with the maximum output value of 10000.
Making this extension suitable to be used as a calibration display encoding function extension with the purpose of retaining color values (hue) as ouput from the glTF BRDF.

The choice of perceptual quantizer is to make this extension future proof and compatible with the existing HDR standards.
Hdr monitors are already on the market and support will only grow, for instance fully embraced by Windows 10/11 and MacOS.
This makes sure the extension is future proof.

Sample Viewer implementation - drag and drop you models as usual.
Go into "Advanced controls" and enable "Force Displaymapping PQ"
The "Compensatory light intensity" is one directional light with the intensity in LUX (lumen / m2) as defined by KHR_lights_punctual.

KhronosGroup/glTF-Sample-Viewer#391

Babylon playground 'hack' - note that this is not fully implemented - light values must be kept within 0 - 10 000 lumen / m2, don't mix with IBLs as they will look totally wrong.

https://playground.babylonjs.com/#AHVICB#12

[bhouston]

I see a lot of value in automatic exposure control in order to avoid over bright areas or underbought areas. Overbright areas tend to have color shift because of the per-channel per-pixel tone mapping operators in use.

I think that we should expect viewers to just use the proper color space for their displays. I am not sure we should put into a 3D file what its expect transfer function to that display is.

I guess we could put exposures into individual glTF files but that feels a little weird. What happens if you open up multiple glTF files that have conflicting exposures or tone mapping parameters?

My recommendation may be that this is a viewer certification issue and we come up with guidelines for handling automatic exposure, tone mapping and transfer functions. Tone mapping and automatic exposure controls and proper transfer functions should be part of viewers, rather than part of the individual glTF files.

I would love to see that happen!

Because to put it into a glTF file we are sort of assuming that viewers are going to be using incorrect, less than optimal OOTF, unless you use this extension. That is a bit weird.

[rsahlin]

To solve the questions of how to handle multiple loaded glTFs with or without this extension:

• Remove the option of specifying rangeExtension and gamma for the OOTF - they will always be the default values.
• The PQ (HDR or SDR) is enabled for rendering in a model that includes this extension in the root element
• Move the sceneAperture setting to the scene object - this makes it possible to set sceneAperture control on a scene level.
  If no sceneAperture is explicitly set then no adjustment is done to light contribution before the PQ step.
  This also makes it natural to assume that any model that is brought into this scene will be displayed using the same sceneAperture values (as the new model is inserted into an existing scene)

[romainguy]

Note that PQ is used in game engines as the transfer function for HDR output (as dictated by HDR10 and Dolby Vision), it is not used as the "tone mapping" curve (that bit is done by the "S curve" in the Lumberyard diagram for instance). PQ is also sometimes used as a log encoding for 3D LUT lookup.

[hybridherbst]

One are where this might be considered too prescriptive is that the output for HDR and SDR screens is still subject to a W3C Draft and not finalized, including a number of open issues.

Adding the entire chain until PQ-encoded data is sent through the wire to a display as part of this extension will, if I'm not understanding these proposals wrong, for the foreseeable future not be possible in web viewers.
It could also happen that a viewer ends up implementing HDR support (for example, converting floating point Rec709 to Rec2020 and encoding with PQ to send to a HDR10 display seems to be what Unity is doing in HDR mode) and that this ends up "double-encoding" with PQ, effectively breaking any intended rendering effects. Am I correct to assume that in this case, the extension must not be used? Or would you actually propose double-encoding with PQ in this case (where the viewer doesn't have control over the following technical pipeline towards an output device)?

[hybridherbst]

Side note, when following your steps (open "http://gltf.ux3d.io/", enable "Force Displaymapping PQ" in Advanced Settings) results in a non-clipped image that nonetheless looks bad:

Would this be your intended result for this file?

[rsahlin]

Adding the entire chain until PQ-encoded data is sent through the wire to a display as part of this extension will, if I'm not understanding these proposals wrong, for the foreseeable future not be possible in web viewers.

Sure, and then web-viewers will use the SDR target until they can get access to swapchain format and colorspaces.
I don't see how this would be a problem.

It could also happen that a viewer ends up implementing HDR support (for example, converting floating point Rec709 to Rec2020 and encoding with PQ to send to a HDR10 display seems to be what Unity is doing in HDR mode) and that this ends up "double-encoding" with PQ, effectively breaking any intended rendering effects.

Then it seems that they have already implemented the HDR part of this extension.
Without knowing exactly how they have implemented the HDR support I would guess that they just need to adapt to using the OOTF + OETF when this extension is used.

Why would they do "double-encoding"?

[rsahlin]

Would this be your intended result for this file?

Absolutely not, this seems to be a bug in the Sample-Viewer use of the transmission or volume extension.

Try without the displaymapping extension enabled, not using 'image based' and change the "compensatory light intensity" - you will notice that nothing changes.
I guess the reason being light values are clipped somewhere - never going above 1.

[rsahlin]

I think ultimately KHR_image_formation and KHR_displaymapping_pq in their current forms have the same goal: to specify a scaling factor and a neutral range mapping to make sure assets display perceptually similar in different viewers under different lighting conditions, instead of the current situation where this is entirely up to viewers. Please @elalish, @romainguy and @rsahlin correct me if I'm already wrong in that assumption.

This assumption is wrong.

KHR_displaymapping_pq - is designed to be compatible with a number of usecases that mix several glTF models, can have varied and changing light conditions and at the same time render with expected result (to both SDR and HDR displays)
Metaverse, AR, 'Furnish the home' and 3D Commerce.
This is why this extension has no configurable parameters and avoids doing any model or scene based filtering

KHR_image_formation has one usecase and that is the 'Main product picture' type of usecase - showing one model under fixed light conditions and at the same time applying color-grading (a filter)
It is not compatible with the other usecases.

[rsahlin]

Also note that this extension here could be implemented by #2128 with the appropriate LUT.

Not while retaining compatibility with different usecases such as Metavers, AR, 'Furnish the home' or 3D Commerce.
It will only work for a very limited use of the 'Main product picture' usecase - you would not even be able to change the light setup without breaking the look.
That extension as it is simply not an alternative to KHR_displaymapping_pq

I have already proposed to change that extension into one that defines a physical camera with aperture/shutter/ISO - nothing else - to be used together with this extension.

I think that is the best way forward.