Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Basis Universal and IBL with KTX2 images #1612

Closed
wants to merge 7 commits into from
Closed

Conversation

lexaknyazev
Copy link
Member

@lexaknyazev lexaknyazev commented May 16, 2019

Here's a set of work-in-progress extensions that will provide high-quality IBL with GPU-compressed textures.

Glossary

KTX2 - An image container file format optimized for GPU uploads. The spec is available here.

Basis Universal - Texture compression technology developed by Binomial. It enables JPEG-sized textures that can be transcoded on-the-fly to natively supported compressed GPU formats.

ASTC HDR - Block-compressed floating point GPU image format. Supported by many mobile devices and some desktop platforms.

BC6H - Block-compressed floating point GPU image format. Supported by many desktop and console platforms.

Proposed extensions

KHR_image_ktx2

Defined on glTF image, provides JSON form of the KTX header for better workflow on the web.
Does not restrict allowed KTX2 features / formats.

KHR_texture_basisu (requires KHR_image_ktx2)

Defined on glTF texture, similar to WebP extension with optional fallback.
This allows usage of Basis Universal images for IBL (normal quality) and material textures (albedo, roughness, etc).

EXT_texture_bc6h and EXT_texture_astc_hdr (require KHR_image_ktx2)

Defined on glTF texture. The same texture object should have KHR_texture_basisu extension object for better compatibility.

KHR_lights_image_based (requires KHR_image_ktx2 and KHR_texture_basisu)

Defines image-based lights on scene.

@UX3D-nopper
Copy link
Contributor

As you already created a pull request, can you please add the diffuseEnvironmentMap as discussed yesterday? Or do you want me to make the changes in your branch?
You can gather the information from my previous mails.

@lexaknyazev
Copy link
Member Author

lexaknyazev commented May 16, 2019

Sure. Should I simultaneously remove SH? I don't think we have decided on a final form:

  • diffuseMap + SH;
  • diffuseMap + optionally SH;
  • SH + optionally diffuseMap.

@UX3D-nopper
Copy link
Contributor

UX3D-nopper commented May 16, 2019

Hmm, thought we agreed on supporting both. But it is an either or. So do not kick out SH. But either SH or the diffuseMap has to be present.

@lexaknyazev
Copy link
Member Author

Thanks, I'll update the PR.

@lexaknyazev
Copy link
Member Author

lexaknyazev commented May 16, 2019

@UX3D-nopper
Done.

A few actionable items that should be addressed before marking this as ready for trial implementation.

  • Does rotation affect both maps? Yes.
  • Should we disallow KHR_texture_transform on IBL textures? Yes
  • Does the IBL coordinate system match glTF default (Y-up)? Yes
  • Provide complete sampling math for SH, as the ratified extension cannot refer to external sources.
  • Verify that linear transfer function doesn't introduce banding with normal quality diffuse maps.
  • Decide whether an asset can provide both SH and a diffuse map so that engines can decide at runtime.


### Normal Quality

Data must be stored as a CTTF image with alpha channel conforming to RGBD (or RGBE, **TBD**) encoding.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm assuming that we're suggesting CTTF here so that the texture can remain compressed in memory. However, block-based compression schemes really don't work well with RGBE because they don't take into account that the alpha is being used as an exponent. The result is usually a lot of ugly banding between exponent boundaries. I don't remember how RGBD behaves when block-based compression is used but I expect it would be better. We should do some tests.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RGBD, RGBE or RGBM should be decoded on CPU before uploaded to GPU. So, in general, not the perfect format but defined as mostly used.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm assuming that we're suggesting CTTF here so that the texture can remain compressed in memory.

Regardless of an effective GPU format, CTTF should give us better transmission bandwidth than typical web formats.

RGBD, RGBE or RGBM should be decoded on CPU before uploaded to GPU.

CPU decoding of RGBD (RGBE) from CTTF is expected to be supported.

@UX3D-nopper
Copy link
Contributor

If we want to achieve consistency, we should only allow either SH or difusse maps in a scene.
E.g. if in some case SH is sufficient, one can use SH. If Sh is not sufficient, the a diffuse map is provided.
But this decision is made by the content creator and the rendering engine should just use what provided.

@UX3D-nopper
Copy link
Contributor

Agree regarding "Provide complete sampling math for SH".
@bghgary Can your team please gather this information?

@donmccurdy donmccurdy changed the title IBL with KTX2 images CTTF and IBL with KTX2 images May 22, 2019
@bghgary
Copy link
Contributor

bghgary commented May 24, 2019

From @sebavan

So our implementation is here:
https://github.com/BabylonJS/Babylon.js/blob/master/src/Maths/sphericalPolynomial.ts

Containing all the various references in the header comment.

It is called for each pixel of the cubemap here:
https://github.com/BabylonJS/Babylon.js/blob/master/src/Misc/HighDynamicRange/cubemapToSphericalPolynomial.ts

And applied by the shader here:
https://github.com/BabylonJS/Babylon.js/blob/master/src/Shaders/ShadersInclude/harmonicsFunctions.fx

The comments in the code include all the math and relevant references.

extensions/2.0/Khronos/KHR_image_ktx2/README.md Outdated Show resolved Hide resolved
extensions/2.0/Khronos/KHR_image_ktx2/README.md Outdated Show resolved Hide resolved
extensions/2.0/Khronos/KHR_image_ktx2/README.md Outdated Show resolved Hide resolved

## Defining an Image-Based Light

The `KHR_lights_image_based` extension defines a array of image-based lights at the root of the glTF and then each scene can reference one. Each image-based light definition consists of a single cubemap that describes the specular radiance of the scene, the l=2 spherical harmonics coefficients or another cubemap for diffuse irradiance, and also rotation, brighness factor, and offset values.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be helpful to call out that the specular radiance cubemap has prefiltered data in its mips and that the diffuse irradiance map has only one mip?

```json
"extensions": {
"KHR_lights_image_based" : {
"imageBasedLights": [
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the vendor extension, we had a list of images and the expected image size. This allows the possibly to drop a few mip levels at the end of the mip chain. Is this possible with this extension?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KTX2 allows random access to mip levels, so loaders can do whatever they see fit.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, but it needs to be specified here how to interpret the mip levels or we will get inconsistent behavior?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For simplicity, I would recommend to just use the full mip map chain. What were Babylon.JS reasons, to have this feature?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

extensions/2.0/Khronos/KHR_lights_image_based/README.md Outdated Show resolved Hide resolved
{
"rotation": [0, 0, 0, 1],
"brightnessFactor": 1.0,
"brightnessOffset": 0.0,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is brightnessOffset? Is this supposed to be a color? If so, why do we want this?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

brightnessOffset is for shifting the color range more into the darker or brighter area. E.g. required if the HDR maps are RGBM encoded and to support negative values.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Below, this section says:

finalSampledColor = sampledColor * brightnessFactor + brightnessOffset;

brightnessOffset is currently a number. Is it supposed to be an offset per component?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

finalSampledColor = sampledColor * vec3(brightnessFactor) + vec3(brightnessOffset);
Maybe more obvious like this.

@zeux
Copy link
Contributor

zeux commented Jun 3, 2019

In KHR_image_ktx2, is the JSON image header optional? In terms of ease of integration into existing pipelines and maintaining the simplicity of the baseline format it seems like it's better to follow something like MSFT_texture_dds, that just allows specification of .dds path without requiring producers of glTF files to parse KTX files and emit duplicate header information.

@lexaknyazev
Copy link
Member Author

@zeux
The JSON form of the binary KTX2 header is mandatory. Just like buffer views and accessors, it gives the information required to allocate GPU resources before actual texture data comes in.

@donmccurdy
Copy link
Contributor

@lexaknyazev to my understanding a .basis file contains much of the same header information that the KTX2 header will offer, so with the JSON form we're specifying that in triplicate. I'm still not sure I understand the need for the extra header. In this and other threads, we've offered reasons such as:

  • allocate GPU resources before actual texture data comes in.
  • check compatibility before loading a texture
  • reduce the amount of network requests in case of partial data fetching

At least for the use case of Basis textures referenced by materials, none of these reasons are self-explanatory to me... when might a Basis file be incompatible in a way that can only be known from the header? What implementations preallocate GPU resources this way?

I'm a bit worried this is extra data that implementations will not actually use, and it has the side effect that users cannot easily swap out textures without modifying the JSON. It would be helpful to have a clear and robust example of why the extra data is necessary.

@donmccurdy
Copy link
Contributor

Maybe another way of putting this – why are we using a KTX2 wrapper if we're going to replicate everything that wrapper provides?

@lexaknyazev
Copy link
Member Author

.basis file contains much of the same header information that the KTX2 header will offer, so with the JSON form we're specifying that in triplicate.

We're not going to use .basis file format wrapper, KTX2 will directly embed codebooks and slices. Compression tools will be able to directly output .ktx2 files with Basis payloads.

Basis textures referenced by materials

Mip-levels cannot be generated online for compressed textures, so they have to be provided by an application. Without JSON form, apps would have to make an additional request (or two) to download texture file header to be able to progressively stream mip levels.

I'm a bit worried this is extra data that implementations will not actually use

JSON copy of the header makes glTF clients simpler since they would be able to skip parsing the binary header and pull texture data directly.

users cannot easily swap out textures without modifying the JSON

Putting extra burden on exporters is aligned with other glTF choices. Geometry cannot be easily swapped out either.

@donmccurdy
Copy link
Contributor

We're not going to use .basis file format wrapper, KTX2 will directly embed codebooks and slices. Compression tools will be able to directly output .ktx2 files with Basis payloads.

Ok, that's good thanks!

JSON copy of the header makes glTF clients simpler ...

This seems like a very minor simplification, and in my opinion should be considered a non-argument compared to more important pros (fewer requests) and cons (confusing user workflows, redundant data).

Without JSON form, apps would have to make an additional request (or two) to download texture file header to be able to progressively stream mip levels.

We discussed a number of concerns about this on our call today, namely – the data is redundant, and common workflows today involve unpacking a .glb, editing the textures, and repacking the .glb. Users will certainly be surprised to learn they can't swap out .ktx2 files referenced by an unpacked .gltf like they would with .png or .jpeg; that is a guaranteed cost we'd be paying with this design.

We discussed an alternative – that the header information could be optional, or could be provided by a separate extension, such that it can be omitted until/unless tools exist that are actually progressively streaming mip levels.

I am still open to that alternative, but an even better outcome would be this: can we get at least one common engine to implement support with streaming before ratification? The scenario I strongly want to avoid here is one where we've complicated the format for hypothetical streaming support, but later find that no one has actually implemented the perceived benefit, or that other features are actually necessary for that benefit to be practical. If we knew, for certain, that streaming textures will be implemented without requiring additional extensions, I'd be much more open to accepting the known cost of end-user confusion when externally referenced .ktx2 files are not interchangeable.

@lexaknyazev
Copy link
Member Author

lexaknyazev commented Jun 6, 2019

The size of the KTX2 header index fields is:

80 + 24 * levelCount

For 16384*16384 textures (currently the second most common max texture size), the levelCount is less than or equal to 15. This makes the index part of the header less than 500 bytes. It's almost certain that the first partial resource fetch would include more than 500 bytes, so an engine would have all the information needed to access mip-levels independently.

This already unlocks streaming at the cost of one extra request compared to having all indices in JSON.

@atteneder
Copy link
Contributor

I have a working prototype implementing parts of this PR. Still very raw, but a first step:
grafik
Unity package: https://github.com/atteneder/glTFast/tree/ktx

I tested it on 4 models DGG3D kindly provided to me.

Now I'd like to encode more glTFs to do more testing/stabilizing, but I could not get glTF-Compressonator to build yet. Asking for support did not yield in a response so far.
Is there a way to encode glTFs with KTX/BasisUniversal textures? maybe @UX3D-nopper can tell?

Thanks!

@UX3D-nopper
Copy link
Contributor

When we took over the Compressonator project, it did not build for us on Linux and/or Mac either.
At the moment, it only builds and runs on Windows.

If you need the IBL images, please use this tool:
https://github.com/KhronosGroup/glTF-IBL-Sampler

@zeux
Copy link
Contributor

zeux commented Nov 18, 2019

Does anybody know if any web glTF viewer implements KHR_image_ktx2 (possibly in some development branch)? I'd like to add support to this to gltfpack but not sure how to test the resulting files.

@atteneder
Copy link
Contributor

Does anybody know if any web glTF viewer implements KHR_image_ktx2 (possibly in some development branch)? I'd like to add support to this to gltfpack but not sure how to test the resulting files.

I could provide a Unity project + WebGL build with an URL input field. Current alpha state can load glTF (non-binary; separate files) with KTX textures (no levels, no mipmaps yet). Should take me a day or two to.

I'm much interested in test files and ktx support in gltfpack on the other hand. Would be awesome if you share your results!


JSON type of each metadata entry depends on the type of the original value. The JSON-Schema provides types for metadata keys defined in the KTX2 specification.

This extension does not restrict texel formats or other features of the KTX2 images. Such limitations must be defined by other glTF extensions referring to this extension.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this necessary when the extension is marked as required? (I understand that it's possible to replace references to KTX2 images in "texture" object with references through extensions, but I'm not sure I understand why this is better than simply using an image/ktx2 MIME type on the image objects that are referenced directly)

Copy link
Contributor

@donmccurdy donmccurdy Nov 20, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding is that KHR_image_ktx2 may be dropped entirely, and that the other extensions in this PR would simply independently allow KTX2 images. @lexaknyazev @bghgary is that a correct understanding of #1705 (comment)? This was confusing me as well.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(quick clarification, the comment got misplaced and refers to line 65 - implementation note - of the spec, not line 63)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KHR_image_ktx2 may be dropped entirely

This was my suggestion after attempting to implement it in Babylon. Unless we need some other metadata that goes with the extension, it will likely end up with a no-op implementation.


The `KHR_texture_basisu` extension is added to the `textures` node and specifies a `source` property that points to the index of the `images` node which defines a reference to the KTX2 image with Basis Universal supercompression.

The following glTF will load `image.ktx2` in clients that support this extension, and otherwise fall back to `image.png`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(just clarifying) I am assuming that in absence of "source" property on the original texture JSON element, the loaders that support the extension must use the image specified through the extension metadata?

@zeux
Copy link
Contributor

zeux commented Nov 26, 2019

I've implemented support for these extensions in gltfpack: zeux/meshoptimizer#81. To build this yourself, make sure you have basisu in your path (you can build & install it from the official repository: https://github.com/BinomialLLC/basis_universal/). To compress textures with Basis & KTX2, command line flag -tc is required; gltfpack will automatically run basisu when this flag is specified.

Here's an example - it's the Corset model from glTF-Sample-Models, converted to .glb with embedded KTX2 images:
corset.glb.zip

Note that the code is somewhat untested; please let me know if there are any issues with this! It's hard to test this without renderer support...

@MarkCallow
Copy link
Collaborator

@zeux, what software can I use to view corset.glb? Firefox (on macOS) doesn't seem to understand .glb files. I tried Windows' 3D Viewer but it says "couldn't load model, please try again later."

@zeux
Copy link
Contributor

zeux commented Nov 26, 2019

@MarkCallow This file requires support for KHR_texture_basisu, KHR_image_ktx2 and KHR_mesh_quantization so I would not expect existing renderers to open it, but it could be used to validate support for the extensions from this PR in some work-in-progress renderer...

@mlimper
Copy link
Contributor

mlimper commented Nov 27, 2019

I tested it on 4 models DGG3D kindly provided to me.

Now I'd like to encode more glTFs to do more testing/stabilizing, but I could not get glTF-Compressonator to build yet. Asking for support did not yield in a response so far.
Is there a way to encode glTFs with KTX/BasisUniversal textures? maybe @UX3D-nopper can tell?

hehe happy that the models were helpful

we have an experimental KTX2 encoder branch, it was tested with a special (also experimental) BabylonJS branch for our joint demos for the BOF at SIGGRAPH

the main reason it is called "experimental" is because the spec is still in draft state

would it help if our devs run it on a couple of samples?

@atteneder
Copy link
Contributor

Here's an example - it's the Corset model from glTF-Sample-Models, converted to .glb with embedded KTX2 images:
corset.glb.zip

Note that the code is somewhat untested; please let me know if there are any issues with this! It's hard to test this without renderer support...

I tried to load this glb in my prototype but I get an KTX_FILE_UNEXPECTED_EOF (using KTX-Software). I also extracted the first ktx file/glTF buffer view. Running ktxinfo on it works, but loading it (in the glloadertests executable) fails as well.

@MarkCallow
Copy link
Collaborator

@atteneder please provide more details and, if it looks like a bug in KTX-Software, open an issue over there. How can I extract the ktx2 file(s) from the .glb? I tried opening with 7-zip but it says it can't open it. I also tried The Unarchiver on macOS. Be aware that ktxinfo just looks at the file header. I'd suggest trying ktx2check but I know it will fail as I haven't yet updated it for the latest KTX2 spec. draft and I know @zeux's files are compliant to that draft. However loading should still work, providing the file doesn't have other issues.

@zeux
Copy link
Contributor

zeux commented Nov 29, 2019

I'll investigate this, since this implementation is written from scratch this is probably a bug on my end.

Also attaching a .zip file where texture files aren't embedded into .glb for easier analysis.
corset-separate.zip

@zeux
Copy link
Contributor

zeux commented Nov 29, 2019

Ah, the mip levels are written from smallest to largest but the mip level index must be written from largest to smallest :-/ ktxcheck doesn't catch this, but loading the entire file does.

After fixing this, the only other change that seems to be needed to get libktx to load these files is to align the last mip by 8 bytes. I'm not sure why this is necessary - the spec seems to explicitly call out that last mip doesn't need to be padded? - but libktx texture2.c has the following code that necessitates this:

    // Calculate size of the image data.
    This->dataSize = 0;
    for (ktx_uint32_t i = 0; i < This->numLevels; i++) {
        This->dataSize += _KTX_PAD8(private->_levelIndex[i].byteLength);
    }

@zeux
Copy link
Contributor

zeux commented Nov 29, 2019

Here's a new version; for easier testing, it contains the .glb (with embedded textures) as well as .gltf (with separate .bin + texture files)

corset-fixed.zip

@atteneder
Copy link
Contributor

Here's a new version; for easier testing, it contains the .glb (with embedded textures) as well as .gltf (with separate .bin + texture files)

corset-fixed.zip

glb works smooth now! found some remaining bugs on my side though.
thanks for the super quick answer!

@MarkCallow
Copy link
Collaborator

ktxcheck doesn't catch this

Please file an issue at KTX-Software.

@ShantanuSriv
Copy link

Here's a new version; for easier testing, it contains the .glb (with embedded textures) as well as .gltf (with separate .bin + texture files)

corset-fixed.zip

how do you generate glb file? I am able to create glTF + bin + basis texture files but unable to pack them into one glb file using gltf-pipeline. Which tool are you using to output glb file?

@zeux
Copy link
Contributor

zeux commented Dec 9, 2019

@ShantanuSriv These files are produced by gltfpack (https://github.com/zeux/meshoptimizer#gltfpack); -tb produces files with Basis container and -tc produces files with KTX2 container format which is what I posted above. -te embeds textures into .bin or .glb. Note that for KTX2 files you should use master version since the last point release contained a bug with mipmap layout.

@donmccurdy donmccurdy added this to the Compressed Textures milestone Jan 27, 2020
@donmccurdy
Copy link
Contributor

Per discussion earlier, we are planning to:

  1. Discontinue work on the KHR_image_ktx2 extension: it isn't necessary. Other extensions Like KHR_texture_basisu can independently allow use of KTX2 files.
  2. Close this PR for now.
  3. Open a new PR to resolve remaining questions on KHR_texture_basisu.
  4. Revisit the IBL extensions at some point in the future.

@lexaknyazev or @bghgary would you be able to open a new issue for (3)?

@lexaknyazev
Copy link
Member Author

The story continues in #1751.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.