KTX format

[lgritz]

@Silverlan requested (on a different ticket):

Support for the KTX-format would be great!
https://www.khronos.org/opengles/sdk/tools/KTX/file_format_spec/

[atteneder]

Updated, correct link:

https://www.khronos.org/ktx

In April 2021 KTX 2.0 was released. Its support for Basis Universal supercompression makes it very appealing for many real-time graphics use-cases. One of particular interest is support for it in Blender (uses OIIO) to import/export glTF 3D assets with KTX textures.

Thanks

[donmccurdy]

Hi! I'm a maintainer of https://threejs.org/ and wanted to voice support here. We view KTX2 as a more modern and portable GPU texture container format, compared to older formats like DDS. These types of formats are critical for realtime 3D applications. KTX2 has several internal formats with different benefits that complement those currently available in OpenImageIO:

• Basis Universal (ETC1S and UASTC): The most popular KTX2 formats. These provide GPU-supported compression of RGBA data. Unlike most image formats, the pixel data does not need to be decompressed on the GPU — saving GPU memory and making image upload to the GPU dramatically faster than PNG or JPEG. Unlike most GPU compression formats, these formats are compatible with any device and GPU.
• BC6H: GPU-supported compression for open-domain / HDR floating-point data. Comparable to OpenEXR, but focused on efficient runtime use rather than interchange.
• R11F_G11F_B10F, RGB9_E5, R16F_G16F_B16F_A16F, R32F_G32F_B32F_A32F, ...: Uncompressed formats (may be combined with lossless supercompression) that can be uploaded directly to GPU without parsing pixel data.

Basis Universal is by far the most common format used in KTX2 today, and supporting that alone would be amazing, but I did want to mention the other options in case they are relevant to others.

For illustration, here's a demo loading a glTF 3D file containing KTX2 compressed textures:

https://threejs.org/examples/?q=gltf#webgl_loader_gltf_compressed

[aras-p]

How would KTX in it's primary use case ("textures compressed for GPU usage") actually work within OIIO context? To me it seems that OIIO is primarily targeted at applications that need to "read images and get their pixel values" (the images can be in multiple formats, scanline or tiled, deep etc., but in all cases you want to get pixel values in the end). And then a bunch of other OIIO functionality (image cache, image algorithms) operate on these actual pixel values.

However, KTX (and especially Basis Universal) use case is very much different - you don't want to get the pixel values; you want to get "data ready to use for the GPU" (in one of GPU supported block-compressed formats). Decoding the KTX compressed image into raw uncompressed form kinda nullifies all advantages of using KTX in the first place?

To some extent this affects other formats like DDS too (#2338). Current DDS implementation in OIIO always decompresses the block-compressed texture data. Which is fine for CPU pixel access, but not really efficient if you want to use the loaded image on the GPU.

[lgritz]

@aras-p I think that's an interesting question. If it's a widely used image format, I still think there's a use case for being able to get to the pixels, if for no other reason than interchange to other file formats, or other pixel-by-pixel operations.

But indeed, as we push forward to (a) using GPUs to accelerate OIIO operations, and (b) having the OIIO texture system or its analogue working for GPU-based renderers, we will sooner or later need to grapple with this very issue. That probably means thinking about extending the current ImageInput API to have new entry points for "load tile/image into a GPU buffer" with an implied desire to keep it in a compact GPU-friendly format if it is already in that state.

[aras-p]

Yeah, so basically it's several questions:

1. "short term" ("today"), how would KTX support in OIIO look like? Would it always just do decoding to r8g8b8a8 pixel values upon reading the files?

2. "long term", is a question for OIIO in general. If it wants to enable use cases like "the data is meant for the GPU only", then it has to support block-compressed GPU formats (BCn on PC, ASTC/ETC on mobile), and also possibly enable usage of DirectStorage (on Windows), "Metal 3 fast resource loading" (on Mac), and <I've no idea whether something similar exists on Linux>. But in any case, this would need extending OIIO public API to deal with this new concept.

[atteneder]

1. "short term" ("today"), how would KTX support in OIIO look like? Would it always just do decoding to r8g8b8a8 pixel values upon reading the files?

This is what I'd be primarily interested in, but with even more emphasis on encoding KTX 2.0. It kicks open the door for authoring (e.g. Blender) and format conversion workflows.
Feeding it r8g8b8(a8) would be a great start. A floating point cube map format for IBL could be up next.

2. "long term", is a question for OIIO in general. If it wants to enable use cases like "the data is meant for the GPU only", then it has to support block-compressed GPU formats (BCn on PC, ASTC/ETC on mobile), and also possibly enable usage of DirectStorage (on Windows), "Metal 3 fast resource loading" (on Mac), and <I've no idea whether something similar exists on Linux>. But in any case, this would need extending OIIO public API to deal with this new concept.

KTX-Software/libktx would support direct upload to GPU (for OpenGL and Vulkan). I assume you mean something like this, right?

I see this useful in "final" applications (games, RT3D apps; only displaying those textures) for which I personally wouldn't use OIIO. I'd rather convert to KTX and use libktx for loading. Then again, I don't know what OIIO users use it for/require in the future.

[aras-p]

Ah, so basically the expectation of what "KTX" would mean in OIIO today is, that it would follow the current OIIO design - reading an image produces uncompressed pixel values for CPU consumption, and writing an image does uncompressed->compressed process. i.e. completely ignoring "GPU" bits, right?

And then if/when at some point OIIO gets functionality/API to target GPU usable textures directly, then all of this would need to be rethought for KTX, DDS etc.?

[atteneder]

Ah, so basically the expectation of what "KTX" would mean in OIIO today is, that it would follow the current OIIO design - reading an image produces uncompressed pixel values for CPU consumption, and writing an image does uncompressed->compressed process. i.e. completely ignoring "GPU" bits, right?

Yes, that aligns with my understanding (and requirements)

And then if/when at some point OIIO gets functionality/API to target GPU usable textures directly, then all of this would need to be rethought for KTX, DDS etc.?

I assume yes, but I don't know OIIO's internals nor plans.

[lgritz]

That sounds right to me -- a near-term implementation of the current APIs (calls to retrieve uncompressed scanlines or tiles into an application buffer in whatever data type is most appropriate from among those we support), and also start thinking about a long-term extension of the APIs to ask for raw compressed data (including, possibly, all the way into handle to a GPU buffer).

I haven't speced out what would be necessary or helpful for the latter, and my experience with compressed GPU texture formats is very limited. I would be interested in somebody sketching out exactly would would be desired in this set of features. Open questions that come to mind (and which I don't claim to know the answer to):

• Should the exposed functions just put the compressed data in a host/app buffer and trust the app to then transfer it to the GPU for whatever graphics APIs it's using? Or should our functions take a handle to a GPU/device buffer and fill it in directly and as efficiently as possible and avoid any app/CPU side buffer? If the latter, does that mean we need to separately implement logic within OIIO that's specific to each of several GPU APIs? (DX, Metal, OGL, Vulkan, Cuda, others??)
• Should we also implement OIIO TextureSystem (purely on the CPU side) support for in-memory block compressed textures for feature parity on the CPU side? (Or implement that by simply fully decompressing to place blocks in the texture tile cache?)
• Does that mean we also need additional methods for querying how much memory the blocks need for allocation?
• Should the ability to read compressed data from texture files only be for block-compressed formats (which have a nice fixed compressed size, I think?), or should the feature be fully generalized to a call for "get compressed tile" for all compression types, all file types? (Currently, we don't support anything like that, which has the advantage of making simpler APIs and lower complexity, but at the expense of, for example, not easily being able to do an efficient file-to-file copy via OIIO API calls without the expense of a decompress-recompress round trip.)
• Would users eventually expect our maketx (or oiiotool -otex) to support writing files with these compression methods?

I assume yes, but I don't know OIIO's internals nor plans.

This is it, right here. These discussions are how the plans are made.

[Nolram12345]

Any updates on KTX2 support in OIIO? Seems like quite an important feature seeing how much is moving to Vulkan these days.

[lgritz]

I think we're just waiting for somebody who knows about KTX2 and needs this feature to come forward and make a PR with the implementation. Any of you can be the hero of this story!

I don't know anything about KTX2 nor do I program for Vulcan, so I'm certainly not the right person to do it.