Texture compression for 3D Tiles and glTF

[pjcozzi]

@bagnell @lilleyse please add comments with your latest notes.

@sbtron we will be very happy to collaborate on glTF texture compression (see linked glTF issues below), but probably need a few more weeks of research.

• Quickly pull together a proof-of-concept pipeline that compresses textures, stores them in a glTF extras properties, and loads them in Cesium. Texture compression #4758
• Figure what formats to support (we’ll need multiple formats to cover mobile and desktop) and what compression tools work well for our use cases.
  • WebGL extensions: S3TC, ATC (likely not worth it), PVRTC, ETC1, ETC2 (WebGL 2)
  • PlayCanvas does DXT (desktop), PVRTC (iOS), and ETC1 (Android)
• Flush out glTF extension spec. What is the container texture file format?
  • Reference in extensionsUsed/extensionsRequired (glTF 1.1)
  • In glTF 1.1, will need to list the GL extensions in glExtensionsUsed; however, we won't want to have, for example, three copies of a glTF file to support each referenced .ktx file.
  • Improving texture support for 1.0.1+, https://github.com/KhronoxsGroup/glTF/issues/620
  • Evaluating texture storage for different APIs, Evaluating texture storage for different APIs KhronosGroup/glTF#739
• Flush out Cesium: 3D Tiles and glTF loader.
• Flush out glTF-pipeline texture compressor(s) integration (see below).
• Tech post for the Cesium blog.

Tools

• Texture compression plugin point for glTF-pipeline. Tools to consider (the first three are likely the strongest):
  • Google’s ETC2Comp. code, blog post, more detailed blog post. Apache 2.0 license.
  • AMD Compressonator. code, website. MIT license.
  • Transcode JPG-like format to DXT using Crunch. ZLIB license. JavaScript loading. (used by Unity)
  • NVIDIA Texture Tools (DXT / BCn Compressor)
  • Imagination’s PVRTexTool. Not open source?
  • astc-encoder Developed by ARM. Used in PVRTexTool. License
  • FasTC. Apache 2.0 license.
  • Etcpak. Open source, license TBA.
  • Apple’s texturetool. Not open source?
  • Fast ISPC Texture Compressor

Longer-term

• Compress leaf tile textures? Different compression rate at different tree levels?
• Use this for Cesium imagery.
• When can we use GST: GPU-decodable Supercompressed Textures?

Misc notes

• DXT: 2-10x faster than ASTC. ASTC 4-20x faster than ETC2
• Tegra is only mobile GPU with DXT
• PVRTC/PVRTC2 - PowerVR proprietary - iOS. Square textures only says the Intel article?!?!?!
• ETC/ETC2 - Khronos ES 2.0 extension - no alpha in ETC1 - better quality than DXT
• ASTC - Khronos ES 3.0 - wide range of compression ratios. adaptive per 128-bit block. Android Extension Pack.
• zip/lsw - better than jpg?!?!!?

Resources

• What texture compression formats does my system support? http://toji.github.io/texture-tester/
• Texture Compression at CppCon
• Texture formats for 2D games, part 4: Overview
• Old: Compressed Textures in WebGL
• Using ASTC Texture Compression for Game Assets
• Android* Texture Compression - a comparison study with code sample
• Efficient Game Textures with Hardware Compression

[pjcozzi]

Initial numbers for a texture from a reality model via @bagnell:

Raw size         3MB    // Also the size in GPU memory without compression
PNG              928KB
JPG              229KB
 
For desktop:
DXT3             1MB    // Size in GPU memory with compression.  Note that DXT3 is RGBA.
DXT3 + gzip      238KB  // Wow – size over the wire is on par with jpg!
 
For iOS:
PVR 4bpp         513KB
PVR 4bpp + gzip  213KB  // On par with jpg
PVR 2bpp         257KB
PVR 2bpp + gzip  114KB  // Better than jpg, but check quality
 
For Android:
ETC1             513KB
ETC1 + gzip      217KB

[pjcozzi]

There's a JCGT paper on FasTC: http://jcgt.org/published/0003/04/07/

[pjcozzi]

• Crunch/Emscripten performance numbers (old): http://www-cs-students.stanford.edu/~eparker/files/crunch/decode_test.html
  • When texture compression is not supported, they transcode DXT to 565. This could be a good option so we don't have an extra set of uncompressed textures to serve. http://www-cs-students.stanford.edu/~eparker/files/crunch/dxt-to-rgb565.js

[pjcozzi]

Fixed in #4758 and CesiumGS/gltf-pipeline#204.

There will be more work for us to expose this through the model converter, do the official glTF spec, and eventually adopt the universal format.