How to handle backwards compatibility in future glTF versions

[pjcozzi]

@javagl said:

Apologies for bringing this up so late, but now that 1.1 is about to be finalized, I think that this is becoming increasingly important (and it's also relevant for all future updates of the spec).

I think we should carefully think about how to handle backwards compatibility. Right now, I see two (related) points that could severely affect backward compatibility:

• Removing properties
• Removing valid values of properties

For example, one change for 1.1 was the removal of technique.states.functions.scissor. The technique.states.functions object contains the setting "additionalProperties" : false. This means that such an object is not allowed to have any property that is not explicitly listed in the properties. The problem I see here is that a loader that strictly obeys the schema would be allowed to report an error when it encounters such a scissor property. Given the schema, it does not know that this property was allowed in an earlier version.

The point is: A JSON that was valid in 1.0 is plainly invalid in 1.1.

(And this obviously cannot be solved by checking the asset.version...)

One strategy to handle this is to never remove properties. This is also suggested in this stackoverflow answer. I know that this has the potentially undesirable side-effect of keeping "deprecated" or "legacy" properties throughout the different versions. But I'm not sure whether there are better solutions for this...

---

Side notes:

• Yes, I know that this is not a problem for a JavaScript-based loader. The lax type checking will usually just throw the unknown properties into the resulting JSON object, and not cause any errors. Users of the resulting objects may have to insert something like an if (defined(function.scissor)) { versionIs_1_0(): } here and there. This is not nice either, but not as critical as in other contexts.

• The point "Removed buffer.type, glTF 2.0 buffer: does it need a type? #786, Little clarification on buffer.type #629" is currently not implemented, as far as I see. Only the allowed value text was removed. Both are similarly critical in the sense that I described above.

• Related: Develop a concept for glTF version handling javagl/JglTF#5 . This, of course, is my problem, and this alone should not affect any decisions here. But the general question of backward compatibility should be tackled nevertheless.

[pjcozzi]

I'm pretty sure we would follow Khronos conventions, which are basically semver. glTF 1.1 is a unique case; it is a "bug fix" release that required some breaking changes. I wouldn't expect a release like this again; only a new x.0 release would include breaking changes.

[javagl]

This is not so much about the version number itself. The question is rather related to the fact that the file may have to be read entirely before its version number can be figured out. So even if it was called glTF 2.0, one still would have to (at least implicitly) support all properties from 1.0, regardless of the "additionalProperties" : false setting.

For 1.1, I think that the number of actual breaking changes is small enough, so that everybody can cope with them (even if they may require one or the other quirky workaround.) And in general, these are things that should not govern decisions that have to be made to keep the format healthy.

(Maybe I'm just overestimating the difficulties here. All this mainly applies to loaders that do not read the JSON into an anonymous, untyped DOM first: For these loaders, any removed property would cause a breaking incompatibility. But it's not unlikely that using an untyped DOM would in fact be the more sustainable strategy. Even if the question about the actual data types for different major versions still has to be answered, it could at least make the loader itself a bit more resilient)

[emackey]

I don't think you're overestimating. I remember firsthand maintaining some code that could ingest multiple past versions of Lightwave objects. I think Lightwave did a great job with both forwards and backwards compatibility, but it didn't happen automatically, they designed for extensibility up front.

The maintainers of glTF should not underestimate how quickly the Internet will become overrun by obsolete glTF files that users still care about. The user-perceived stability of the standard will rest on how well it embraces its own past and future.

[pjcozzi]

All this mainly applies to loaders that do not read the JSON into an anonymous, untyped DOM first

Ah yes, I was only considering JavaScript.

One strategy to handle this is to never remove properties. This is also suggested in this stackoverflow answer. I know that this has the potentially undesirable side-effect of keeping "deprecated" or "legacy" properties throughout the different versions. But I'm not sure whether there are better solutions for this...

Could be viable.

We need to look at the tradeoff between keeping the spec clean and how non-JavaScript loaders can quickly/easily parse out just the version property to see if it is compatible.

@mre4ce do you have any experience here?

[javagl]

The fact that I generated the Java classes based on the schema was basically a mix of an "experiment", "laziness" and the attempt to strictly obey the specs. This may not be the most sustainable solution in general, and again, this should not govern any decisions here.

A (possibly far-fetched) analogy: As another experiment, I recently created classes for COLLADA from the schema ( https://github.com/javagl/JCollada , do not take this too serious!). Of course, this causes entirely different classes for the different versions, e.g. org.collada._2005._11.colladaschema.COLLADA for 1.4 and org.collada._2008._03.colladaschema.COLLADA for 1.5. But for a strictly and strongly typed language, this might be necessary, also for glTF...
However, one difference here is that for the XML input file, one can use a streaming SAX parser to simply read only the root element and detect the version from that (that is: without reading the whole file). In contrast to that, for JSON, the asset property might in the worst case be at the end of the file....

I'm also curious about ideas by people who have more experience with file format versioning here...

[xelatihy]

I am the developer of yocto_gltf [https://github.com/xelatihy/yocto-gl] which is a loader that load into a type object hierarchy for C++. The loading code and C++ hierarchy is automatically generated from the JSON schema.

I started working on glTF 1.1 and the incompatibility causes two issues.

First, the object hierarchy and parsing code is not compatible any more. This means that an entirely new data structure is required. One could manually patch things up, but one thing I like about yocto_gltf is that the code ia automatically generated from the spec, so I am sure that all error checking and conversion code is correct, and not manually tuned.

Unfortunately this also means that backward incompatibility (and breaking changes to the JSON schema) disallow any manual fixing of the backward incompatible changes. Obviously I could switch to manually write a parser, just like many other code bases. But then I loose the correct by default implementation.

This is not to say that changes are bad. In fact, I did suggest some myself in other issues. But just to share an experience of a library for a strongly typed language.

[xelatihy]

BTW, for strongly typed codebases the only option is to have multiple implementation and switch the loader by checking the gltf version at load time. for yocto_gltf this is trivial since the code is automatically generated so supporting multiple versions is very easy.

Note that you can have codebases that are not not strongly types even in strongly-type languages by manually handling incompatible changes.

[javagl]

It seems like the idea of auto-generating code from the schema lends itself for strongly typed languages.

I started looking at the glTF part of yocto-gl, and still have to take a closer look at the actual "parsing" code, but if I understood this correctly, then it still follows the two-step process:

• It reads the JSON into an anonymous, untyped data structure (namely a DOM, from json.hpp)
• It converts this JSON data structure into the typed glTF data structures (using the auto-generated conversion functions for the auto-generated glTF types)

This still allows you to follow the proposed approach:

for strongly typed codebases the only option is to have multiple implementation and switch the loader by checking the gltf version at load time.

You could "peek" into the anonymous JSON DOM to figure out the version, and create an appropriate instance of the (auto-generated) glTF root object for the respective version (Pseudocode: )

if (jsonRoot.property["asset"].property["version"].asString() == "1.0") {
    glTF_t result = parse(jsonRoot); // Current version
} else {
    NEW_glTF_t result = parse_NEW(jsonRoot); // New versions

}

This still brings the necessity to have the whole set of glTF types for each version, even though most of them will be equal, and the few differences may be minor, and might even just refer to whether a property is required or not.
But the convenience of the auto-generation approach as opposed to manually writing a full-blown parser, and particularly the "correct by default" behavior are strong arguments here. (Is someone up to writing some glTF parser unit tests?)

One could manually patch things up, but one thing I like about yocto_gltf is that the code ia automatically generated from the spec

That's exactly the same issue that I ran into for https://github.com/javagl/JglTF/tree/master/jgltf-impl. An additional point here is: When I'm using these classes, I'm not creating a DOM. Instead, I'm using the Jackson library, which magically reads the JSON input directly into the model classes (using data binding - under the hood, this uses loads of reflection magic, but works remarkably well).

From quickly skimming over the README and the code, I did not see how you are generating the implementation classes in yocto-gl. Do you have options to "tweak" or "configure" the generator so that it may be able to generate classes that are capable of representing 1.0 and 1.1 glTF structures?

(For JglTF, I considered a workaround here: Creating a fork of the schema spec, and tweak the schema so that the changes betwen 1.0 and 1.1 are written in a form that causes backward-compatible code to be generated - but I did not yet analyze whether this is a viable approach...)

[xelatihy]

To answer a few of the questions above regarding yocto_gltf:

1. the generator code is not publicly published since it is not documented at all, so it is unlikely useful;
2. I load JSON into its own data structure (DOM) since for small JSONs that is fast enough in C++; with automatic code generation I could switch to a JSON SAX interface, but I suspect the speed up is minimal;
3. I think the schema should be cleaned up a bit to make automatic generation easier. I did a few modification myself mostly for bug fixes (submitted as pull requests that got accepted). But the truth is that the schema was written for dynamically-typed languages rather than strongly typed ones. In my look, a few modification would be sufficient to make automatic code generation remarkably easier. If there is interest from Khronos I am happy to explain more and do a first pass at it.
4. A few of the suggested changes are mostly how I plan to proceed, but I think the code should break when backwards incompatible changes are made. There are many the C++ parsers that are loose. I wanted yocto_gltf to be correct to the schema.

[xelatihy]

One last thing: the idea of code generation from the spec is not new in OpenGL. Many extension loading libraries are automatically created from the Khronos OpenGL specs. For example https://github.com/Dav1dde/glad. yocto_glf tries to do the same for glTF. The problem is that the specs (i.e. schema) have not been developed for strongly typed languages, while the OpenGL specs are developed as strongly typed.

Again, with a few small modification, we can handle glTF just like GL and leave the spec the same. Just a few changes to the way the schemas are written.

[javagl]

I'm not entirely sure what possible schema changes you refer to. There are some places where one either has to refer to an Object (roughly "a void* pointer" in C++), or use "workarounds" like storing possible values as multiple (optional) types. For example, the material.values. I looked at how you solved this in yocto_gltf. (My first thought was: Hopefully there will never-ever be an array with mixed types, like [123, 3.4, true] ....). But I think that this "untypedness" here cannot be avoided, and is solely due to the genericity of what a material should be able to describe. The type simply "does not exist" at compile time.

Nevertheless, I'd at least be curious what changes you would suggest (regardless of whether they can or will be integrated).

[xelatihy]

Probably a certain amount of "untypedness" may be required if material values can bound to anything – sadly this shift the burden on the implementation to perform error checking and proper API call. There are other way to specify this thought that would allow for an easier strongly type feel, for example using type-safe unions (std::variant in C++, standard in all FP languages).

As for the schema changes, the main issues are specifying what to do with parameter/type inheritance and be consistent thought the schema. Also important would be to provide specific type names for the objects. For JS this neither of this is a concern since JS is a prototype language. But it may come back to byte if one wants to switch to Typescript or the newer JS variants.

[javagl]

@xelatihy I know, the question is a bit vague, but: Do you already have plans about how to tackle glTF 2.0? The structures will be "incompatible", and I wonder how the structures (which will have to be entirely different classes in a typed world) may be passed to the consumer. My gut feeling right now is that, in a typed language, there would have to be something like a GltfViewer1 and GltfViewer2 which share maybe 90% of the code, except for one operating on the Gltf1 classes and the other on the Gltf2 classes....

[xelatihy]

Well, the question is not vague. I have being thinking about it too.

Here are the options I considered.

1. Two separate classes, "gltf1"/"gltf2". Ugly and code duplicated but perfectly matching the specs.
2. Compatibility bridge. Make a schema that merges the two with tagged members. Removes duplicated code and feels it will need a lot of manual tuning, loosing the advantage of code generation.
3. High-level classes. glTF is really a low level description. The semantic->attribute->bufferView->buffer indirections are only justifiable if one want to match GL perfectly, not if one where to design a format from scratch. In fact most renderers will have their own geometry pipeline and likely will need to change these indirections anyway to match the internal representation. So taking this view, that raw GL may not be the most important target, provide high-level classes that bridge glTF over a reasonable, more flattened representation. The application now can either access directly the low-level glTF code (in which cases they need to deal with the fact that the formats are very incompatible) or access the higher-level description that handle low-level accesses internally.

For yocto_glft, I plane to support only glTF 2 since I do not think there is enough of a user
base to justify keeping around a backward incompatible version that likely almost none is using anyway. I will follow 3 and start with a trivial description for static-only scenes, to then move to animation.

[xelatihy]

BTW, supporting maps-or-arrays in a typed language will be "fun" :-(

[javagl]

Yes, the change from maps/objects to arrays sounds like a small thing (as in JSON, it roughly boils down to changing { } to [ ] ...), but is a hard cut in terms of backward compatibility. Also, quite some of the simplicity of the format will be lost: No longer copy+paste one entity from one file to another. Now, one has to juggle with indices. (This is one of the main reasons of why I did not yet add a thumbsUp/hooray in the announcment. I'm really looking forward to see the performance numbers here...).

Regarding the options:

1. I agree, that would be simple but ugly. Foreseeing code like

    if (gltfLoader.loadedVersion2()) {
        Gltf2 gltf2 = gltfLoader.getGltf2();
        fineThenRenderThe(gltf2);
    } else {
        Gltf1 gltf1 = gltfLoader.getGltf1();
        Exception up = new Exception("This renderer does not support this version");
        throw up;
    }
   

   is not desirable...

2. Maintaining a proper schema is hard enough. But manually merging them, considering that there may be unresolvable conflicts, sounds like a potential source of headaches. It would have been a way to cope with the few incompatibilities of 1.0 and 1.1, but for 1.0 and 2.0 this does not seem like a viable approach. (And... there will be a 3.0 one day....)

3. Some sort of "abstraction layer" was what I also considered. I'm not yet sure how it would play together with the code generation. (If I understood you correctly: This includes option 1., but the you take the responsibility of hiding the ugliness of gltf1/gltf2 from the user, right?). One could also start to argue whether wrapping an abstraction layer around glTF does not defeat its whole purpose to some extent. But of course, glTF has evolved. The issues that are caused by trying to target e.g. Vulkan are still being resolved. And in the end, it largely depends on how thick this abstraction layer will be. In any case, I think that designing such an abstraction layer is hard: The challange of designing a stable API has many similarities to the challenge of defining a stable file format...

[xelatihy]

One think that seems interesting is that glTF might run at best performance on pure GL-only renderers (not sure how many are there). For the others, the GL-only indirections are a slows down anyway. So, you then convert to your own internal format. Arrays-vs-maps are only a concerns if want to consume it as is, but then any API change in GL/Vulkan might require a drastic glTF change. So portability of the assets and their longevity might not be great in the end.
Only time will tell how it plays out. I was certainly surprise by how backward incompatible glTF 2 is, so I am stopping my implementation a bit till the dust settles.

[javagl]

I think that some of the basic concepts that make glTF compact and fast are shared by the graphics APIs - e.g. raw chunks of float data for what's the lion's share of an asset, namely the vertex attributes. I cannot say this for sure for Vulkan or even DirectX, but would be surprised if they had entirely different requirements here.

And when seeing it like that - glTF being one of the most basic possible descriptions of what has to be rendered - then it might not matter so much whether a particular JSON structure "nicely fits" a certain renderer implementation, but rather that it does not have a large, complicated overhead. And in this regard, I think that e.g. measuring the performance difference between a map-based JSON and an array-based JSON asset description would eventually boil down to measuring the performance of the JSON parser that is actually used there...

However, I hope that the upcoming changes are carefully planned in view of the requirements for Vulkan and DirectX - and in order to keep the momentum of glTF, it's certainly better to make breaking changes early and in one run than later in many small iterations. Nevertheless, supporting both versions at the same time will be challenging on every level - for the loader, the consumers/pipelines, and the renderers. I'm curious to see how e.g. the validator will cope with this, maybe this can be a source of inspiration.

[lexaknyazev]

I cannot say this for sure for Vulkan or even DirectX, but would be surprised if they had entirely different requirements here.

#819 could be the next step to align glTF data structures with modern APIs even more, but the same could be also achieved by adding additional reqs for existing glTF objects without major changes (like requiring all interleaved accessors of the same mesh to use the same bufferView, using same attributes layout for different meshes, etc).

measuring the performance difference between a map-based JSON and an array-based JSON asset description would eventually boil down to measuring the performance of the JSON parser

That's correct. Using string-based IDs as "properties" could lead to noticeable processing overhead. Main culprits are usually nodes and accessors.

[xelatihy]

Agreed on all counts.

[javagl]

@xelatihy There's a first 2.0 version by @lexaknyazev at https://github.com/KhronosGroup/glTF/tree/2.0 with the updated JSON schema (arrays+indices instead of dictionaries+ids). Although there may still be changes for finalizing 2.0, this can serve as a basis for tests of how to cope with the maps-vs-arrays change.

[sbtron]

This is a great discussion and I am also hitting similar questions when flushing out some of the changes in #830

Following the semver versioning means-

• MAJOR version when you make incompatible API changes,
• MINOR version when you add functionality in a backwards-compatible manner, and
• PATCH version when you make backwards-compatible bug fixes.

So say we start off with 2.0 as the Major version that was incompatible with previous 1.0 version. Then we want to have 2.x which can add more functionality but is backwards-compatible with 2.0. But what does that mean for parsers/engines?
There are two possible approaches -

1. An engine/parser that understands 2.0 should continue to be able to read 2.x files and display them accurately just like it would a 2.0 file
   or
2. Any engine that works with 2.x will automatically work with 2.0 but a 2.0 engine doesn't necessarily work with a 2.x file. Whenever there is a new 2.x release the engine needs to be updated to explicitly support that release.

Here is a practical example with #830. If we went with what is currently suggested then model will be an
enum which for 2.0 will only be "METAL_ROUGHNESS".

{
    "materials" : [
        {
            "model" : "METAL_ROUGHNESS",
            "METAL_ROUGHNESS" : {
                "baseColorFactor": [0.5, 0.5, 0.5, 1],
                "metallicRoughnessTexture": 1
            }
        }
    ]
}

But say in 2.x we add another possible model. Since this would be an additional enum a 2.0 engine will have no idea what to do with it and can no longer read the file.
An alternative approach would be to not use an enum at all -

{
    "materials" : [
        {
            "METAL_ROUGHNESS" : {
                "baseColorFactor": [0.5, 0.5, 0.5, 1],
                "metallicRoughnessTexture": 1
            }
        }
    ]
}

In 2.x an additional material model could be added while still requiring that the older model be present for backwards compat. So it could be something like this-

{
    "materials" : [
        {
            "METAL_ROUGHNESS" : {
                "baseColorFactor": [0.5, 0.5, 0.5, 1],
                "metallicRoughnessTexture": 1
            },
         "SPECULAR_GLOSSINESS" : {
                "diffuseColorFactor": [0.5, 0.5, 0.5, 1],
                "specGlossTexture": 1
            }
        }
    ]
}

An older 2.0 engine could still continue to read the METAL_ROUGHNESS properties while ignoring the SPEC_GLOSS properties. So its possible to have some type of forwards compat too but we need to decide whether that's an explicit requirement to support.

What are everyone's expectations around this? @pjcozzi @lexaknyazev
Whichever approach we pick we need to be consistent about it even for 2.0.

[lexaknyazev]

• Backwards compatibility means that new engines can understand old files. E.g., runtime with 2.2 support must be fully compatible with 2.0 files.

• Exporters must use minimum-possible version. E.g., if an asset doesn't use any specific 2.1 features, exporter must write 2.0 version.

• As for your materials example, I'm open to "multiple representations" approach, but there could be several issues here (based on example above):

  • Loosely-defined schema (requires probing all possible material models)
  • Possibly incompatible textures, hence, unwanted duplication

In 2.x an additional material model could be added while still requiring that the older model be present for backwards compat.
...
So its possible to have some type of forwards compat too but we need to decide whether that's an explicit requirement to support.

New (> 2.0) asset having a material for older engines is definitely a "forward-compatibility" feature, and I'm not sure if it's a reasonable approach for data format.

---

Comparable example:
MP4 file can contain video and/or audio of different codecs. Let say that one contains AAC audio and HEVC video. File container itself as well as its metadata and audio track could be read and decoded by old software, but only HEVC-capable players would be able to fully render such file.

[lexaknyazev]

Comparable example:...

On the other hand, if you need to serve clients with different decoding caps (think any major video streaming service), you would provide them different versions of the same video via something like MPEG-DASH.

---

With that in mind, we could think of desired "atomicity" level of glTF asset.

[pjcozzi]

I was only anticipating backwards compatibility here. So far, the glTF community has shown an impressive willingness to upgrade quickly (2.0 will be the ultimate test!). I understand this isn't as easy for desktop apps.

For simplicity, which is key to supporting glTF adoption, I am hesitate to suggest forward compatibility as I expect it would be hard to ensure in practice and we might end up bumping to 3.x sooner than expected when we find out that we can't support it.

@sbtron I am open to ideas if this is an important use case for you and you think this is a reasonable path that generalizes well.

[pjcozzi]

@lexaknyazev @sbtron did we spell out the forward compatibility in the spec or some best practices guide? OK to close this?

[bghgary]

@pjcozzi Versioning is spelled out in the spec here: https://github.com/KhronosGroup/glTF/tree/master/specification/2.0#versioning

There are some discussions in the PR for background: #899 (comment)

[pjcozzi]

Perfect, this can be closed.