Adding LODs to glTF

[sbtron]

We have been experimenting with the idea of adding LODs to glTF through this extension:
https://github.com/KhronosGroup/glTF/tree/master/extensions/2.0/Vendor/MSFT_lod

These could be useful for rendering optimizations where you can render a different LOD based on distance of object and also provide a progressive loading experience by incrementally loading and rendering the lowest to highest LODs

Some wip demos:
https://sbtron.github.io/BabylonJS-glTFLoader/?model=BoomBoxLOD

• load another asset first to get the environment cached
• turn on throttling to see the progressive load of textures

Would love any feedback?

[lexaknyazev]

Some questions after first glance:

• Why geometry LOD is defined on node instead of mesh?
• How does it work with morph targets (e.g., all linked meshes must have the same set of morph targets)?
• How does it work with skins?
• How does this extension affect children nodes?

[stevepg]

I like what you have here. We have an extension in the works that does something very similar, but also offers the LOD extension at the mesh level.

one question : typically, the tools creating LODs will do so based on some switching criteria e.g. this resolution is valid at this screen coverage or this distance from the camera. You don't appear to have any criteria defined here that may provide a hint to a renderer as to when to switch LODs. DO you plan to add anything like this?

[space2]

Nice idea! A quick question:

Assume node#0 is the main LOD, and node#1 and node#2 are the medium and low lod alternatives. What if node#0 has a child node, node#3. If rendering the medium or low lod, should node#3 be ignored, or not?

On one hand, the LOD system can be interpreted as an in-place replacement for the geometry, so maybe the hierarchy should be kept intact (e.g. the child node could be the sword the hero character holds in his/her hands).

On the other hand, it's also possible that the child node is not needed any more, since the meshes were merged into one into lower LOD levels.

[sbtron]

I have been assuming node#3 will be ignored when displaying a LOD level that doesnt contain it.
Effectively Node#0 acts as the root node for the high LOD and any child nodes are part of that LOD. Similarly Node#1 and Node#2 are the root nodes for medium and low LODs and any children to those nodes will be part of the respective LODs.

[Selmar]

I would like to implement this, however, I have a few issues that are not clear to me when reading the spec:

---

Nested LOD nodes
What happens in the case of nested node LODs (e.g. a Node with LODs references a child node that also has LODs)? Should this even be allowed?

Distance-based LOD
If the renderer does not support screen-space LODs, there is no alternative in the spec. Of course, the screencoverage could be used to calculate a distance, but this mapping is not perfect (consider viewing angles, fov, etc). Should the screencoverage parameter be renamed to cover this more generic case and leave it to the client, or should there be a second (optional) parameter to specify distance-LODs?

Material LODs

If both node level and material level LODs are specified then the material level LODs only apply to the specific node LOD that they were defined for.

How should this be interpreted? Should we map the Node LOD to the material LOD? Can we define distance/screencoverage for material LODs? Or are the material LODs intended for progressive loading only?

Personally, I don't think I will implement material LODs at all and just use the highest LOD by default. Is this acceptable behaviour?

--
Looking forward to your opinions.

[sbtron]

Can't think up of a good use case for nested LODs. They should probably not be allowed.

Good points about distance based and material lods. We came up with this for a very specific use case and the idea was to try and generalize it so that others could also implement it. Clearly, we will need to balance out ease of implementation and ability to support specific capabilities.
The material LODs were an optimization to avoid repeating some of the json for node level lods. So a simpler implementation could definitely just implement node level and achieve the same capabilities without implementing material level.

Same goes with screen coverage - we actually put it in extras at the moment because we weren't sure if there was a good generic way to do it across all engines. One approach could be having an additional properties that engines could pick from - like screen distance maybe?

If you are interested we could develop this further as a more generic multi-vendor extension (EXT) which can be used more broadly?

[Selmar]

I am on the fence when it comes to nested LODs.

A not so obvious use case for nested LODs may be when an object has, say, 5 LODs, ranging from a very simple representation (say two quads), to a very complicated model with several hundreds of thousands of triangles. For the most complicated level, it makes sense to want to subdivide it into LODs again. This has the advantage of being able to zoom in with much higher detail than what would normally be possible. This is something I want to support...

However, nested LODs may make implementing it properly a bit more difficult than it could be. I think the same behaviour can maybe be achieved by simply configuring the LODs for the subtrees separately, making sure the complicated LODs don't display when the simpler LODs do. However, it seems difficult to prevent issues where either several LODs display at the same time, or none, due to floating point errors, modeling mistakes and so forth.

A different solution would be to define both Scene LODs and Node LODs, or Node LODs and Mesh LODs, but I don't feel like either of those is a very good solution.

Perhaps being able to define one level of nested LODs is good.. I need to think about this a bit more. It is not the simplest thing to want to implement in a standardized format, especially when nested LODs have their own "screen coverage", because of different sizes. This makes setting up correct LODs difficult, but it is required to be able to zoom in properly. At least a client implementation can implement this in a relatively straightforward manner, simply by traversing the tree downwards.

Without nested LODs, the implementation seems relatively straightforward. Although I haven't given animation a serious thought, yet.

---

LOD levels seem difficult to standardize in a way that works with every technique. I feel like screen coverage is the best "standard", because it maps to a visual result, as opposed to distance. Additionally, a distance could be computed by combining the bounds and camera FoV and used with somewhat accurate results. The other way around may lead to significantly different results for different cameras. Forcing a client to support several techniques seems a bit awkward, but perhaps specialized values can be defined, if a client supports it.

It should also be defined which LOD level bounds to use for calculating screen coverage.

---

What do you mean by "to avoid repeating some of the json for node level lods"? Do you mean node level LODs were intended to use a matching Material LOD? If so, then that usage conflicts with the progressive loading in the other implementation.

---

I hope all of this makes sense. I've changed my opinion multiple times while writing this comment. :)

It would be interesting to develop it as a multi-vendor extension, certainly. I'll have to get the base glTF implementation to work first, though, before I can work on a LOD implementation!

[mlimper]

This has the advantage of being able to zoom in with much higher detail than what would normally be possible. This is something I want to support...

Did you have a look at 3D Tiles? It already uses a spatial data structure and supports Hierarchical LOD (HLOD), always loading the "right" glTF files for a given view / zoom level etc.

[Selmar]

Yes, I have. It is a little more than what we need. We have some relatively large moving animated objects with many moving parts, which is not quite the same, either.

I think I was overthinking it a little in my previous comment. One level of nesting probably suffices. This means that we would get around with implementing LODs on scene and node level, while disallowing nested LODs for nodes.

[motorsep]

Just as a reference, here is how LOD groups are handled in UE4/Maya/FBX: https://docs.unrealengine.com/en-US/Engine/Content/FBX/StaticMeshes#staticmeshlods

[joejohndoe]

Has anyone clarified these questions with the authors of MSFT_lod? The unclear cases I can think of are:

• Nodes with MSFT_lod that have child nodes: are those part of the LOD or not?
• Nodes with MSFT_lod that have siblings (i.e. a parent node/scene that points to other nodes): are the LODs supposed to be matched up? What if the siblings have a differing number of LODs (or none)?
• Nodes with MSFT_lod that have a parent or children that also use MSFT_lod: nested LOD case as discussed above.

[Selmar]

Just as a reference, here is how LOD groups are handled in UE4/Maya/FBX

So they only support mesh LODs, which we are already going beyond with this extension.

Nodes with MSFT_lod that have child nodes: are those part of the LOD or not?

From what I have been assuming (as has the author), yes, the LOD is applied down the hierarchy.

Nodes with MSFT_lod that have siblings (i.e. a parent node/scene that points to other nodes): are the LODs supposed to be matched up? What if the siblings have a differing number of LODs (or none)?

I assume this is completely independent. Every LOD handles himself.

Nodes with MSFT_lod that have a parent or children that also use MSFT_lod: nested LOD case as discussed above.

In discussion. I think it's not so hard to implement (it comes naturally when you have a hierarchy) and has a few use cases, but it may not be as easy to implement in some engines that have a specific idea about LODs (?).

[donmccurdy]

I’m on the fence about having LODs defined at the node/material level — how widely can that be implemented, and what are the advantages vs multiple LOD sets? As far as I can tell LODs are associated with meshes in Unity and Unreal, and screen coverage for nested nodes is more complex. By swapping LODs at the ‘mesh’ level, like...

node: [
  {
    mesh: 0,
    extensions: {
      EXT_lod: {
        lods: [
          {mesh: 0}
          {mesh: 1, coverage: .25},
          {mesh: 2, coverage: .5},
        ]
    }
  }
]

... we keep the ability to change meshes or materials at each LOD, without the complexity of handling other affected nodes. Note that implementations should be able to reuse geometry when two meshes reference the same attributes but different materials — this is already an important optimization today.

Also note that min/max on a POSITION attribute is not fully sufficient to compute a bounding box. Providing a future extension for storing a bounding box would very likely be valuable, but for this extension I think it could be sufficient to say “unless another extension provides dimensions, implementations may use dimensions derived from the static geometry - consideration of animation and morph targets is allowed but not required or typically expected.”

Those points aside, I think this would be a valuable extension to push forward. We’d certainly implement it in three.js and A-Frame VR, and leadwerks engine is already working on it. I think this type of extension is pretty central to the efficiency goals of glTF.

[steveghee]

The benefit of kids at the node level is that you (potentially) get efficiencies is the representation of more complex ‘assemblies’ of parts. Being able to switch from a 10,000 part (10,000 meshes) to a single mesh, at some distance/coverage, is valuable. Material LODs I would agree are a little bit more esoteric but I can see how they could be valuable.

[Selmar]

In our case, because we wanted to be able to load and stream different model LODs independently, we ended up bypassing the glTF LODs for this and instead use different glTF files for the different LODs.

We will certainly implement mesh LODs at some point, but because of the above choice we actually do not require it on a node level.

However, if you were to consider one glTF model a complete object and render a large world full of them, then very clearly it is necessary to reduce the number of draw calls (i.e. meshes). However, it seems better to then implement this on a scene level, and not on a node level. This avoids many of the above issues about nesting and keeps implementations relatively straightforward (no worries about something modifying your scene graph halfway through the hierarchy).

Of course, if you then were to put many objects in one glTF file, a reasonable LOD mechanic would not be available, but I'm not sure if this is an actual use case?

[joejohndoe]

We (ArcGIS API for JavaScript) are in the same boat as Selmar: we have a single asset per glTF file and no need for a fine-grained LOD, just multiple representations of the full asset. We're currently using MSFT_lod, but we only support LODs at the "root" node level, and we don't handle cases where a scene contains multiple root nodes correctly. This use case could be solved more cleanly by putting the LOD spec at the scene level.

[xissburg]

By the looks of this discussion I assume this is still largely undecided. Any updates on this?

[zellski]

@xissburg My sense is that there's too many use cases, too different from one another, for easy agreement to emerge on a simple extension... and banging out a more sophisticated one that addresses the majority of use cases is a huge amount of "soft" work –– someone, or some ones, to re-ignite and then lead discussion patiently, over time, in finding and merging common interests and working towards a solution acceptable for most, and tolerated by all.

And that doesn't even include weighting the sophistication of the mechanism against how likely it is to quickly see support in major loaders.

I think the situation is both helped and hindered by the fact that MSFT_lod actually captures a decent range of simple use cases.

It'll be interesting to see if the newly founded commerce group (which has quite a strong incentive for major industry actors to identify and formalise common needs) will come up with something. Or alternately, perhaps rather than a six-month, baldness-inducing conversation towards sufficient consensus, someone will just write EXT_fancier_lods that capture 80% of the use cases.

[dkoerner]

Not sure if this is the right place to bring this up: the MSFT_screencoverage is used to decide when to switch between different LOD levels. It is computed in a way which is independent of the screen resolution. Our LOD-metric takes resolution into account and therefore we can not use that extension directly.

It would be good to allow support of different LOD-metrics by generalizing the concept. e.g..

MSFT_thresholds : [number, number, number, ...]
MSFT_metric : string // <- e.g. "screencoverage", "rasterspacecoverage", ...

[stevepg]

You really only want one metric and resolution independent has to be the way. The ms proposal is a good solution for lods.

…

-Steve

On Fri, 23 Aug 2019 at 11:36, David Koerner ***@***.***> wrote: Not sure if this is the right place to bring this up: the MSFT_screencoverage is used to decide when to switch between different LOD levels. It is computed in a way which is independent of the screen resolution. Our LOD-metric takes resolution into account and therefore we can not use that extension directly. It would be good to allow support of different LOD-metrics by generalizing the concept. e.g.. MSFT_thresholds : [number, number, number, ...] MSFT_metric : string // <- e.g. "screencoverage", "rasterspacecoverage", ... — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1045?email_source=notifications&email_token=AC2B5PEIHXUAQIEHWBHG7F3QF64UVA5CNFSM4DTTZKR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD472RLA#issuecomment-524265644>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AC2B5PCSJW3HSUT2RV536ETQF64UVANCNFSM4DTTZKRQ> .

[vpenades]

Please, look at #1660, where I proposed an alternative solution for LODs.

[takahirox]

Please have a look at and review #2228 EXT_node_lod extension proposal.

[vpenades]

@takahirox: In my case, I used Scenes to define LODs, and considering that a Node can be reused across multiple Scenes, it is possible to rearrange, add or remove nodes on every scene to achieve the same layout as with your extension

So the first Scene in the glTF is the 1st LOD, the 2nd Scene is the 2nd LOD and so on.

The only thing you would need is a "coverage" property for every Scene, so the engine would know how to "switch" scenes based on the coverage.

One of the advantages is that, from the authoring point of view, it could be possible for an artist to export 3 models, one for every LOD, and then have a tool that merges these 3 models into 3 scenes in the same file. Also every "LOD" Scene can have less nodes than the previous Scene.

Essentially, you can achieve pretty much the same functionality as this extension, with much mess pain (and much much less pain than MSFT_LOD.... if anyting you would need an extremely easy to implement extension called EXT_Scene_Lod_Coverage just to define a given Scene as being a LOD of another scene.

[dkoerner]

You really only want one metric and resolution independent has to be the way. The ms proposal is a good solution for lods.
…

If I double the resolution in our renderer without changing anything else, I want LOD geometry to adjust according to the new workload. This is a common approach and the reason other standards use LOD thresholds in a pixel based unit (e.g. https://github.com/Esri/i3s-spec/blob/master/docs/1.7/lodSelection.cmn.md).