How to streaming file glTF 2.0 from server ?

[ProFive]

Hi all,
I want to streaming file glTF 2.0 from server because the models is very very big(huge model). I try to use https://github.com/fl4re/rest3d-new but it supports streaming only file glTF 1.0
Information of file glTF 2.0:

• File glTF is 40MB
• File bin is 180MB
• Geometries: 19164
• Calls: 19164
• Vertices: 20140196
• Faces: 6713382

Thanks and best regards!

[pjcozzi]

@ProFive for an HLOD approach to streaming, check out 3D Tiles which is a spatial data structure containing nodes with glTF.

For progressive streaming, there is interest (but no extension AFAIK) in using POP buffers: https://x3dom.org/pop/

Note that 3D Tiles and pop buffers could even be used together. Definitely let me know if you do that! 😄

CC @mlimper

[donmccurdy]

I have a hunch that progressive streaming could be done without an extension... For example you might write a serverside layer to stream bufferViews gradually over a websocket, and a web client can update gradually... to me this is an area where I'd like to see it implemented a few times, and then if there's clear benefit to turning that into a standard, we can figure out how to do so.

Also worth considering: #1045

[mlimper]

I think it makes sense to differentiate between two kinds of streaming here:

1. Streaming an entire scene by progressively adding mesh geometries
2. Streaming a single mesh geometry by progressively refining bufferViews

For progressive streaming, there is interest (but no extension AFAIK) in using POP buffers: https://x3dom.org/pop/

This is case 2, and we made a kind of extension draft quite a while ago (glTF 1.0), checking what would be necessary to support POP buffers or similar progressive LOD methods. The idea was to provide an additional layer of bufferChunks that will correspond to meaningful subdivisions of a bufferView. In the case of the POP buffers, a new progressive LOD refinement is available with each new bufferChunk, it's possible to imagine other progressive LOD methods using this concept. Here's a minimalistic example video:
https://www.youtube.com/watch?v=393lkD3PBm4

I have a hunch that progressive streaming could be done without an extension... For example you might write a serverside layer to stream bufferViews gradually over a websocket

If there is no such thing as a progressive LOD / reordering of the mesh data, and if indexed rendering is being used, getting aribtrary portions of a bufferViews may not be very useful (for example, parts of a vertex data buffer and parts of an index data buffer may not be used for rendering because the index data buffer may refer to vertices that haven't been received yet).
However, when non-indexed rendering is being used, you'll get a progressive streaming at the granularity of individual triangles, without the need for any modifications or extensions to glTF, which may be an interesting case.

Regarding case 1, the client may download the main glTF JSON file, quickly analyze it and then prioritize and manage the downloads of all the mesh elements, or, to be more precise, of their respective bufferView containers. For example, you'll be way better off by first downloading the indices for a large, visible mesh along with its vertex attributes, then doing that for the next larger one, etc., instead of downloading all indices for the entire scene and then loading all vertex attributes, or using any other "random" order... we did this in our engines by using a small JS download manager that supports priorities. On the glTF level, it's very helpful if you know the 3D bounding boxes of meshes, so the client can get an idea where things are (-> viewfrustum culling) and what objects are large, that gives you pretty good priorities. This approach worked quite well for us some years ago, here's an example of a pretty large model (using also some additional hints from the server to prioritize downloads): https://www.youtube.com/watch?v=3UyLGLYJ5pY

@ProFive for an HLOD approach to streaming, check out 3D Tiles which is a spatial data structure containing nodes with glTF.

I guess the experiences we made with large CAD models are very similar - using spatial data structures to make sure to always just "download the right things" is very important.

@ProFive Given that your scene has almost 20K geometries, maybe you can already get pretty far with prioritized downloads? Or is this something you already do, and you really miss the streaming of data for individual meshes?

Thanks, and sorry for the long post :-P

[donmccurdy]

Great details @mlimper! Before we get too far here, is it worth merging threads with #755?

[mlimper]

Before we get too far here, is it worth merging threads with #755?

It probably makes sense to distinguish both cases and put them into different threads:

Case 1 (progressively streaming scene content) does not require any modifications to glTF (as far as I can see), agree it should probably be discussed within #755.

Case 2 (progressively streaming a single mesh geometry) could require modifications to the spec, or an extension. I guess it could be discussed within this thread, but please go ahead and do whatever you believe works best.
FWIW, just did a quick search and found an old PR #434 (seems I should have opened a new one by now... but no strong claims, let's discuss first), and there's also #596

[fferri]

@mlimper would it make sense to consider also the use-case where one wants to stream an animation in real-time?

[mlimper]

Hm... I'm not sure if I get the idea correctly - you mean a case where the animation data is so big that it needs to be cut down into multiple streamable chunks? Like, having a lot of 3D position data for different morph targets, and loading them one by one?

[fferri]

That could be a use-case.

Another one could be that an application (simulation, game, etc) can transmit the content (buffers, meshes, poses, animation tracks) in real-time for remote visualization. Is this last one beyond the scope of glTF?

[donmccurdy]

you mean a case where the animation data is so big that it needs to be cut down into multiple streamable chunks?

One case that is already available is to put different animations into different .bin files, downloading each animation when it is needed. Could be used for breaking an animation into chronological chunks, or lazy-loading animations in a game that aren't needed until the player unlocks them.

... transmit the content ... in real-time for remote visualization. Is this last one beyond the scope of glTF?

You'd need something considerably more complex than glTF to have one application reading from a file at the same time as another application is writing arbitrary data into it. That feels more like the role of an interchange format, perhaps. But I imagine someone could define an extension that allows open-ended buffers for animation samplers, allowing animation to stream in without fundamentally changing the structure of the scene.

[synth]

One case that is already available is to put different animations into different .bin files, downloading each animation when it is needed.

@donmccurdy Is there any reference for how to do this? We're just getting started in this field (rendering glTF in the browser via A-Frame) and quickly hitting network limitations. Not sure if this is normal, but our bin file for :20s of video when exported to glTF is 4Gb (recorded from lidar ipad).

[donmccurdy]

@synth that's a lot of data for 20s of video, yes. I'd expect that raw LiDAR data coming from the iPad will need a lot of processing to optimize, cleaning up and simplifying geometry. If it has textures, those would need to be optimized or baked to vertex data too. If each mesh is only going to be shown for a fraction of a second, you'll want each mesh to be pretty small.

The animation styles that work best in glTF are keyframe T/R/S, skinning, and shape keys / morph targets. There are some formats like Alembic that focus specifically on streaming a geometry sequence, but it's pretty hard to support on the web and A-Frame does not, as far as I know.

If you've got the meshes down to a reasonable size, you can split each mesh into a separate .bin file using the gltf-transform partition command:

❯ gltf-transform partition --help

  gltf-transform 0.10.2 — Commandline interface for the glTF-Transform SDK.

  USAGE — partition

    ▸ gltf-transform partition <input> <output> [OPTIONS...]


  Partition binary data for meshes or animations into separate .bin files. In
  engines that support lazy-loading resources within glTF files, this allows
  restructuring the data to minimize initial load time, fetching additional
  resources as needed. Partitioning is supported only for .gltf, not .glb, files.


  ARGUMENTS

    <input>                              Path to read glTF 2.0 (.glb, .gltf) model
    <output>                             Path to write output

  OPTIONS

    --animations                         Partition each animation into a separate .bin file
                                         boolean
    --meshes                             Partition each mesh into a separate .bin file
                                         boolean

Depending on your use case you might want to group things into .bin files differently, and would probably need to use the scripting API to control that more closely.

[synth]

Got it, thank you @donmccurdy !!

[CITIZENDOT]

@donmccurdy, Is there any way to split a single animation into multiple files?

Our model has only one animation which has ~500 frames, and we want to split it and stream to client. Currently, we're streaming the individual geometries for every frame and updating it. We're thinking streaming animation data is more efficient than that, but stuck on how to do it.

[donmccurdy]

@CITIZENDOT strictly to answer the question, the keyframe data from a single channel in a single animation cannot be further subdivided. For example, the "translation" keyframes for joint "LeftFoot" must all be contained in a single accessor and buffer, and thus the same file.

However:

1. The layout of a buffer containing only keyframe data would be very simple, and it could be streamed incrementally without any changes to the glTF specification.
2. It's unclear to me what sort of animation would be a replacement for swapping geometries every frame. Maybe morph targets, if your topology never changes? The animation keyframe data will be very small compared to the mesh geometry, in that case.

[CITIZENDOT]

It's unclear to me what sort of animation would be a replacement for swapping geometries every frame. Maybe morph targets, if your topology never changes?

Yes, We have a alembic file which contains animation data. Currently we're extracting OBJ files for every frame and compressing them using draco. Each draco frame/file is rendered as a frame.

The animation keyframe data will be very small compared to the mesh geometry, in that case

That's what we're going for.

The layout of a buffer containing only keyframe data would be very simple, and it could be streamed incrementally without any changes to the glTF specification

My line of thought is:

• Fetch initial GLB with first few frames of animation data (morphTargets, morphAttributes etc...) PS: Our GLB has only object/mesh
• Incrementally fetch just animation data (morphAttributes).

But I found, morphAttributes cannot be updated on existing model. So, When loading new frames, should we be creating new BufferGeometry instance?

If there is better workflow than this to stream and render animation data, We'd be grateful to know.

[donmccurdy]

The animation keyframe data will be very small compared to the mesh geometry, in that case

That's what we're going for.

I should perhaps rephrase this. When using a morph target animation, the keyframe data that defines when each "frame" is shown will be fairly small. The definition of what that frame looks like — the morph target — may or may not be any smaller than the sequence of meshes you are using now.

But I found, morphAttributes cannot be updated on existing model. So, When loading new frames, should we be creating new BufferGeometry instance?

If there is better workflow than this to stream and render animation data, We'd be grateful to know.

The glTF specification has no opinion here, so a thread in https://discourse.threejs.org/ might be the best place to discuss it. A "workflow ... to stream and render animation data" can mean a lot of different things depending on the specific animation involved; I expect it will be hard to give useful advice without more technical details here.

Your use case also very much like what Vertex Animation Textures (VAT) are intended to solve in software like Houdini, Unity, and Unreal Engine. VAT is not a feature of glTF or three.js, but advanced users have occasionally implemented the technique in three.js (example).