High memory consumption if tiles.json is big

[SunBlack]

We have a KdTree with ~400.000 (each tile is 200KB) nodes in a single tiles.json (82MB Json file in compact mode). If I open cesium with this tiles.json Chrome crashes and Firefox starts using ~3GB of memory - even if no tile will be loaded. A possible solution for me will be to split tiles.json in multiple once (I hope), but nevertheless someone should take a look at parsing tiles.json to save memory/cpu time (e.g. create Cesium3DTile on demand and not at loading time of tiles.json)

[pjcozzi]

3D Tiles supports a tileset referencing another tileset by having a tile's content point to another tileset.json file. See the External Tilesets part of the spec. It should work well for your case.

I also agree that 400,000 empty tiles should not use up that much memory; we'll look into that.

By the way, what are you building with 3D Tiles?

[SunBlack]

We are currently trying to get a some of our point clouds into cesium. This was a try with ~5% of our data of a single city ;-).

[lilleyse]

Some early results for this...

If the tileset.json is completely empty it works fine. My tileset.json is 62 MB with 349525 empty tiles, basically it's just an 9-level quadtree. It takes about 10 seconds to load for me but afterwards runs fine. The chrome process is hovering around 1328 MB.

With the same tiling scheme but with content in each tile (around 6K per b3dm), it crashes for me.

[lilleyse]

It seems like the problem is related to bogging down the promise system.

I turned off the Cesium3DTileset update loop. The app doesn't crash if I comment out addContentReadyPromise from the constructor. Maybe when is not well suited for waiting for 349525 promises.

Just for reference:

function addContentReadyPromise(tile) {
    // Content enters the READY state
    when(tile._content.readyPromise).then(function(content) {
        if (defined(tile.parent)) {
            --tile.parent.numberOfChildrenWithoutContent;
        }
    }).otherwise(function(error) {
        // In this case, that.parent.numberOfChildrenWithoutContent will never reach zero
        // and therefore that.parent will never refine.  If this becomes an issue, failed
        // requests can be reissued.
    });
}

[lilleyse]

With some tweaks I can load the tileset. I'll try to open a PR soon.

[mramato]

• I'm sure this isn't the problem, but when(tile._content.readyPromise) is unneeded if readyPromise is always a promise. Just use readyPromise.then.
• While I have a branch that starts moving us from when to bluebird (which might use less memory), I'm not sure having 349525 asynchronous operations in flight at any one time is ever a good idea no matter what library you are using.
• However, looking at the code, readyPromise is always going to be resolved for empty tiles, so all of this code execute immediately for the empty tileset use case anyway and I'm not sure why it would bog anything down or affect memory. Perhaps it's what happens after these promises are all resolved that's the problem?

[lilleyse]

However, looking at the code, readyPromise is always going to be resolved for empty tiles, so all of this code execute immediately for the empty tileset use case anyway and I'm not sure why it would bog anything down or affect memory. Perhaps it's what happens after these promises are all resolved that's the problem?

In the case with tiles that have content, none of the ready promises are ever resolved because I turned off the update loop so no requests are made. The empty tiles have always worked fine since they resolve immediately.

[pjcozzi]

@SunBlack can you test if the fix in #4201 works for you?

[SunBlack]

Had now time to test it: No Chrome is still crashing.

And just an general question: At the time Cesium is loading this big tileset.json it stopping rendering. So that Cesium works asynchronous means only order of execution is not fixed, but it is still single threaded and uses no web worker, or?

[pjcozzi]

@SunBlack can you share your tileset with us for testing?

As of now, web workers are used for processing vector data, but we plan to extend this usage.

[lilleyse]

@SunBlack if you have a chance can you retry this tileset on the latest 3d-tiles?

[SunBlack]

Cesium isn't crashing anymore, but still requires 1,3GB RAM for this 80MB tileset.

[pjcozzi]

@SunBlack can you share your tileset with us for testing?

[lilleyse]

Brought up on the forum: https://groups.google.com/d/msg/cesium-dev/2ORBapD2kBg/2Q-6ZgJ9BAAJ

#6390 may help by reducing the number of properties a Cesium3DTile contains, but probably won't solve this completely.

[intckarl]

As for now, what will you suggest if I have so many modules to show in Cesium?
To put all in one json or split them up?

[lilleyse]

Usually it is a good idea to split up a large tileset.json file into multiple smaller tileset.json files and link them via external tilesets.

It saves initial download time and means Cesium3DTile object are only created on demand when an external tileset is loaded, which is the root of the issue here.

[ggetz]

Also reported in #12049. There are some useful profiling results listed in that thread as well.

[jjhembd]

External tilesets themselves create a memory issue, because they are never deleted from the tree. See this forum thread and the related issue in cesium-native.