Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Granular progress tracking ? #906

Closed
sguimmara opened this issue Jan 6, 2025 · 6 comments · Fixed by #920
Closed

Granular progress tracking ? #906

sguimmara opened this issue Jan 6, 2025 · 6 comments · Fixed by #920
Labels
enhancement New feature or request
Milestone

Comments

@sguimmara
Copy link
Contributor

Is your feature request related to a problem? Please describe.

In Giro3D, we have an API to query the "percentage of progress" of a given object. The exact semantics depend on the object itself. For example, in the case of a tiled image layer, the progress is the number of downloaded tiles over the total amount of visible tiles.

To implement such an API for 3D Tiles-based entities, we need to know the "amount of total work" versus the "amount of work already done".

Describe the solution you'd like

For the TilesRenderer, this could be an API (typically an event) to know when the loading of a tile starts and an event to know when the loading has finished (this event already exists, load-model).

The simplest implementation would simply add the load-model-start event that would be raised at the very beginning of parseTile().

Describe alternatives you've considered

I tried dabbling with the .stats property, but it does not provide the information I need.

@sguimmara sguimmara added the enhancement New feature or request label Jan 6, 2025
sguimmara added a commit to sguimmara/3DTilesRendererJS that referenced this issue Jan 6, 2025
@gkjohnson
Copy link
Contributor

The simplest implementation would simply add the load-model-start event that would be raised at the very beginning of parseTile().

For a few reasons I don't think this is going to get you the load detail you want. For one it wouldn't include the download time or time that an item is queued. The time from the beginning to end of parseTile also may not even last more than 1 frame so it will likely flicker between 0 and 100 pretty quickly. And including things like time queued for parsing and downloading will mean accounting for tile cancellation, etc.

I think using the lru cache, download, and parse queue fields can be used to calculate the load percent, though:

const loading =
    downloadQueue.itemList.length + downloadQueue.currJobs +
    parseQueue.itemList.length + parseQueue.currJobs;

// or

const loading = stats.downloading + stats.parsing;

const percent = 1.0 - ( loading / lruCache.itemList.length );

console.log( `Percent loaded: ${ 100 * percent }%` );

ParseQueue.itemList.length lists the total number of jobs that are currently waiting to run. ParseQueue.currJobs lists the current jobs that are currently running (and have therefore been removed from the item list queue). And LRUCache.itemList lists the total number of tiles that have either been loaded or have been queued to load. If this looks good we can add something like a loadPercent member field to TilesRendererBase to expose this more directly. Alternatively we could list the number of tiles actively loading and the number of tiles in the lru cache so it can be calculated.

This will still result in cases where the load percent will reach 100 (or near 100) and then jump back down to a low percent again even when not moving the camera since not all tiles-to-be-loaded are immediately added to the queue / cache.

@sguimmara
Copy link
Contributor Author

sguimmara commented Jan 8, 2025

@gkjohnson Thanks! I will have a look at your suggestion.

Note however that we have a problem since we share the queues and cache between renderers, so this would prevent computing per-tileset progress.

@sguimmara
Copy link
Contributor Author

sguimmara commented Jan 8, 2025

Another issue: the ratio between downloading+parsing tiles over cache item list tends to asymptotically go to 1, which in turn will make the progress oscillate between 99% and 100% without moving on the whole 0-100% interval.

To solve this issue, we have a utility class in Giro3D called OperationCounter, which is incremented and decremented when operations start and finish. However, when all operations are finished, the counter resets, so that we avoid this asymptote. That's why I was proposing the tile-load-start event, so that we can track individual tile loads and increment/decrement accordingly.

@gkjohnson
Copy link
Contributor

Note however that we have a problem since we share the queues and cache between renderers, so this would prevent computing per-tileset progress.

Maybe we can keep track of the number of tiles added to the cache in the "stats" field, as well. So calculating progress will look like so: 1.0 - ( stats.downloading + stats.parsing ) / stats.inCache.

Another issue: the ratio between downloading+parsing tiles over cache item list tends to asymptotically go to 1, which in turn will make the progress oscillate between 99% and 100% without moving on the whole 0-100% interval.

To be clear there already are tiles-load-start and tiles-load-end that fire when loads start while nothing is loading or when the last tile geometry has finished parsing. Either of these could be used as indicators that a loading should be "reset". I suppose you'd still need to know when "new" items are added into the cache.

That's why I was proposing the tile-load-start event, so that we can track individual tile loads and increment/decrement accordingly.

I'm assuming you mean a model-load-start event but I'm still not fully seeing how a will help here. This and the model-load function will call nearly back to back and the proposed location for the event isn't indicative of a tiles content load beginning.

@sguimmara
Copy link
Contributor Author

I suppose you'd still need to know when "new" items are added into the cache.

Yes exactly. We could have LRU Cache events so that we can create a new batch of "new items", then compare that to the downloading+parsing count. Or maybe we could leverage the LRUCache.setLoaded() function somehow ?

@gkjohnson
Copy link
Contributor

gkjohnson commented Jan 9, 2025

#915 has added support for tracking per-tilesrenderer tile-in-cache count (as suggested in #906 (comment)) but I'll have to think through how to cleanly support "reseting" the load counter once it has reached 100% as you're suggesting. I think it's a good use case to support but I'd like to avoid littering the code with more events for just this use case and the user having to track that data themselves.

Options

  • Keep track of a time that the stats were reset and add / remove from a total inCache state based on the time added to the lru cache (already used here). When the load completes the in cache value and time are reset.
  • Keep track of a set of tiles that have been added since reset. Clear the set on reset.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants