Viewport collision detection

[ChrisLoer]

Switching from tiled label placement to global viewport label placement. #4704 WIP.

This PR dramatically changes our approach to collision detection. Instead of trying to pre-calculate collisions across a range of zooms in the background, we now calculate collisions synchronously in the foreground. Fade animations are no longer tied to zoom level, but instead based on time elapsed since collision occurred. This approach helps solve many problems:

• More reliable collision detection between tiles (buffering/avoid-edges are no longer necessary)
• Collision detection now works on labels from different sources (e.g. a GeoJSON set of labels added on top of a base map)
• Calculating collision boxes at (or near) render time is much more accurate for pitched maps, line labels, sizes based on zoom functions, etc.
• Fade in/out animations are no longer limited to zoom operations (i.e. collisions related to pitch/pan/rotate will all be animated now)

The main structural changes are:

• CollisionTile is now a global object that lives only in the foreground. Insertions/queries into the collision index are now done entirely in viewport coordinates instead of tile coordinates.
• The "placement" step no longer happens in the background thread, and static buffers for symbols are created during the "prepare" step.
• The new foreground "placement" step is primarily responsible for updating a dynamic array of opacities that are used to animate collision states. For each vertex, the opacity state is expressed as an "opacity at time of last update" and a "target opacity". The symbol shaders animate towards the "target opacity" based on an animation start time set for each bucket.
• It's now necessary to place a hard time limit on how long an individual placement call can take, to avoid frame stutter -- so full placement can now be split across multiple frames.
• Collision features for lines are now represented in the collision grid as an array of circles, instead of as an array of boxes. This makes their behavior more stable under pitch/rotation operations.
• Because fading is now tied to "collision time" instead of to zoom level, we need a way to tell when two symbols in tiles of two different zoom levels are "the same symbol", so that we can avoid triggering a fade in/fade out when we cross between zoom levels. This is primarily handled by the new CrossTileSymbolIndex.

TODO:

• Handle re-ordering for overlapping symbols (requires re-generating static buffers on rotation -- might be time to address issue Fix label clipping across tile boundaries #2706?)
• Fix remaining test failures/update tests to reflect new behavior
  • Render tests
    • 10 remaining render test failures for symbol-avoid-edges/tile-clipping (current behavior is actually better)/overlap-sorting
  • Query tests
  • Unit tests
• Performance improvements:
  • Don't require full placement when a new tile is added -- instead, start progressive placement for the tile as soon as it's added, but don't start rendering until the first full placement has completed.
  • Allow progressive placement to pause at the bucket level. (Currently placement pauses at the layer level, where a single dense layer of street labels on a large viewport can take ~8ms on my MacBook Pro)
• Implement anti-aliasing for collision circle debugging (need them to look good for Studio)
• Extend collision detection to a buffer around the viewport (so that animations for changes around the edge can be mostly finished animating before they're actually visible)
• Update comments, rationalize class names (e.g. CollisionTile), remove dead code
• Rebase on master
• Flow-ify changes
• Handle wrapped tiles in CrossTileSymbolIndex
• Figure out home for grid_index_experimental (adds support for circle geometry queries; removes support for cross-thread serialization): either update https://github.com/mapbox/grid-index, make a new library, or just continue to include the code directly.
• Port to gl-native
  • Figure out how to continue doing tile-based collision detection in api-gl

This PR will enable further changes that we're not tackling yet:

• Collision detection for labels with rotation-alignment map.
• text-offset applied to collision detection
• Adding collision features with arbitrary geometry
• Cross-fading for layout property changes
• Multiple label placement (Variable text placement for point labels #5038 / https://github.com/mapbox/mapbox-gl-js/tree/multiplace)
• Label density reduction: Reduce density for labels based on distance from camera #5086
• Reduce data usage by creating vector tiles without collision buffers
• Fade out symbols from removed tiles (requires holding on to tile or at least symbol buckets over period of fade animation)

Latest Benchmark

/cc @ansis @nickidlugash @mollymerp

[nickidlugash]

@ChrisLoer A few notes based on my visual review of this branch today:

• Overall I think this is looking awesome 💯 Super excited to see this improved behavior on all our styles!

• I'm noticing a lot of labels that are initially visible and then disappear while panning. It looks like that is what would be addressed in this TODO item:

  Extend collision detection to a buffer around the viewport (so that animations for changes around the edge can be mostly finished animating before they're actually visible)

  How will we calculate how big this buffer needs to be? Would we approximate it based on the default collision fade duration, and a reasonable panning speed? Would we be able to make a buffer large enough to resolve this issue for the background in very pitched maps?

• text-offset for labels along lines isn't taken to account in collision. Is this going to be part of further work listed?:

  Collision detection for labels with rotation-alignment map.

• As I mentioned in chat, might be worth us thinking about whether collisionFadeDuration should be a map property (as currently implemented in this branch), or a global style property (or both)? I could see this being a useful property in both.

• Now that you've identified exactly what updates will and won't be part of this initial PR, I'll open a couple more tickets for specific future changes so we can start tracking those discussions separately.

[ChrisLoer]

How will we calculate how big this buffer needs to be? Would we approximate it based on the default collision fade duration, and a reasonable panning speed? Would we be able to make a buffer large enough to resolve this issue for the background in very pitched maps?

These are all good questions, and I don't know the answer. We should experiment with different values to find the best stability/performance tradeoff. I can try to make the padding a configurable value just so we can experiment more quickly.

text-offset for labels along lines isn't taken to account in collision

Actually, I think that's just a bug -- there are two failing test cases for that too. It's on my list to look into, I've just been mentally filing it under "fix tests".

As I mentioned in chat, might be worth us thinking about whether collisionFadeDuration should be a map property (as currently implemented in this branch), or a global style property (or both)? I could see this being a useful property in both.

👍

[ChrisLoer]

OK, the query tests are now passing and the ten remaining render test failures are all caused by the known avoid-edges/tile-clipping/overlapping-symbol-sorting issues. If we removed support for symbol-avoid-edges and stopped doing tile clipping for overlapping symbols, we'd be down to just two failures, both in icon-translate-anchor, caused by changing the sort order of overlapped symbols on a rotated tile.

text-offset for labels along lines isn't taken to account in collision

Actually, I think that's just a bug -- there are two failing test cases for that too. It's on my list to look into, I've just been mentally filing it under "fix tests".

I was too optimistic here. The cause of those test failures was a bug in handling collision boxes for single-glyph labels. Actually handling text-offset correctly in collisions for line labels isn't included in this PR, but it should be much more tractable after this PR goes in. I've added it to "further work".

[ChrisLoer]

Benchmark results overall look pretty good, but frame-duration has gone up. A modest increase in the average may be acceptable, but for the zoom 13 test it caused at least one frame to take longer than 16ms:

master b203a40	viewport-collision 1cff6c7
7.6 ms, 0% > 16 ms at zoom 13	8.4 ms, 1% > 16 ms at zoom 13

benchmark	master b203a40	viewport-collision 1cff6c7
map-load	168 ms	126 ms
style-load	70 ms	67 ms
buffer	1,002 ms	1,007 ms
fps	60 fps	60 fps
frame-duration	5.3 ms, 0% > 16ms	6 ms, 0% > 16ms
query-point	0.55 ms	0.59 ms
query-box	39.98 ms	39.50 ms
geojson-setdata-small	1 ms	2 ms
geojson-setdata-large	123 ms	109 ms

[ChrisLoer]

I have a local stash that replaces the "flattened" arrays of collision boxes and collision circles with arrays of javascript objects. It's much easier to read. However, it also seems to be slower. My test methodology was to rotate a pitched full screen map with lots of road labels while recording with the CPU profiler, and then look at the percent of render time spent on the relevant code (looking at ratios is more reproducible than looking at absolute render time):

Before:

Run	redoPlacement	placeCollisionCircles (self)
First	18.2%	8.5% (2.9%)
Second	16.7%	7.5% (2.4%)
Third	16.7%	7.4% (2.5%)

After:

Run	redoPlacement	placeCollisionCircles (self)
First	21.0%	11.5% (3.5%)
Second	18.6%	8.5% (2.7%)
Third	23.4%	13.5% (4.1%)

I don't have a good explanation for why this makes such a noticeable difference, but it looks like in this particular example it pushes up placement time by at least a millisecond.

[ChrisLoer]

I did four more test runs, this time with a build that represented circles/boxes as javascript objects, but with only number data members (i.e. instead of containing a Point member, they just contained an x and a y). Results looked pretty similar -- still a noticeable win for the earlier "flat" approach.

[mourner]

@ChrisLoer I also noticed that working with flat number arrays generally gives a huge boost to performance compared to arrays of objects. V8 treats them differently. Earcut and Delaunator both take advantage of this.

[ChrisLoer]

D'oh. I found that at least part of the cause of the flattened array being faster was that it had a bug that caused it not to place some collision circles that it should have (of course the bug derived from the awkwardness of the code!).

After fixing the bug, the difference is looking narrower, but the "flattened" approach is still reliably faster...

Before:

Run	redoPlacement	placeCollisionCircles (self)
First	22.1%	11.8% (2.3%)
Second	21.4%	11.0% (2.0%)
Third	20.7%	11.3% (2.0%)

After:

Run	redoPlacement	placeCollisionCircles (self)
First	22.7%	12.7% (4.0%)
Second	23.9%	13.5% (4.0%)
Third	22.6%	12.8% (2.5%)

[ChrisLoer]

@mourner That's good to know, thanks! @kkaefer had theorized that v8 would be able to be clever about recognizing that objects were "plain ol' data" and do the flattening internally, but I guess not, or at least not yet...

[ChrisLoer]

@nickidlugash I pushed a change that extends collision detection past the edge of the viewport by 100 pixels in all directions. Unfortunately, there's no getting around that doing collision over a broader area makes it significantly more expensive (the cost scales at least linearly with the number of symbols included in the detection area). I ended up at padding of 100px because the hit to collision times wasn't that bad, and it seemed to eliminate most of the notable cases of label instability at the edge of the viewport. Maybe you could evaluate if it's good enough? If we can get away with an even lower value, that'd be great, too (just change the viewportPadding constant in collision_index to try different values).

[ChrisLoer]

@nickidlugash I just pushed a change that draws the collision circles with anti-aliasing, so they should be "ready" for use in Studio. I could use your help figuring out the color strategy though. The current black and blue (with fading to indicate unused collision boxes/circles) works well for me, and I think it looks reasonable. We could go back to green and red, but those are really difficult for my kind to figure out. Overall, I think the decision is a lot easier because we used to have four colors for the various states, but now there are just two colors.

[ChrisLoer]

Thanks @anandthakker for making it easy to do a density plot (although I don't have the R skills you have, were you using ggplot?).

Overall, frame durations are looking pretty similar, but there's definitely a bit of a long tail that's been introduced with these changes.

(steeper/taller line is master, flatter/longer line is this branch)

[ChrisLoer]

I pushed a change to allow placement to pause per-bucket instead of just per-layer (although the pause/restart logic is a bit ugly 😬 ). On my machine even the most expensive/dense buckets don't take much more than a millisecond to place, so we can pretty effectively constrain how long placement takes. I've updated the benchmark at the top of the ticket, along with a density plot showing that the behavior is now looking pretty close to master.

The remaining flat-lined long tail represent the frames on which a new tile loads, in which case full placement has to take place (in this run, the longest frame-duration came out to 16.035ms, so that probably would have been a one frame stutter). We can't do a partial placement when new tiles come in because they may be higher or lower-zoom versions of tiles we're already showing and to avoid flicker we need to transfer opacities from the previous zoom level.

What we could potentially do to solve the tile loading problem is:

• Split collision detection apart from updating the opacity buffers
• Add a single-frame "commit" that updates all of the opacity buffers at once whenever a collision detection pass is finished
• Mark new tiles as non-renderable until they've been included in a completed collision detection pass

[ChrisLoer]

reloading and expired tiles present an extra challenge for the "don't render until you've done first placement" strategy, because in the case of reloading tiles, we want to keep rendering the old data until the updated tile is fully placed. That would require us to keep around two copies of the tile data during that period (the old one for rendering, the new one for processing the next placement).

@kkaefer (or @mourner), do you think we can get away with doing a full placement in those two (relatively uncommon) cases?

[ChrisLoer]

@ansis When you get back, I'd like to brainstorm with you about how to solve the overlapping-symbol-ordering problem, which I think is the only big thing left on the JS side. Ideally, I'd like to fix #2706 and not reopen #470.

• current viewport-collision implementation: Sort symbols on y-order using bearing of 0, and then never change them. Don't use tile clipping, only draw symbols whose anchor is in the tile we're currently drawing.
  • Pros:
    • Easy!
    • Fixes Fix label clipping across tile boundaries #2706.
    • Consistent ordering across all tiles (previously, asynchronous placement meant that tiles could change ordering independently)
  • Cons:
    • Rolls back Sort point features by y-axis #470: although ordering is consistent, it doesn't maintain the "higher icons are rendered on top" behavior during rotation
    • Ordering can get messed up at tile edges (a lower symbol in one tile can be drawn over a higher tile in an adjacent one)
• Re-generate sorted buffers in background: This would basically be a re-implementation of master's redoPlacement behavior, except its only purpose would be to sort buffers
  • Pros:
    • Preserve current behavior
  • Cons:
    • Asynchronous redoPlacement adds a ton of complexity to the code that we'd be preserving for just this one relatively minor case.
• Re-generate sorted buffers in foreground: We would serialize buffers with enough metadata to make it easy to do a sort/re-generate operation on the foreground at placement time (not more often because we don't want to constantly be re-uploading static buffers).
  • Pros:
    • Preserves master behavior, with modest improvement that sorting for all tiles would happen at the same time.
    • Preserves viewport-collision simplification of loading symbol tiles
  • Cons:
    • Still can't simultaneously fix Fix label clipping across tile boundaries #2706 and preserve Sort point features by y-axis #470
  • Variant 1: Keep master's tile-clipping behavior, thus ensuring that overlapping symbol order is consistent even at tile boundaries.
  • Variant 2: Stop tile-clipping and instead only draw symbols whose anchor is within the current tile. Solves all the problems of Fix label clipping across tile boundaries #2706, but introduces potential ordering inconsistencies at tile boundaries.
• Combine and sort symbol buffers on the foreground: Whenever we're finishing a placement on the foreground, we combine all of the symbol buffers for currently loaded tiles into one sorted buffer with a common coordinate system, and draw_symbols would use the unified buffer instead of per-tile buffers.
  • Pros:
    • Fixes Fix label clipping across tile boundaries #2706 and preserves Sort point features by y-axis #470 behavior
    • Reducing GL draw calls should be a performance win
    • Provides a relatively simple way to keep rendering current tiles/symbols while a new placement operation is in progress.
  • Cons:
    • Performance: sorting/transforming buffers could be expensive (this part might be amenable to backgrounding)
    • Performance: requires uploading more static buffers -- any time a tile is added or symbol ordering changes, the entire buffer has to be re-uploaded, vs. current behavior where only new tiles have to be uploaded.
    • Not clear what the right transformed coordinate system would be -- I'm guessing it would be something like "a tile zoomed out far enough to include all the tiles currently in the tree", but for each zoom level we'd lose a bit of precision. We can get three bits of precision back because we wouldn't need tile buffers any more (and what used to be the line normal bit is free). We can probably get away with dropping another couple bits of precision, but at some point we'd risk symbols starting to jump around as the coordinates get transformed.
    • Have to do all the implementation work just to find out if the hypothetical performance issues are a problem!

My inclination is to do "per-tile re-sort on foreground, without tile-clipping". I think the problems in #2706 are overall more severe than the potential for overlapping symbols to have their order messed up at tile boundaries -- but I may not have enough context for why that would be a serious regression. I think "combine and sort buffers on the foreground" is promising, but it would be a big change and this PR is already a lot to digest.

[nickidlugash]

I pushed a change that extends collision detection past the edge of the viewport by 100 pixels in all directions. Unfortunately, there's no getting around that doing collision over a broader area makes it significantly more expensive (the cost scales at least linearly with the number of symbols included in the detection area). I ended up at padding of 100px because the hit to collision times wasn't that bad, and it seemed to eliminate most of the notable cases of label instability at the edge of the viewport. Maybe you could evaluate if it's good enough? If we can get away with an even lower value, that'd be great, too

@ChrisLoer I reviewed this in one of the areas of the Mapbox Streets style with the most collisions (Manhattan at z16), with a 60deg pitch, with the default collision fade duration of 300ms, and I think it looks good enough 👍 I only tested it with manually panning/zooming/rotating/pitching the map, but for all practical speeds of doing these actions, the map handled this issue pretty well.

I haven't tested camera animations, but as we discussed offline, fly to animations where the zoom level + center is changing rapidly probably won't load new tiles fast enough to be very affected by this (but I guess we should test this?). The nature of animation used for turn-by-turn guidance seems like it is slow enough (and typically uses styles that have sparse enough labels) for this to not be an issue, despite the extreme pitch typically used. There may be use cases we haven't discussed that need very fast panning on pitched maps, but I think this viewport padding solution is fine to proceed with for now.

If performance is a big concern, we could decrease the padding a bit – a 50px padding was noticeably worse than 100px, but I thought 75px was still pretty reasonable.

[ChrisLoer]

@nickidlugash Thanks for taking the time to run those tests! Sounds like your impressions were broadly similar to mine. I think I'll stick with 100px vs 75px for now just to give the padding a little padding. ;)

[ChrisLoer]

My latest change tries to be more methodical about choosing an optimal number of cells for the grid-index on the size of the viewport. The results at the end of all the tweaking don't come out all that different on my device/viewport from what I was seeing with the earlier choice of "make a 20x20 grid", but they should be a little more robust to different viewport sizes. I evaluated choices by collecting the average time to complete a full placement (i.e. one placement split across multiple frames) during a 10 second 180 degree rotation in a 60 degree pitched map full of line labels using Nicki's label-placement-debug style (#13.86/33.9612/-118.3133/0/60):

Basically anything between 15px and 40px seemed to work pretty similarly, so I went with a 25px cell size.

[ChrisLoer]

After rebasing on master and combining with the expressions PR (#4777), the benchmark results look problematic. 😞

In both density plots, the flatter line is viewport-collision.

Both branches seem to have a longer tail than we'd like -- I'm not sure what we expect is causing that on the expressions side (I'm assuming the changes on master are because of #4777, that could be wrong). On the viewport-collision side, I know the tail can be long because it's sometimes necessary to synchronously perform a full placement. The tail also seemed to get fatter, and I'm not sure why. I haven't found any obvious regressions in performance otherwise (i.e. average placement time hasn't increased), but I'll poke into that more.

Before expressions

benchmark	master c48a56b	viewport-collision 67e3ca6
map-load	114 ms	82 ms
style-load	55 ms	54 ms
buffer	1,006 ms	977 ms
tile_layout_dds	962 ms	n/a
fps	59 fps	59 fps
frame-duration	5.2 ms, 0% > 16ms	5.7 ms, 0% > 16ms
query-point	0.55 ms	0.61 ms
query-box	41.54 ms	42.29 ms
geojson-setdata-small	1 ms	2 ms
geojson-setdata-large	106 ms	52 ms

After expressions

benchmark	master 8a674b8	viewport-collision 669c9c3
map-load	87 ms	83 ms
style-load	110 ms	108 ms
buffer	1,052 ms	1,083 ms
tile_layout_dds	1,493 ms	1,453 ms
fps	60 fps	60 fps
frame-duration	6 ms, 0% > 16ms	7.1 ms, 4% > 16ms
query-point	0.83 ms	1.03 ms
query-box	68.71 ms	74.89 ms
geojson-setdata-small	7 ms	9 ms
geojson-setdata-large	141 ms	104 ms
filter	n/a	n/a

/cc @anandthakker

[ChrisLoer]

I experimented with trying to cut the time spent on updating the dynamic opacity buffer by reducing 8 uint16s per glyph (opacity + target opacity for each of the four corners) to one uint32 per glyph. I packed the opacity information into one uint8 per vertex, and then I packed all four vertices into one uint32 (and just cast to uint8s when I bound the buffer), so that there'd be fewer assignments/multiplications on the javascript side.

It worked just fine, but it didn't make a noticeable dent in the time spent updating the buffers (watching in the profiler). Also, if making the buffers smaller improved rendering times, it was too small of a difference for me to be able to reliably pick up in the benchmarks.