Skip to content
This repository has been archived by the owner on Aug 8, 2023. It is now read-only.

[core] Global/Viewport Collision Detection #10436

Merged
merged 21 commits into from
Nov 17, 2017
Merged

Conversation

ChrisLoer
Copy link
Contributor

@ChrisLoer ChrisLoer commented Nov 9, 2017

This is the native port of mapbox/mapbox-gl-js#5150: it switches us to doing collision detection globally, in a grid based on viewport coordinates instead of tile coordinates. This allows us to solve many problems, but mainly boils down to (1) more accurate collision detection, and thus denser/more-stable labeling, and (2) cross-source collision detection.

This is a squashed/rebased version of WIP PR #10103.

Differences from JS

The core collision detection algorithm is a very close port of the JS code, and the shader code is fully shared. The biggest differences are in the CrossTileSymbolIndex code and the render thread placement logic. These differences were partly driven by the requirements of native, but mostly by us trying to make improvements based on hindsight.

  • CrossTileSymbolIndex uses the same logic to find duplicates, but instead of trying to update opacity information directly, it now just tags duplicates with a shared crossTileID. The foreground placement code then runs placement and updates opacities using those IDs to handle duplicates. This approach makes it easier to separate opacity updates from the actual placement logic, which makes it so that we can add tiles to the render tree w/ transparent labels without blocking waiting for a full placement to complete.
  • All placements are "full placements" in this PR -- we don't have an "interruptible" concept as in JS. @ansis made a branch, start-collision-async that successfully runs placement on a background thread, but so far it doesn't look like a significant performance win (less foreground CPU, more total CPU, same frame-rate), so we haven't merged it.

TODO

Performance

mbgl-benchmark looks pretty similar, with the exception of API_renderStill_reuse_map, which takes about twice as long (this makes sense because we do full placement for each still frame, wheres before we could re-use the already-completed tile placements).

  • Store results somewhere public?

I've done the bulk of performance testing using the iOS bench app running on my iPhone 5s and an ancient iPad. I've modified the bench app to include two challenging scenes:

  • A ten second pitched rotation animation looking at LA city streets
  • A ten second animation zooming into LA city streets

I added printf-style timing logs to the render code to collect detailed frame timing information for analysis in a spreadsheet.

  • Store results somewhere public?

Both master and viewport-collision clocked in at 60 FPS for the zoom animation. Delving into the individual frame timing data, I could see that there were frames in which the cost of placement drove the frame time over 16ms -- but most of the skipped frames were driven by time spent in the render pass, so that overall the two branches looked pretty similar.

viewport-collision was far superior to master in the pitched rotation animation, coming in at 58-59 FPS vs 41-43 FPS. Again, although placement showed up as a significant cost in some frames, render time was far more important. My guess is that frequently re-uploading static symbol buffers is responsible for trashing the performance on master (@mourner?).

Binary size looks mostly unaffected. Running bloaty on release builds of the macOS framework showed a -0.7% change in binary size, and at a quick glance the individual function deltas seemed reasonable (less code for running in the background, more for running in the foreground).

/cc @ansis @jfirebaugh @kkaefer @anandthakker @mollymerp

Copy link
Contributor

@kkaefer kkaefer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could be looking into replace std::unordered_* with their ordered counterparts to see if the different runtime characteristic (more uniform timing) has a beneficial effect, e.g. for reducing jank.

class IndexEntry {
Point<float> anchorPoint;

};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class is unused (and has only private members)

};

class IndexedSymbolInstance {
public:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: we're not indenting visibility keywords, and class members are only indented once, so overall the indentation level is one too many.

}

void Placement::updateLayerOpacities(RenderSymbolLayer& symbolLayer) {
std::unordered_set<uint32_t> seenCrossTileIDs;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could be using std::set. uint32_t already has a well defined sort order, so using a set removes the need to "hash" integers.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we somehow estimate the size of this set beforehand so that we can reserve some memory as a larger chunk instead of having multiple smaller allocations?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Switched to std::set.

Without actually iterating over the tiles/buckets, I don't think we can know how many cross tile IDs will find. We could start with a guess reservation like "100", but I think that kind of change would only make sense if profiling led us to do it.

@@ -134,14 +116,22 @@ void GeometryTile::setLayers(const std::vector<Immutable<Layer::Impl>>& layers)
worker.invoke(&GeometryTileWorker::setLayers, std::move(impls), correlationID);
}

void GeometryTile::setShowCollisionBoxes(const bool showCollisionBoxes_) {
if (showCollisionBoxes != showCollisionBoxes_) {
showCollisionBoxes = showCollisionBoxes_;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: indentation is off by one space ;)

MapMode mapMode;
TimePoint commitTime;

std::unordered_map<uint32_t,PlacementPair> placements;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: space after comma

, y(y_)
{}

size_t index;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make these fields constant?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 Making them constant makes the class non-swappable and thus non-sortable. Is there a way to get around this with an explicit move constructor or something like that?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah ok; I haven't found a good way around that, since we need modifiable properties for movable types.

for (auto& element : circleElements) {
if (resultFn(element.first, convertToBox(element.second))) {
return;
};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: no semicolon after block

@@ -18,16 +18,17 @@
namespace mbgl {

FeatureIndex::FeatureIndex()
: grid(util::EXTENT, 16, 0) {
: grid(util::EXTENT, util::EXTENT, util::EXTENT >> 5) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why >> 5?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Uh, I'm not sure -- that's dividing it into a 32x32 grid, when it was a 16x16 grid before. Both cell sizes probably perform pretty similarly, but I'll revert to 16x16 and get rid of the confusing bit shift operator.

@kkaefer
Copy link
Contributor

kkaefer commented Nov 10, 2017

I'm seeing situations where zooming in across a tile boundary results in most (but not all!) labels vanishing without fading, then different labels are fading in. It looks like the algorithm retains the labels that it can match across tiles of different zoom levels. For the labels we can't match, can we fade those out instead of removing them instantly?

3 successive screenshots

@kkaefer
Copy link
Contributor

kkaefer commented Nov 10, 2017

Similarly, I'm also sometimes labels fading forth and back between different tiles(?):

6 successive screenshots

a1 a2 a3 a4 a5 a6

@ChrisLoer
Copy link
Contributor Author

I'm seeing situations where zooming in across a tile boundary results in most (but not all!) labels vanishing without fading, then different labels are fading in. It looks like the algorithm retains the labels that it can match across tiles of different zoom levels.

This looks like expected (although suboptimal) behavior, unless I'm missing something. The labels that are popping in and out are line labels, and since we do line label placement independently for each tile, it's rare for their anchors to match up exactly enough for the cross-tile-fading algorithm to pick them up. It would be great to address this issue, but I think it's outside the scope of this PR. Status quo on master is that the old line labels will instantly disappear and be replaced by new ones.

For the labels we can't match, can we fade those out instead of removing them instantly?

We'd like to do that, but I think it's also outside the scope of this PR. The tricky part (not terrible, but not trivial) is adjusting the retain/renderable algorithm to support an additional state for tiles ("render only symbol buckets until fade is complete"). When we looked at this on JS, we decided that doing a one-sided fade (fade in, but not out) still looked smoother than no fading at all -- although I don't know how sure we are about that decision (@nickidlugash?).


if (symbolInstance.placedTextIndex) {
PlacedSymbol& placedSymbol = bucket.text.placedSymbols.at(*symbolInstance.placedTextIndex);
const float fontSize = evaluateSizeForFeature(partiallyEvaluatedTextSize, placedSymbol);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't use the font size for point features, so we could avoid the calculation here. The fact that we don't dynamically evaluated font size for point feature collision boxes means they're a little off from their ideal size (and if a label has its font size changing with zoom, you can see a little jump in the box as you cross integer zoom levels), but the difference is usually pretty minor.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fact that we don't dynamically evaluated font size for point feature collision boxes means they're a little off from their ideal size

Should we use the exact size?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤷‍♂️ I'm not sure. It seems like a minor fidelity win in exchange for a minor performance cost.

Copy link
Contributor

@jfirebaugh jfirebaugh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A point of divergence that took me a minute to understand was that "symbol layout" is used to refer to different subsets of the process between JS and native.

In native, SymbolLayout is a class, instances of which (prior to this PR) form part of the persistent state of GeometryTileWorker. This state tracked information that was needed between subsequent background placement operations, and is now likely no longer necessary. SymbolLayout also handled the first steps of the process: calculating the needed glyphs and icon for each feature, and merging line geometries as appropriate. When the glyphs and icon data comes back from the main thread, SymbolLayout is then responsible for creating the SymbolBucket.

In JS, the SymbolBucket class itself handles the initial glyph and icon calculation, and then the bucket is passed in to code in symbol_layout.js which fills it out.

Doesn't have to be done in this PR, but if you're tracking things to un-diverge, this would be a good thing to add. Looking at the requirements now, I think a good way to structure it would be to extract the initial glyph and icon calculations to a plain old function that takes as inputs the layout properties and tile features, and returns symbol features and icon and glyph dependencies. The latter would be requested from the main thread, and when the response is received, the former would be passed into another function to create the bucket. Doesn't look like the bucket itself needs a features member.

@ChrisLoer
Copy link
Contributor Author

@kkaefer Your comments have gotten me to pay a lot more attention to the transition between two zoom levels in cities with dense road networks. There are a lot of cases where road labels have relatively small position changes between tiles, and on master the non-animated transition from one position to the other is not that jarring. On this branch there's more of a "blink" effect because they disappear before fading back in. Being able to do a cross-fade would help here, but I also wonder if there might be some heuristics we could develop for when to just skip the fade animation entirely.

if (symbolInstance.hasIcon) {
auto opacityVertex = SymbolOpacityAttributes::vertex(opacityState.icon.placed, opacityState.icon.opacity);
if (symbolInstance.iconQuad) {
bucket.icon.opacityVertices.emplace_back(opacityVertex);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before a known quantity of insertions, we could try calling reserve

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The standard gives clear the option to release memory, but I don't think it usually does, which means these vectors will effectively be pre-reserved at the time they're initially built up on the background thread.

matrix::multiply(posMatrix, projMatrix, posMatrix);

// Two versions of the query here just because util::polygonIntersectsPolygon requires
// GeometryCoordinates (based on int16_t). Otherwise, work with floats.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we somehow fold this into just one query?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our intersection test code is integer based, which is kind of an awkward fit now that we're using float/pixel units in the grid. I'll push a change that pulls the float->integer conversion out into a separate function so it doesn't interrupt the legibility of queryRenderedSymbols so much.

@@ -11,7 +11,8 @@ using EnumType = uint32_t;

enum class MapMode : EnumType {
Continuous, // continually updating map
Still, // a once-off still image
Static, // a once-off still image of an arbitrary viewport
Tile // a once-off still image of a single tile
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting idea!

, y(y_)
{}

size_t index;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah ok; I haven't found a good way around that, since we need modifiable properties for movable types.


if ((writingModes == WritingModeType::Vertical) ?
(firstPoint.y < lastPoint.y) :
(firstPoint.x > lastPoint.x)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume this code is run after we do text shaping so that all of our horizontal text runs have been converted to LTR?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, text shaping takes place much earlier in the layout phase in the background.

bool dirty = true;

mbgl::optional<gl::Texture> texture;
};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

\o/

@@ -81,14 +81,13 @@ class RenderLayer {

friend std::string layoutKey(const RenderLayer&);

// Stores current set of tiles to be rendered for this layer.
std::vector<std::reference_wrapper<RenderTile>> renderTiles;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does this have to be public?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Placement and CrossTileSymbolIndex both need (non-const) access to the set of render tiles -- this is how they go through updating all the opacity information.

It'd just be a band-aid, but maybe it would be better to explicitly "friend" them instead of making renderTiles fully public?

CrossTileSymbolLayerIndex::CrossTileSymbolLayerIndex() {
}

uint32_t CrossTileSymbolLayerIndex::maxCrossTileID = 0;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the strategy for keeping this static value from overflowing when the map runs for a long time?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ansis won me over with some back-of-the-envelope calculations suggesting at least many months of continuous panning of symbol-dense tiles before overflow would allow for the possibility of collisions.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also, it's only actually a problem if a label with that id has stuck around in the index for all that time with all those changes, which is unlikely

stale = false;
if (mapMode == MapMode::Continuous) {
// Only set in continuous mode because "now" isn't defined in still mode
recentUntil = now + Duration(std::chrono::milliseconds(300));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we make this configurable?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd lean against making placement frequency configurable with the idea that it's hard to reason about the effects of changing it.


namespace mbgl {

class RenderSymbolLayer;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Indentation

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@@ -75,11 +75,18 @@ class Context : private util::noncopyable {
}

template <class DrawMode>
IndexBuffer<DrawMode> createIndexBuffer(IndexVector<DrawMode>&& v) {
IndexBuffer<DrawMode> createIndexBuffer(IndexVector<DrawMode>&& v, const BufferUsage usage=BufferUsage::StaticDraw) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: spaces before and after =.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

~GeometryTileWorker();

void setLayers(std::vector<Immutable<style::Layer::Impl>>, uint64_t correlationID);
void setData(std::unique_ptr<const GeometryTileData>, uint64_t correlationID);
void setPlacementConfig(PlacementConfig, uint64_t correlationID);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any plans to take advantage of the work to reduce the state space of GeometryTileWorker? If we change collision box toggling to do a full tile relayout, we should be able to eliminate the "placement" side of the state transition diagram. (If that's for a later PR, can you at least update the block comment in the cpp file to indicate that "placement" is vestigial?)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I definitely want to simplify GeometryTileWorker by removing the two-phase loading and all the accompanying state transitions. I started in on making those changes and realized it would significantly increase the footprint of this PR, so I decided it makes sense to defer to a follow-up.

I'll add an explanatory comment for now.

@@ -132,10 +127,11 @@ class GeometryTile : public Tile, public GlyphRequestor, ImageRequestor {
optional<PremultipliedImage> iconAtlasImage;

std::unordered_map<std::string, std::shared_ptr<Bucket>> symbolBuckets;
std::unique_ptr<CollisionTile> collisionTile;
std::unordered_map<std::string, std::unique_ptr<SymbolLayout>> symbolLayouts;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unused (so the added include can be removed too).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, good catch.

@@ -607,7 +622,7 @@ void Renderer::Impl::render(const UpdateParameters& updateParameters) {

observer->onDidFinishRenderingFrame(
loaded ? RendererObserver::RenderMode::Full : RendererObserver::RenderMode::Partial,
updateParameters.mode == MapMode::Continuous && (hasTransitions() || frameHistory.needsAnimation(util::DEFAULT_TRANSITION_DURATION))
updateParameters.mode == MapMode::Continuous && (hasTransitions(parameters.timePoint))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: remove the surrounding parentheses too.

@@ -105,6 +105,9 @@ class Renderer::Impl : public GlyphManagerObserver,
std::unordered_map<std::string, std::unique_ptr<RenderLayer>> renderLayers;
RenderLight renderLight;

std::unique_ptr<CrossTileSymbolIndex> crossTileSymbolIndex;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do these need to be pointers?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CrossTileSymbolIndex definitely doesn't need to be, I'll change that.

@ansis can say better, but I think the idea with using unique_ptr<Placement> is that we can have one Placement "in progress" while the previous one is still being used, and then efficiently swap out the new one when it finishes. That's the approach the start-collision-async branch uses. I think we're interested in continuing to experiment with the asynchronous approach, so it might be nice to leave this in for now?

}

template <class DrawMode>
void draw(gl::Context& context,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix whitespace.


// Only used for symbol features
std::string sourceID;
uint8_t z;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use CanonicalTileID instead of individual x y z members.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same problem as here #10436 (comment): CanonicalTileID has const members and thus prevents swap-ability and thus sort-ability.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should make them non-const in CanonicalTileID. (I'm beginning to think const member values are largely an anti-pattern.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

textCollisionFeature(line_, anchor, shapedTextOrientations.second ?: shapedTextOrientations.first, textBoxScale, textPadding, textPlacement, indexedFeature),
iconCollisionFeature(line_, anchor, shapedIcon, iconBoxScale, iconPadding, iconPlacement, indexedFeature),
textCollisionFeature(line_, anchor, shapedTextOrientations.first, textBoxScale, textPadding, textPlacement, indexedFeature, overscaling),
iconCollisionFeature(line_, anchor, shapedIcon, iconBoxScale, iconPadding, SymbolPlacementType::Point, indexedFeature),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you comment why SymbolPlacementType::Point is hard-coded for icons?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

symbolInstances.size(),
textBoxScale, textPadding, textPlacement, textOffset,
iconBoxScale, iconPadding, iconOffset,
glyphPositionMap, indexedFeature, index, feature.text ? *feature.text : std::u16string{}, overscaling);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

feature.text.value_or(std::u16string())

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

This class is responsible for tracking which symbols are "the same" between tiles at different zoom levels, so that symbol opacities (fade animations) can be copied smoothly between tiles.
 - Adds early exiting "hitTest" query for fast collision detection
 - GridIndex now determines cell count separately for x and y axes based on grid dimensions.
Responsible for running global collision detection/symbol placement algorithm and updating symbol opacity buffers accordingly.
 - Switches from tile to viewport coordinates
 - Represents line labels with circle geometries
 - Projects line labels at collision detection time to improve accuracy
 - Adapts tile-based symbol queries to viewport coordinates
ChrisLoer and others added 12 commits November 17, 2017 08:40
 - CollisionTile
 - FrameHistory
 - PlacementConfig
`Tile` makes sure the symbols in the resulting tile are tileable while
symbols in `Still` match rendering in `Continuous` mode.
…creen.

Don't mark items that are outside the collision grid range as placed.
Requires new ignore because GL JS issue #5654 allows insertion of symbols outside the CollisionIndex range, and those symbols can cascade in to affect items within the viewport.
This is necessary so that when there are mixed zoom levels, child symbols get placed before parent symbols.
Update ignore links to specific issues.
Bump mapbox-gl-js version to get latest text-pitch-spacing test.
 Hold onto tiles after they've been removed from the render tree long enough to run a fade animation on their symbols.
@ChrisLoer ChrisLoer merged commit 25cfb58 into master Nov 17, 2017
@jfirebaugh jfirebaugh deleted the viewport-collision branch March 14, 2018 16:30
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants