Full screen symbol flicker on pitched map with text-translate

[jfirebaugh]

http://jsbin.com/hinivotoqi/edit?html,output

Rotate / pan / zoom enough (mostly rotating), and you should see full-screen flashes the same color as the text. The presence of the text-translate property seems to be key -- without it, I haven't been able to reproduce any flickering.

cc @malwoodsantoro @ChrisLoer

[ChrisLoer]

I put in a fix for behavior very similar to this in #6060 (I just saw the behavior as I was testing, this is different from #6041 which was the main object of that PR):

mapbox-gl-js/src/shaders/collision_circle.vertex.glsl

Lines 20 to 23 in e0da3b3

	highp float collision_perspective_ratio = clamp(
	0.5 + 0.5 * (u_camera_to_center_distance / camera_to_anchor_distance),
	0.0, // Prevents oversized near-field circles in pitched/overzoomed tiles
	4.0);

In that case, what was happening was that for a long vertically-oriented label, the perspective ratio was calculated based on the anchor, but the actual circle was well above the anchor. So when the anchor was way below the screen, the circle would get a giant perspective ratio, and since it was closer to the viewport it was able to actually expand to fill the viewport.

In this case, I think what's happening is that the translation is getting exaggerated for anchor points that are well below the bottom of the viewport, because pixelsToTileUnits is only an approximation once you're pitched (say you're at the bottom of the viewport on a 60 degree pitched map, and the tiles are currently showing at their ideal zoom level: you'll get a pixel to tile unit ratio of 8, but in fact the ratio at that point will be much less).

mapbox-gl-js/src/render/draw_symbol.js

Lines 112 to 117 in e0da3b3

	gl.uniformMatrix4fv(program.uniforms.u_matrix, false, painter.translatePosMatrix(coord.posMatrix, tile, translate, translateAnchor));
	const s = pixelsToTileUnits(tile, 1, painter.transform.zoom);
	const labelPlaneMatrix = symbolProjection.getLabelPlaneMatrix(coord.posMatrix, pitchWithMap, rotateWithMap, painter.transform, s);
	const glCoordMatrix = symbolProjection.getGlCoordMatrix(coord.posMatrix, pitchWithMap, rotateWithMap, painter.transform, s);
	gl.uniformMatrix4fv(program.uniforms.u_gl_coord_matrix, false, painter.translatePosMatrix(glCoordMatrix, tile, translate, translateAnchor, true));

u_matrix gets a translation based on the pixelsToTileUnits approximation, while u_gl_coord_matrix gets a translation done directly in viewport units (that 'true' passed into translatePosMatrix means inViewportUnits). In the shader, the perspective ratio is based off of u_matrix (the anchor), but the actual layout of the glyphs is done with u_gl_coord_matrix. This discrepancy is what allows the calculation to go berserk (the giant perspective ratio happens as you get close to the plane of the camera, but as long as everything is in sync, it will never grow large enough to enter the viewport).

I just looked back to my discussion with @ansis about this in January, and found this prophetic quote:

But I’m wondering why we don’t ever see that cause a problem with other shaders on pitched/overzoomed tiles (or whether maybe we do, it’s just hard to notice). And then I’m theorizing that it just doesn’t come up for point labels because the y value of the anchor is usually not that far off from the y value of the glyph center.

I think the best fix may be just to clamp the perspective ratio every time we generate it in a shader. It's a weird kind of late-stage fix, but it has the advantage of being easy to reason about.

[ChrisLoer]

Fixed in #11487.