Merge branch 'master' into x/visible-callback

dev-docs/RFCs/proposals/loader-worker-triangulation.md

@@ -0,0 +1,56 @@
+# RFC: Loader Worker Triangulation
+
+* **Author**: Felix Palmer
+* **Date**: Apr, 2021
+* **Status**: **Conceptual Draft**
+
+## Summary
+
+This RFC proposes shifting the triangulation that is performed by the `earcut`
+library on solid polygons in the MVTLayer in deck.gl to a web worker,
+allowing the costly operation to be performed off the UI thread.
+
+## Motivation
+
+For layers with a large number of polygons performance suffers
+as the triangulation operation performed as part of the `updateState`
+method. As this is invoked on the main thread, the UI is unresponsive while
+this operation is performed.
+
+## Observations prior to design proposal
+
+### Preprojection
+
+Deep within the solid polygon layer we find the call out to earcut, including an optional preprojection [layers/src/solid-polygon-layer/polygon.js]. The preprojection is invoked for LNGLAT coordinate system and thus whether it occurs is coupled to the layer and potentially viewport configuration
+
+When invoking the triangulation in the worker we need to specify whether to project or not. The MVTLayer is a geospatial layer and as such only needs to support either LOCAL or LNGLAT projections. Which of these to use, is already specified in the MVTLoader options.
+
+### Moving earcut to loaders.gl
+
+While I was hesitant about putting the triangulation within loaders.gl, the advantage of doing the work there (rather than in a worker launched within deck.gl) is that we only have to transfer the data to-and-from the worker once. If the worker was launched from the deck.gl code it would lead to two workers being used, adding latency.
+
+### Earcut (non)removal from deck.gl
+
+A number of different Layers (not just MVTLayer) draw polygons (and thus invoke earcut) and they pass their data in different formats, e.g. MVT/GeoJSON. Additionaly SolidPolygonLayer is designed to be a general-purpose layer and should support the consumer of the library in accepting just polygons (for example GeoJSON) as data, and thus the triangulation capability must be retained.
+
+## Design
+
+At a high level the loading routhines for the MVTLayer (binary version) are modified so that they also perform the triangulation, passing through the indices corresponding to the triangles to deck.gl. If present, deck.gl then applies these directly to buffers, rather than doing the triangulation again.
+
+Changes are required both in loaders.gl and deck.gl
+
+### loaders.gl
+
+MVTLoader is modified to automatically triangulate polygons when `binary` is set to true.
+
+The MVTLoader will then invoke earcut as part of the worker process and then return as part of the arrays it produces an additional array:
+
+- `triangles`, a set of indices into the positions array. 
+
+### deck.gl
+
+The PolygonTesselator already supports external indices. Thus the triangle data from loaders.gl will be passed through, and deck.gl will not needlessly recompute the triangles on the main thread. 
+
+## Future expansion
+
+The scope of this change is limited to the binary version of the MVTLoader & MVTLayer, but in principle other loaders could also supply the triangulation and be picked up by the Tesselator in deck.gl with minimal changes required. This should not be done without careful benchmarking, as the cost of passing the data to a worker may exceed the performance gains. In the case of the MVTLoader where the parsing already happens in a worker, this overhead is avoided.

docs/api-reference/aggregation-layers/heatmap-layer.md

@@ -78,9 +78,9 @@ Radius of the circle in pixels, to which the weight of an object is distributed.
 
 * Default: [colorbrewer](http://colorbrewer2.org/#type=sequential&scheme=YlOrRd&n=6) `6-class YlOrRd` <img src="https://deck.gl/images/colorbrewer_YlOrRd_6.png"/>
 
-Specified as an array of colors [color1, color2, ...]. Each color is an array of 3 or 4 values [R, G, B] or [R, G, B, A], representing intensities of Red, Green, Blue and Alpha channels.  Each intensity is a value between 0 and 255. When Alpha not provided a value of 255 is used.
+The color palette used in the heatmap, as an array of colors [color1, color2, ...]. Each color is in the format of `[r, g, b, [a]]`. Each channel is a number between 0-255 and `a` is 255 if not supplied.
 
-`colorDomain` is divided into `colorRange.length` equal segments, each mapped to one color in `colorRange`.
+See the `colorDomain` section below for how weight values are mapped to colors in `colorRange`.
 
 ##### `intensity` (Number, optional) ![transition-enabled](https://img.shields.io/badge/transition-enabled-green.svg?style=flat-square")
 
@@ -102,16 +102,24 @@ The `HeatmapLayer` reduces the opacity of the pixels with relatively low weight
 
 * Default: `null`
 
-Weight of each data object is distributed to to all the pixels in a circle centered at the object position, weight a pixel receives is inversely proportional to its distance from the center. Pixels that fall into multiple circles will have sum of all weights. And the weight of the pixel determines its color. When `colorDomain` is specified, all pixels with weight with in specified `colorDomain` will get mapped to `colorRange`, pixels with weight less than `colorDomain[0]` will fade out (reduced alpha) and pixels with weight more than `colorDomain[1]` will mapped to the highest color in `colorRange`.
+Controls how weight values are mapped to the `colorRange`, as an array of two numbers [`minValue`, `maxValue`].
+
+When `colorDomain` is specified, a pixel with `minValue` is assigned the first color in `colorRange`, a pixel with `maxValue` is assigned the last color in `colorRange`, and any value in between is linearly interpolated. Pixels with weight less than `minValue` gradually fade out by reducing alpha, until 100% transparency representing `0`. Pixels with weight more than `maxValue` are capped to the last color in `colorRange`.
+
+- If using `aggregation: 'SUM'`, values in `colorDomain` are interpreted as weight per square meter.
+- If using `aggregation: 'MEAN'`, values in `colorDomain` are interpreted as weight.
+
+When this prop is not specified, the maximum value is automatically determined from the current viewport, and the domain is set to [`maxValue * threshold`, `maxValue`]. This default behavior ensures that the colors are distributed somewhat reasonably regardless of the data in display. However, as a result, the color at a specific location is dependent on the current viewport and any other data points within view. To obtain a stable color mapping (e.g. for displaying a legend), you need to provide a custom `colorDomain`.
 
-When not specified, maximum weight (`maxWeight`) is auto calculated and domain will be set to [`maxWeight * threshold`, `maxWeight`].
 
 ##### `aggregation` (String, optional)
 
 * Default: `'SUM'`
 
 Operation used to aggregate all data point weights to calculate a pixel's color value. One of `'SUM'` or `'MEAN'`. `'SUM'` is used when an invalid value is provided.
 
+The weight of each data object is distributed to all the pixels in a circle centered at the object position. The weight that a pixel receives is inversely proportional to its distance from the center. In `'SUM'` mode, pixels that fall into multiple circles will have the sum of all weights. In `'MEAN'` mode, pixels that fall into multiple circles will have their weight calculated as the weighted average from all the neighboring data points. And the weight of the pixel determines its color. 
+
 ### Data Accessors
 
 ##### `getPosition` ([Function](/docs/developer-guide/using-layers.md#accessors), optional)

docs/api-reference/geo-layers/mvt-layer.md

@@ -119,14 +119,13 @@ On top of the [default options](/docs/api-reference/core/layer.md#loadoptions),
 
 ##### `binary` (Boolean, optional)
 
-* Default: false
+* Default: true
 
-Use tile data in [binary format](https://github.com/visgl/loaders.gl/blob/master/modules/gis/docs/api-reference/geojson-to-binary.md) to improve performance. It removes the need for serialization and deserialization of data transferred by the worker back to the main process.
+Use tile data in [binary format](https://github.com/visgl/loaders.gl/blob/master/modules/gis/docs/api-reference/geojson-to-binary.md) to improve performance (2-3x faster on large datasets). It removes the need for serialization and deserialization of data transferred by the worker back to the main process. 
 
 Remarks: 
 
 - It requires using `GeoJsonLayer` in the `renderSubLayers` callback.
-- In binary format, there are some rendering issues with polygons or multipolygons which contain holes (the holes won't appear as expected), we're working to fix it for the next release.
 
 ### Callbacks
 

docs/api-reference/geo-layers/tile-3d-layer.md

@@ -196,6 +196,14 @@ When [`picking`](/docs/developer-guide/custom-layers/picking.md) is enabled, `in
   - `url`: the url of the failed tile.
   - `message`: the error message.
 
+##### `_getMeshColor` (Function, optional)
+`_getMeshColor` is a function which allows to change color of mesh based on properties of [tileHeader](https://loaders.gl/docs/specifications/category-3d-tiles#tileheader-fields) object.
+It recieves `tileHeader` object as argument and return type is array of [r, g, b] values in the 0-255 range.
+This value is only applied when tile format is `mesh`.
+Can be used only for I3S debugging purposes.
+
+- Default: `_getMeshColor: (tileHeader) => [255, 255, 255]`
+
 ## Sub Layers
 
 The Tile3DLayer renders the following sublayers based on tile [format](https://github.com/AnalyticalGraphicsInc/3d-tiles/tree/master/specification#introduction):

docs/upgrade-guide.md

@@ -19,6 +19,7 @@ The module entry point is now only lightly transpiled for the most commonly used
 - `GeoJsonLayer`'s `lineJointRounded` prop now only controls line joints. To use rounded line caps, set `lineCapRounded` to `true`.
 - Dashed lines via `PathStyleExtension` now draw rounded dash caps if `capRounded` is `true`.
 - `@deck.gl/geo-layers` now requires `@deck.gl/extensions`, due to `ClipExtension` dependency.
+- `HeatmapLayer`'s `colorDomain` prop has redefined the unit of its values. See updated layer documentation for details.
 - `MVTLayer`'s `binary` prop is now set to `true` by default.
 
 ### onError Callback

modules/aggregation-layers/src/heatmap-layer/heatmap-layer.js

@@ -95,7 +95,7 @@ export default class HeatmapLayer extends AggregationLayer {
       return;
     }
     super.initializeState(DIMENSIONS);
-    this.setState({supported: true});
+    this.setState({supported: true, colorDomain: DEFAULT_COLOR_DOMAIN});
     this._setupTextureParams();
     this._setupAttributes();
     this._setupResources();
@@ -133,23 +133,6 @@ export default class HeatmapLayer extends AggregationLayer {
       this._updateColorTexture(opts);
     }
 
-    if (oldProps.colorDomain !== props.colorDomain || changeFlags.viewportChanged) {
-      const {viewport} = this.context;
-      const {weightsScale} = this.state;
-      const domainScale = (viewport ? 1024 / viewport.scale : 1) * weightsScale;
-      const colorDomain = props.colorDomain
-        ? props.colorDomain.map(x => x * domainScale)
-        : DEFAULT_COLOR_DOMAIN;
-      if (colorDomain[1] > 0 && weightsScale < 1) {
-        // Hack - when low precision texture is used, aggregated weights are in the [0, 1]
-        // range. Scale colorDomain to fit.
-        const max = Math.min(colorDomain[1], 1);
-        colorDomain[0] *= max / colorDomain[1];
-        colorDomain[1] = max;
-      }
-      this.setState({colorDomain});
-    }
-
     if (this.state.isWeightMapDirty) {
       this._updateWeightmap();
     }
@@ -459,15 +442,26 @@ export default class HeatmapLayer extends AggregationLayer {
   }
 
   _updateWeightmap() {
-    const {radiusPixels} = this.props;
+    const {radiusPixels, colorDomain, aggregation} = this.props;
     const {weightsTransform, worldBounds, textureSize, weightsTexture, weightsScale} = this.state;
     this.state.isWeightMapDirty = false;
 
-    // #5: convert world bounds to common using Layer's coordiante system and origin
+    // convert world bounds to common using Layer's coordiante system and origin
     const commonBounds = this._worldToCommonBounds(worldBounds, {
       useLayerCoordinateSystem: true
     });
 
+    if (colorDomain && aggregation === 'SUM') {
+      // scale color domain to weight per pixel
+      const {viewport} = this.context;
+      const metersPerPixel =
+        (viewport.distanceScales.metersPerUnit[2] * (commonBounds[2] - commonBounds[0])) /
+        textureSize;
+      this.state.colorDomain = colorDomain.map(x => x * metersPerPixel * weightsScale);
+    } else {
+      this.state.colorDomain = DEFAULT_COLOR_DOMAIN;
+    }
+
     const uniforms = {
       radiusPixels,
       commonBounds,

modules/core/src/controllers/controller.js

@@ -19,12 +19,19 @@
 // THE SOFTWARE.
 
 /* eslint-disable max-statements, complexity */
-import TransitionManager from './transition-manager';
+import TransitionManager, {TRANSITION_EVENTS} from './transition-manager';
+import LinearInterpolator from '../transitions/linear-interpolator';
 
 const NO_TRANSITION_PROPS = {
   transitionDuration: 0
 };
 
+const LINEAR_TRANSITION_PROPS = {
+  transitionDuration: 300,
+  transitionEasing: t => t,
+  transitionInterruption: TRANSITION_EVENTS.BREAK
+};
+
 const DEFAULT_INERTIA = 300;
 const INERTIA_EASING = t => 1 - (1 - t) * (1 - t);
 
@@ -48,6 +55,15 @@ export default class Controller {
       onViewStateChange: this._onTransition.bind(this),
       onStateChange: this._setInteractionState.bind(this)
     });
+
+    const linearTransitionProps = this.linearTransitionProps;
+    this._transition = linearTransitionProps && {
+      ...LINEAR_TRANSITION_PROPS,
+      transitionInterpolator: new LinearInterpolator({
+        transitionProps: linearTransitionProps
+      })
+    };
+
     this._events = null;
     this._interactionState = {
       isDragging: false
@@ -61,6 +77,10 @@ export default class Controller {
     this.setProps(options);
   }
 
+  get linearTransitionProps() {
+    return null;
+  }
+
   set events(customEvents) {
     this.toggleEvents(this._customEvents, false);
     this.toggleEvents(customEvents, true);
@@ -688,8 +708,23 @@ export default class Controller {
     return true;
   }
 
-  _getTransitionProps() {
-    // Transitions on double-tap and key-down are only supported by MapController
-    return NO_TRANSITION_PROPS;
+  _getTransitionProps(opts) {
+    const {_transition} = this;
+
+    if (!_transition) {
+      return NO_TRANSITION_PROPS;
+    }
+
+    // Enables Transitions on double-tap and key-down events.
+    return opts
+      ? {
+          ..._transition,
+          transitionInterpolator: new LinearInterpolator({
+            ...opts,
+            transitionProps: this.linearTransitionProps,
+            makeViewport: this.controllerState.makeViewport
+          })
+        }
+      : _transition;
   }
 }

modules/core/src/controllers/first-person-controller.js

@@ -1,8 +1,6 @@
 import Controller from './controller';
 import ViewState from './view-state';
 import {mod} from '../utils/math-utils';
-import LinearInterpolator from '../transitions/linear-interpolator';
-import {TRANSITION_EVENTS} from './transition-manager';
 
 import {Vector3, _SphericalCoordinates as SphericalCoordinates, clamp} from 'math.gl';
 
@@ -15,13 +13,6 @@ const DEFAULT_STATE = {
   minPitch: -90
 };
 
-const LINEAR_TRANSITION_PROPS = {
-  transitionDuration: 300,
-  transitionEasing: t => t,
-  transitionInterpolator: new LinearInterpolator(['position', 'pitch', 'bearing']),
-  transitionInterruption: TRANSITION_EVENTS.BREAK
-};
-
 class FirstPersonState extends ViewState {
   constructor({
     /* Viewport arguments */
@@ -303,8 +294,7 @@ export default class FirstPersonController extends Controller {
     super(FirstPersonState, props);
   }
 
-  _getTransitionProps() {
-    // Enables Transitions on double-tap and key-down events.
-    return LINEAR_TRANSITION_PROPS;
+  get linearTransitionProps() {
+    return ['position', 'pitch', 'bearing'];
   }
 }

modules/core/src/controllers/globe-controller.js

@@ -3,15 +3,6 @@ import Controller from './controller';
 
 import {MapState} from './map-controller';
 import {mod} from '../utils/math-utils';
-import LinearInterpolator from '../transitions/linear-interpolator';
-import {TRANSITION_EVENTS} from './transition-manager';
-
-const LINEAR_TRANSITION_PROPS = {
-  transitionDuration: 300,
-  transitionEasing: t => t,
-  transitionInterpolator: new LinearInterpolator(['longitude', 'latitude', 'zoom']),
-  transitionInterruption: TRANSITION_EVENTS.BREAK
-};
 
 class GlobeState extends MapState {
   // Apply any constraints (mathematical or defined by _viewportProps) to map state
@@ -44,8 +35,7 @@ export default class GlobeController extends Controller {
     this.touchRotate = false;
   }
 
-  _getTransitionProps() {
-    // Enables Transitions on double-tap and key-down events.
-    return LINEAR_TRANSITION_PROPS;
+  get linearTransitionProps() {
+    return ['longitude', 'latitude', 'zoom'];
   }
 }