Introduce a deriveIntegerId vector source option

[mourner]

A proof of concept that introduces a deriveIntegerId option for vector tile sources. It allows you to specify an existing feature property (that may be a string) to generate a corresponding integer feature ID using murmur3 hashing, enabling feature-state for vector tile sources without integer IDs but with unique string ids in properties (e.g. administrative boundary ids like "CN.A.230000.1").

To use it, add a deriveIntegerId: <propertyName> option to the source:

map.addSource('test', {
    'type': 'vector',
    'url': 'mapbox://foo.bar`),
    'deriveIntegerId': 'pk'
});

Seems to work well for an admin boundaries sample I tested this on. If the concept looks good, I'll work on adding tests and addressing any review comments.

TODO:

• Fix querySourceFeatures not including derived ids.
• Add query tests for feature-state with derived IDs.

Launch Checklist

• briefly describe the changes in this PR
• write tests for all new functionality
• document any changes to public APIs
• post benchmark scores
• manually test the debug page
• tagged @mapbox/studio if this PR includes style spec changes

[peterqliu]

@mourner how does the developer experience change? Can we simply remove the type restriction for the id property, or does the user have to specify that hashing is required?

(might need a docs update)

[mourner]

@peterqliu oh, good point — updated the description. You have to explicitly enable it and specify a property name to use.

[asheemmamoowala]

First of all - this is a mind-blowingly simple solution!

Couple thoughts:

• naming: deriveIntegerId doesn't seem very intuitive. geojson-vt uses promoteId for the same option.
• source-layer. The tileset from the example below uses different property names for the state and county source layers. Should this option be exposed per source-layer? Would that be necessary for composited tiles?

It works great with the vector source used in: https://docs.mapbox.com/mapbox-gl-js/example/updating-choropleth/

[mourner]

• naming: deriveIntegerId doesn't seem very intuitive. geojson-vt uses promoteId for the same option.

@asheemmamoowala I initially named it promoteId, but then thought that it would be misleading because the semantics are different. promoteId implies that you "promote" the same value (already a unique integer id) from feature.properties to feature.id, while here, the resulting id is derived, so it won't be the same when the property is anything but an unsigned integer.

Or do you think we should change the semantics in geojson-vt instead, adding murmur hashing there?

• source-layer. The tileset from the example below uses different property names for the state and county source layers. Should this option be exposed per source-layer? Would that be necessary for composited tiles?

Good point. We could either accept a string value (if we want the same property name for all source layers) or an object of the form {<source-layer-id>: <string>}.

[asheemmamoowala]

Or do you think we should change the semantics in geojson-vt instead, adding murmur hashing there?

I think they should be consistent - so adding the murmur hashing would allow generating the id from property of any type, which is highly requested. Having the same name for the option would also make it easy for developers to use it in all contexts.

I initially named it promoteId, but then thought that it would be misleading because the semantics are different. promoteId implies that you "promote" the same value (already a unique integer id) from feature.properties to feature.id, while here, the resulting id is derived, so it won't be the same when the property is anything but an unsigned integer.

I agree, promote doesn't feel right, given the hashing. Some options:

• Use a name that implies that the id will be different than the raw property value - derive/generate
• Make the transformation an internal detail. i.e keep promoteId or idProperty and transform the feature to use the correct id value whenever it is being returned through an API like queryRenderedFeatures. This implies that the source would have to be referred to whenever features from it are being processed.

accept a string value (if we want the same property name for all source layers) or an object of the form {<source-layer-id>: <string>}.

👍

[ryanbaumann]

@mourner this is excellent!

If we were to expose getIntegerId as a public function from the mapboxgl object, then this would allow the use of featureState for the data-join technique on vector tilesets that do not have a feature.Id value.

Here's an example of how a public method would allow developers to convert existing data-join maps (like our public example) to use feature state, while simplifying their code:

map.on('load', function () {

        // Add source for state polygons hosted on Mapbox, based on US Census Data:
        // https://www.census.gov/geo/maps-data/data/cbf/cbf_state.html
        map.addSource("states", {
            type: "vector",
            url: "mapbox://mapbox.us_census_states_2015",
            deriveIntegerId: 'STATE_ID'
        });

        // Set the color for the feature ID using featureState
        data.forEach(function (row) {
            map.setFeatureState({ 
                source: 'states', 
                sourceLayer: 'states', 
                id: mapboxgl.getIntegerId(row["STATE_ID"]) 
            }, 
            { 
                unemployment: row["unemployment"] 
            });
        };

        // Add layer from the vector tile source, styled based on the `unemployment` value 
        // in featureState with an expression
        map.addLayer({
            "id": "states-join",
            "type": "fill",
            "source": "states",
            "source-layer": "states",
            "paint": {
                "fill-color": ['interpolate', 
                    ['linear'],
                    ["feature-state", "unemployment"], 
                    0, '#e5f5f9',
                    5, '#b2e2e2',
                    10, '#66c2a4',
                    15, '#238b45'
                ]
            }
        }, 'waterway-label');
    });

Advantages:

• Performance - featureState allows higher performance using the data join technique than large match expressions.
• Style with expressions - instead of calculating explicit color values calculated outside of expressions to use with match
• Migrate to featureState - developers could use the getIntegerId public method to calculate a unique and stable feature state value on the fly given a unique property in any MVT. The developer would not need to check if the MVT contained a feature.id to use featureState or not.

Disadvantages:

• Another public method, complicates public API

[ansis]

Can we simply remove the type restriction for the id property

I have the same question as @peterqliu. Can we remove this restriction on the type and then have a promoteId that works similar to the one in geojson-vt?

The external API is simpler if the user can provide the value directly in the feature state api (map.setFeatureState({ source, sourceLayer, id: row["STATE_ID"] })) instead having a integer hash exposed to them.

[asheemmamoowala]

The external API is simpler if the user can provide the value directly in the feature state api (map.setFeatureState({ source, sourceLayer, id: row["STATE_ID"] })) instead having a integer hash exposed to them.

@ansis this is what I was hoping for with the suggestion in #8448 (comment):

BTW - should we combine this conversation with #6019 ?

[mourner]

Can we remove this restriction on the type and then have a promoteId that works similar to the one in geojson-vt?

Note that the vector tile spec only supports uint ids, so ideally we would fist need to adopt a new major spec version, otherwise this could get confusing. E.g. you can use promoteId in GeoJSON for string properties but you can't use them in GL JS because they're lost upon transfer to the main thread, which is a different issue.

Also, there can be major performance implications to supporting arbitrary IDs — e.g. currently FeauturePositionMap relies on ids in typed arrays for fast sorting and transferring.

So @asheemmamoowala's suggestion (converting to integers internally but exposing as original types) might be more feasible, although I would still prefer explicit conversion for a much simpler, less magical implementation.

[ansis]

@ansis this is what I was hoping for with the suggestion in #8448 (comment):

Yep, that sounds good!

Note that the vector tile spec only supports uint ids, so ideally we would fist need to adopt a new major spec version, otherwise this could get confusing. E.g. you can use promoteId in GeoJSON for string properties but you can't use them in GL JS because they're lost upon transfer to the main thread, which is a different issue.

We could work around this, right? by having a special property (.properties._mapboxID) that is used for transferring but gets moved to .id before being shown to a user maybe

[samgehret]

@mourner @chloekraw This is excellent! I have a couple points of feedback, mostly of things to keep in mind for product management.

1. Have you all seen the tippecanoe flags for --use-attribute-for-id=name and for --generate-ids. This seems like it could be related to this problem, however, I can definitely see the value from doing this client side in GL-JS before and removing the data processing step before hand.
2. Have we thought about implementing something like this as part of MTS?

[chloekraw]

@samgehret, ID generation will also be a part of MTS; we prototyped this as another interim solution that will hopefully unblock some customer use cases.

Are there specific challenges/drawbacks you have in mind with using this feature in GL-JS?

[samgehret]

@chloekraw I think it will be important for people to be able to translate both ways between the feature ID, and the attribute it was generated from. This will be important when doing client side data join. Often times developers will need to take an attribute from their geojson (say some FIPS ID or Post code ID), use that attribute to generate feature IDs (this is doable in this PR).

Their local data will still only have the FIPS ID, so how can they reference the feature ID in setFeatureState to write a metric into the feature state (to be used with a styling expression). Basically they will need know how to translate from FIPS ID -> Feature ID in order to accomplish this.

[asheemmamoowala]

@samgehret Thanks for the feedback! #8448 (comment) and #8448 (comment) are considering ways to make the id work with feature state by making it an internal detail.

[mannnick24]

@mourner I've tried this change out and it's going to work for us. We will be able to reduce our footprint and improve performance.
One question:
We will need to use {getIntegerId} from './util/integer_id' I had to add this to the exports so I could use it, as we need to get back to the feature ID from the property - we don't always have the feature when we want to do highlighting... sometimes its prompted from the data.
Nick

[mourner]

Alternative proposal in #8987