Rewrites the osrm-components tool to dump GeoJSON, resolves #2176 #1738 #1602

[daniel-j-h]

Resolves #2176 #1738 #1602.
Unblocks node-osrm from bundling osrm-components Project-OSRM/node-osrm#283.

On Monaco, unstyled raw dump: http://bl.ocks.org/d/9d6265b064c0f29a71503e0f6527769d

Would love some input here: at the moment I expose OSM ids as properties for the LineString's from and to node. Is this helpful and are these use-cases where osm node ids help here? Otherwise I would remove it, since it bloats the file quite a bit.

cc @MoKob @geohacker @planemad @srividyacb @oini

Tasklist

• review
• adjust for comments
• decide on node osm id properties
• check on planet how large the new geojson file will get (~2 GB, ~100 MB compressed)
• adapt demo server integration
• notify Fred from Geofabrik and figure out new integration with them

[daniel-j-h]

The previous behavior (and what I replicated so far) is to highlight all ways inside small components.

What I want to do in addition is to highlight segments where

component(source) != component(target)

and one of the components being a small one. This can be used to more easily find the edges going into or out of the small components (if such edges exist at all).

[daniel-j-h]

Now adds "type": "inner", "type": "border" to let users style edges completely inside the small component vs edges crossing the small component border. Should be helpful for gated areas etc.

http://bl.ocks.org/d/63117690b40e48e0105b437dc72aaed5

[srividyacb]

@daniel-j-h Looking at the Monaco Dump of small-components, the id stored is node-id present in the LineString. What are from and to node-ids?

Is there a way we can keep the way-id ? This will be easy if we wish to open these features from a map to any of the editors.

[daniel-j-h]

We don't store way ids in any form, all we have at that point are osm node ids.

At the moment, the LineStrings have properties:

"from": 1,
"to": 2,
"type": "border"

• from is the way segment's starting osm node id
• to is the way segments's ending osm node id
• type is either border for a way segment crossing components (if available) or inner for a way segment starting and ending inside the same component

We can talk about the properties here or how to represent the way segments in GeoJSON. I see no way to add osm way ids though.

[TheMarex]

Instead of running another components search, we could also just read the .osrm.file_index file from the StaticRTree and use the small component information from it. That would side-step the need to hard-code the thresholds.

That said the general approach here is fine and I see no problem merging this.

[daniel-j-h]

Here's an example, styled based on the inner vs border type. Not really that helpful for monaco, but could help for gated areas, airports and so on.

• inner (inside the same component): orange
• border (crossing two components): teal

[daniel-j-h]

@geohacker @oini @srividyacb - could you check if these modifications to the osrm-component tool suit your needs. If that's fine I'd like to merge it and bundle it with node-osrm for the next release. I also just ran the tool on the planet and it produces a ~2 GB GeoJSON file, compresses down to ~100 MB.

cc @woodpeck for context on ticket

[woodpeck]

For OSMI integration, if you can create and transmit the GeoJSON, I'd probably simply convert to shape on my end and dump into OSMI as before. If creating and uploading the file is an extra burden for you then I could probably also run that somewhere on OSMI.

[daniel-j-h]

@woodpeck not sure if you are aware of it but we are trying out the following for a more realistic small components layer: use a custom profile and enable routing over tags such as access=destination and others which are routing engine limitations but not data issues.

The small components layer then shows data issues not influenced by the absence of features in the routing engine. Here is this experimental connectivity profile: #3551

The demo server will still continue to run the regular car profile since we don't want the connectivity profile for routing. I'm not sure yet how we should handle this split: we would need to keep the connectivity profile in sync and run it regularly for small components. If you think you can run this on your servers that would be great (osrm-extract + osrm-components is all you need). I just fear this might accidentally go out of sync. What's your take here?

[woodpeck]

Hm, I'm not too deep into OSRM development but are there many things that actually need to be kept in sync? I'd guess that tuning on the standard profiles will mostly be changing a speed here, changing the handling of gates there, and so on; if the connectivity profile just looks at the bare road layout then I can't see many things that actually would need syncing?

[daniel-j-h]

With syncing I mostly mean you running the latest OSRM release and us keeping the profile changes here #3551 in sync with master.

The profile changes shouldn't be too hard to keep in sync with master. I don't know how often you want and can update your OSRM instance. We provide Docker images and pre-built binaries via npm, so installation does not require compiling from source.

What I really want to avoid is having 1/ a stale OSRM installation creating this layer and 2/ a stale components layer which we then show to the OpenStreetMap community.

[woodpeck]

Updating the software frequently should be the easier bit; as for updating the data, I'd aim at daily but we'd have to see how it turns out, performance-wise...

[TheMarex]

@daniel-j-h from my side this is ready to merge, anything else left?

[daniel-j-h]

api-osrm-routed needs to be adapted, otherwise we break the next cycling phase - will take a look now.

[daniel-j-h]

Demo server integration adapted, updated and cycled anew.

[daniel-j-h]

@woodpeck the demo server will now no longer upload the car small components layer - please set up your components pipeline to create the layer from the connectivity profile (see #3551) which does not show routing engine issues.