density-dependent editing zoom limit

[maning]

Currently, editing is allowed only at z16 or greater. In some areas where there is less data, I think z14 is OK (tested in Chrome).

[jfirebaugh]

Yes, density-dependent editing would be a great feature. z14 is fine for rural areas, but z16 is too low for dense urban areas.

It's fairly easy to calculate the density once all the tiles for a given area are downloaded. The tricky parts are:

• Managing the download process in a way that doesn't result in downloading a large number of dense data tiles, thus overloading the server with too many expensive requests and the client with too much data. iD currently downloads data tiles at a fixed zoom level (z16). Ideally, this would vary depending on density. But how do you know if an area is dense or not without downloading something first? There has to be some kind of feedback loop.
• Balancing first-load responsiveness (i.e. show the data as each tile comes in) with the desire not to start showing data at a particular zoom and then having it disappear when it turns out to be too dense. Maybe we just have to accept that that's going to happen sometimes?

[jfirebaugh]

I see this coming up pretty frequently as a usability issue. It's more pronounced at extreme latitudes, where the mercator projection stretches out data that tends to be very sparse already.

[tyrasd]

As an idea: @woodpeck made an analysis of densely mapped areas. For example, there is a list of all zoomlevel 10 tiles with more than 1000 nodes. I guess one could allow zooming out up to that level if there are less than a certain threshold number of nodes in that tile (limiting at 10k nodes, this would result in a blacklist of about 20k tiles). There is also a similar list for zoomlevel 13, but there's too much data to be stored on the client. Maybe it would make sense to make a query-able API out of this…

[bhousel]

Ugh looks like we have to actually solve this now. At a minimum we'll need to handle a response with "400 Bad Request" and the response error message like "You requested too many nodes (limit is 50000)" and then increase the tile zoom.

iD does not work when loading this very densely mapped place (lots of indoor mapping happening here):
https://www.openstreetmap.org/edit#map=19/41.89317/12.49330

[woodpeck]

The node density script I made at the time has been superseded by @joto's https://github.com/osmcode/osmium-contrib/tree/master/node_density program. My plan at the time was to regularly generate an analysis and modify the download limit on the OSM server to "preflight" the request against the node density "database" (essentially an 8bpp PNG file with 1 pixel per z14 tile or so), dropping the 0.25°² limit and replacing it with a true estimate of how many nodes there will be.

However that was intended to help editing in sparsely mapped areas, allowing you to download larger areas at once; what we have here is a problem with densely mapped areas. The pre-flighting would at best enable a fast-fail on the server but not help the client much.

Perhaps the node density lookup could be made accessible to the editor directly so the editor would know what's possible, and force the user to zoom in when in a dense area...

[pnorman]

This issue seems to be about allowing users to edit at a zoom that better suited what they were doing, not changing how data is downloaded.

[bhousel]

This issue seems to be about allowing users to edit at a zoom that better suited what they were doing, not changing how data is downloaded.

Yes, this is different but related. This problem of requesting data from the API needs to be solved first anyway (see @jfirebaugh's comment above about "iD currently downloads data tiles at a fixed zoom level (z16). Ideally, this would vary depending on density. But how do you know if an area is dense or not without downloading something first?")

Maybe instead of hardcoding a tilezoom, we could determine a reasonable starting tilezoom based on latitude, then adjust if the server responds with "400 Bad Request"? Thankfully we get back that result really quickly and could easily break a dense tile up into z+1 quads and re-request those instead.

We need to store tile statistics somewhere too and make it available to other parts of the iD code - this relates to #2248 (comment) too, which has been on my mind.

[bhousel]

CC @kriscarle for visibility - relates to maphubs/moabi work.

[kriscarle]

Would the data from http://osm-analytics.org/ help with this, or making our own density tiles based on the QA tiles? I guess if a HOT activation significantly increased the density of an area overnight it might cause a problem? Maybe a safety check that disables it somehow in that scenario, if the node counts downloaded are significantly higher than what is in the density tiles?

Another option could be to combine it with a config in the url API, off by default, so only projects that are doing rural mapping enable it, possibly by adding a toggle to tasking manager? HOT building mapping tasks wouldn't need this anyway? For those of us mapping roads/rivers in remote parts of the rainforest, the density is not going to change that rapidly, just don't want to melt the servers if the user happens upon a denser area.

[pnorman]

Would the data from http://osm-analytics.org/ help with this, or making our own density tiles based on the QA tiles?

Making density tiles isn't really a problem, there's half a dozen ways to do it. https://github.com/osmcode/osmium-contrib/tree/master/node_density is probably the best. It's what to do with that data that's in question.

[kriscarle]

The discussion above seems to around whether to have iD

1. react to dense data after it encounters it and adapt, or
2. making a density dataset available to iD so it can determine when it is safe to go above z16

Option 1 sounds like a complicated build? Plus it still means downloading the data on the initial load before figuring out how to optimize later requests, and that could still put a strain on both server and client?

For option 2 it was mentioned a couple times that "maybe we should make an API" followed by months/years of silence, because that is a hard thing to do :) OSM Analytics is basically that API now? The buildings vector tiles are a density aggregate down to z12 (before switching to actual buildings). What I'm proposing is that iD download the z12 tile from OSM Analytics, count the density pixels in the tile, if they are greater than an agreed upon threshold, then iD sticks with the z16 limit, otherwise we can allow z14, or possibly even a sliding scale up to z12.

@tyrasd it would be awesome to get your thoughts on this, just realized you were on this thread back in 2013 :)

The next issue to me seems to be dealing with the inevitable lag between any density dataset and OSM. That could be an issue when an area changes rapidly and there are hundreds of people loading that area in iD, like during HOT activations mapping buildings in urban areas that were previously unmapped. To handle that, we can allow this feature to be toggled off (or until we are happy with it, off by default and toggled on), and HOT can decide if they want it on by default in the links generated from tasking manager.

[geohacker]

Glad to see the activity on this ticket! I feel like if we can store quadkey densities as part of the OSM API that's probably the best and let other editors also use this information. OSM Analytics could give this as a byproduct, but is dependent on QA tiles but I somehow think this should in API core.

Do we know what densities translate to what zoom level? It sounds like to me that even z16 is soon going to be difficult.