OSM data download problems

[eehpcm]

Since the 2.15.0 release, download of OSM data is not happening as it should.

1. It takes 15 minutes or more for the data to show up. All the controls are there and do what they should, background imagery shows; OSM data takes a long time before it suddenly appears, all at once.

2 Search for a location is impacted. If I search for a named location (in this case, Llanboidy) I get two matches. Clicking on either does nothing. I suspect it's another case of data not downloading. I tried that while I was waiting for the data to appear, in case it kicked something into action.

3. After the data finally appeared, I added a couple of buildings. I tried to square them (after remembering Q not S) to be told they couldn't be squared because the data for them hadn't been downloaded. Why would iD need to download data for something I had just added but not yet saved?

4. I saved those buildings. I know iD refetches the data after a save. I've spent 10 minutes writing this, data has not yet appeared.

Firefox 60.6.1 ESR.

Even stranger, I have a laptop with the same version of Firefox and the same plugins (Adblock Plus, uBlock Origin and Privacy Possum all disabled for this site on both browsers). It's fine. I'd use it, but the laptop is too fiddly for mapping.

[quincylvania]

@eehpcm That bug sounds bad!

I just tested iD 2.15.0 on Firefox 67.0 and Firefox 60.7.0 ESR on macOS and couldn't reproduce the problem. Everything appeared to work normally. Is there anything logged to the JavaScript console? Would anything else about your environment affect your connection?

[eehpcm]

@quincylvania

I just tested iD 2.15.0 on Firefox 67.0 and Firefox 60.7.0 ESR on macOS and couldn't reproduce the problem

This is on Linux, but that shouldn't make a difference. Theoretically.

Is there anything logged to the JavaScript console?

Many things. Either identifiably from other windows or unidentifiable (to me). Looking at the Net XHR, if I search for a location, what shows is "GET XHR https://www.openstreetmap.org/api/0.6 /node/255140813 [HTTP/2.0 200 OK 230ms]. What should I be looking for? There's a lot of errors and warnings, most of the ones I can identify from gmail.

But I have a lot of windows open on different things. And back at 2.14.0 I found that being logged in to Facebook caused this sort of data load problem. So I don't do that any more on my desktop. Facebook is why I have the laptop running. In any case, things aren't supposed to leak like that. Then again, I had to disable the funky new sandboxing in Mozilla as soon as it appeared - it may make things faster if you have a lot of memory but on my old box it's a killer. Literally a killer - after getting slower and slower as time went by, Firefox would then crash.

Would anything else about your environment affect your connection?

The connection, no. The desktop (where data load takes 15 minutes to show up) is acting as a router for the laptop (where data load is almost instantaneous). Not a connection problem.

[quincylvania]

@eehpcm Hmm sounds like there are a lot of factors that could be affecting iD. I don't have a Linux setup for testing. Let us know if you can isolate iD's console messages from those of other tabs.

We did change a lot of our dependencies for 2.15.0 which may have affected things. See #6245 and #6087.

[eehpcm]

@quincylvania

Hmm sounds like there are a lot of factors that could be affecting iD. I don't have a Linux setup for testing. Let us know if you can isolate iD's console messages from those of other tabs.

I just tried closing the tabs that are heavy on the XHR stuff (gmail, yahoo mail, youtube). Closed the tab with iD and re-opened it. Didn't help, but I did get some console stuff that is probably mostly yours (and probably not helpful), see at the end of this.

So then I fired up gmail again, whilst iD was still waiting for data load. The result had my system thrashing for memory until Firefox crashed. Something isn't playing nicely with the other children.

This sorta reminds me of a problem in the past with data loading. That turned out to be iD trying to grab the data for the whole world instead of the current area in view.

Something else I noticed earlier. While the data download is pending, the spinner doesn't show. But when I search for a location it does.

Aaaaaaaaaaargh! While writing this post I fired up iD again and the data loaded almost instantly. I squared a couple of buildings, saved, and the data loaded after a few seconds instead of 15 minutes. I have almost everything open I had before, save for one tab I decided I no longer needed (but didn't have anything clever going on that I could see).

Console dump...

XHR
https://www.openstreetmap.org/assets/iD/img/tnp-sprite-6f7c9c698d4767cf54058c16ecdec9699db5d9e1b2498766082b9b51e57109e6.svg
[HTTP/2.0 200 OK 73ms]
GET
XHR
https://www.openstreetmap.org/assets/iD/img/community-sprite-3ad3b38e3ad9ae8535148175795655eb79c2d33e87d266b93cd7eddd7ec23c19.svg
[HTTP/2.0 200 OK 110ms]
GET
XHR
https://www.openstreetmap.org/api/capabilities
[HTTP/2.0 200 OK 197ms]
GET
XHR
https://www.openstreetmap.org/assets/iD/img/mapillary-sprite-852c205929e3a3c1be0e5fe6e1e6e2206ea3a475169aeab04b567772674ebc22.svg
[HTTP/2.0 200 OK 0ms]
GET
XHR
https://www.openstreetmap.org/api/capabilities
[HTTP/2.0 200 OK 258ms]

And then it just hung around for 15 minutes until the data showed up.

[bhousel]

This sorta reminds me of a problem in the past with data loading. That turned out to be iD trying to grab the data for the whole world instead of the current area in view.

This is what I am thinking too - there are a few things in iD now that will initiate download of offscreen tiles, and I can imagine that you might have clicked on a road or river that is pulling down a lot more data than we were anticipating. Can you send me the lat/lng around where this happens?

[eehpcm]

@bhousel

This is what I am thinking too - there are a few things in iD now that will initiate download of offscreen tiles, and I can imagine that you might have clicked on a road or river that is pulling down a lot more data than we were anticipating.

Didn't click on anything. Just fired up the browser and iD started at the same place it had been before (which was fine the day before). So not that.

Today it's not a problem. At all. In fact, it's faster than it has been in recent months. I'm starting to wonder if there's some leakage between supposed sandboxes and yesterday Gmail or YouTube or Yahoo Mail or somebody implemented a change that caused the problems and later they backed it out or implemented it differently.

Hmmm. Is there more than one server that can supply the data and, if so, are they on a DNS round-robin? If one of them went on a major go-slow a DNS round-robin could explain why one of my computers was having problems and the other wasn't (both run their own local DNS resolver because my ISP's resolvers are crap).

I'll close this because even I can't reproduce it any more. But if the problem recurs...

[bhousel]

Thanks @eehpcm - I did try to get iD to fetch a lot of extra tiles in this area by right clicking on some roads and streams, and while some of the roads do stretch out a bit, the data is very sparse and loads quickly.

For example, a road that extends out 10 tiles will fetch these tiles during browser idle times. Just to be on the safe side, I think I will still introduce a limit so that the queue can not grow unchecked.

(For comparison 20 tiles is about what it takes to cover the screen at z16)