Do not use OpenStreetMap export endpoint

[pnorman]

The OpenStreetMap tile usage policy states

Calls to /cgi-bin/export may only be triggered by direct end-user action. (For example: "click here to export".) The export call is an expensive (CPU+RAM) function to run and will frequently reject when server is under high load.

Stitching images together like BigMap 2 or StaticMap do is a better option to avoid heavy load, but any usage of tile.openstreetmap.org still needs to follow the usage policy. I couldn't find a User-agent in the source, but all the technical usage requirements need to be met.

cc @tomhughes

[dkahle]

Thanks, Paul. Can you make updates and submit a PR making appropriate changes, please?

[pnorman]

Thanks, Paul. Can you make updates and submit a PR making appropriate changes, please?

No, I don't know R.

[SK53]

@dkahle it is your responsibility to make releases which abide by OpenStreetMap's T&Cs. Asking someone who devotes a considerable amount of time to OSM (@pnorman is a board member of the Foundation, has been lead of the main website Carto-CSS, sits on the Data Working Group, and develops osm2pgsql inter alia) to do this instead just ain't fair.

I came to this issue as we have had a query on the OSM Help site https://help.openstreetmap.org/questions/53975/get-osm-map-in-r about the issue. Static map or tile stitching queries cause a considerable load on donation supported and volunteer maintained OSM servers, and while not completely banned are strongly discouraged.

I do realise that it is considerably harder to implement something similar with OSM data as there is no obvious 1:1 substitute for a simple static map call. MapBox do offer something similar, but as was the case with Cloudmade and MapQuest you need an API key. So in the short term that is probably the simplest thing to recommend.

[dkahle]

@SK53 thanks for the comments. I'm not opposed to resolving the issue, but I do have a few thoughts:

1. It's not clear to me how best to resolve the issue. Thus the comment to @pnorman.
2. It's not clear to me that ggmap's use does violate the T&Cs. The function in ggmap that queries OSMs servers is an end-user action: it hits the server once per evaluation, just like a "click here to export" button. The onus is thus on the user to follow the T&Cs, and in that sense is no different from the user simply using other R functions to query the server. (In other words: ggmap simply helps R users format a URL and then uses standard non-ggmap functions to make the query, which would be possible whether or not ggmap is doing it.) I'm happy putting a reference to a page with the T&Cs; if you think that'd be appropriate, which link do you think would be most appropriate?
3. ggmap has been around for years with this functionality, and this is the first time that I have heard of this. I (strongly) suspect OSM to be rarely used by R users – it's not defaulted in any functionality, and other outlets are faster, more reliable, and easier to use.

As you can tell from the commit log, I work on this project very rarely. As possible, it would be very helpful to have a clear and convincing recommendation as to what to do.

[tomhughes]

I think that section of the usage policy is actually weaker than I at least intended.

That endpoint was added purely to support the "export" function on www.openstreetmap.org and was never intended to be used by anybody else. In fact as of last week there are now technical measures in place to try and ensure that, which is likely the reason for that help.openstreetmap.org question.

I can't recall the details now but I believe this ticket was originally opened because we had encountered a problem that appeared to be down to a user of this function.

In any case unless the function provided here allows the user agent and/or referer to be specified then there is no way for anybody using it to meet the other requirements of the usage policy - the purpose of that requirement is to allow us to know who to contact if overuse is causing a problem and if we can only trace the request back to your library then this is where we're going to wind up coming rather than to the actual end user.

[SK53]

@dkahle I agree that the number of R users making use of this must be very small otherwise we would have seen issues before. It's quite likely that the export call works for very small areas, but even for an end user the chance of success for most requests has been minimal since 2010ish. This is why other map sources are more suitable for this usage, OpenStreetMap tiles are not provided primarily for application usage.

The idea that this is helping users format a URL is I think nitpicking, it can be used programmatically. The relevant T&Cs are (note line on the cgi/export call).

As @pnorman noted no user agent is passed with the call, and this is mandatory.

[pnorman]

I can't recall the details now but I believe this ticket was originally opened because we had encountered a problem that appeared to be down to a user of this function.

Yes. I'm not sure if it was solved by blocking the IP, blocking this library, blocking any downloads with the default R User-Agent, or it went away on its own. If it happens again, any of those blocks could happen.

It's not clear to me how best to resolve the issue. Thus the comment to @pnorman.

I would recommend

1. Specify a default User-Agent and letting the user of the library override it
2. Remove the call to the export API. Aside from the ToS issues, it's not likely to work a lot of the time
3. Replace the call to the export API with one that stitches tiles together.

I can understand that 3 might be too much work to happen.

[ConorIA]

@pnorman, I tried to use the function today, but recieved a 400 error, so unless the University of Toronto ip range is blocked, I imagine the package itself is blocked.

> library(ggmap)
Loading required package: ggplot2
Stackoverflow is a great place to get help:
http://stackoverflow.com/tags/ggplot2.
Google Maps API Terms of Service: http://developers.google.com/maps/terms.
Please cite ggmap if you use it: see citation("ggmap") for details.
> chennai_osm_map <- qmap("chennai", zoom=12, source = "osm")
Source : https://maps.googleapis.com/maps/api/staticmap?center=chennai&zoom=12&size=640x640&scale=2&maptype=terrain
Source : https://maps.googleapis.com/maps/api/geocode/json?address=chennai
Error: map grabbing failed - see details in ?get_openstreetmap.
In addition: Warning message:
In download.file(url, destfile = destfile, quiet = !messaging, mode = "wb") :
  cannot open URL 'http://tile.openstreetmap.org/cgi-bin/export?bbox=80.161026780127,12.9754780394782,80.380753342627,13.189501502698&scale=110000&format=png': HTTP status was '400 Bad Request'
> toronto_osm_map <- qmap("Toronto", zoom=12, source = "osm")
Source : https://maps.googleapis.com/maps/api/staticmap?center=Toronto&zoom=12&size=640x640&scale=2&maptype=terrain
Source : https://maps.googleapis.com/maps/api/geocode/json?address=Toronto
Error: map grabbing failed - see details in ?get_openstreetmap.
In addition: Warning message:
In download.file(url, destfile = destfile, quiet = !messaging, mode = "wb") :
  cannot open URL 'http://tile.openstreetmap.org/cgi-bin/export?bbox=-79.492875919873,43.5735595483433,-79.273149357373,43.7325388331343&scale=110000&format=png': HTTP status was '400 Bad Request'
>

[tomhughes]

No, as I said, the /cgi-bin/export endpoint was never intended for use other than as part of www.openstreetmap.org and that is now being enforced by means of a shared secret so that calls made to that URL which have not come www.openstreetmap.org will be rejected.

That does not apply to tile downloads however, though we still expect the usage policy to be complied with for those.

[topialla]

Are there any viable work arounds?

[phewson]

Would including a dockerised OSM tile server fix this?

[dbuijs]

Would it be possible to allow an alternate IP source to be specified as a parameter in the function call? I could see setting up a private tile server to cache the general areas I want my internal users to be able to map, and this would limit the load on the OSM public servers.

[pnorman]

The export call is restricted to use from osm.org only now, so this issue is now moot.

Tile usage is governed by the tile usage policy