Include appropriate user-agent header when making map requests

[pnorman]

I'm one of the admins of tile.openstreetmap.org. Currently flutter_map uses the default dart:io HTTP User-Agent (e.g. Dart/2.17 (dart:io)). This means that when looking at traffic, flutter traffic gets lumped in with other Dart traffic, and if any Dart users are causing usage problems, all Dart users including flutter could end up blocked. This happened recently, and because it was a generic user-agent, I was unable to figure out who the traffic was coming from.

The tile usage policy requires a Valid HTTP User-Agent identifying application. For flutter, this could be something like flutter_map/1.2.3, where 1.2.3 is a version.

A better option for your users would be to automatically generate a user-agent from the name of their app, if possible. Frequently the name of the app is something like com.example.app, so a user-agent could be com.example.app/456 flutter_map/1.2.3. If an app-specific default is not set, then the instructions should be clear about the need to set an appropriate user-agent

[ibrierley]

Quick thoughts, as not about much the next few days....Here's a possible starting point for a workaround I just pasted...

NetworkImage allows headers, but NetworkImageWIthRetry doesn't afaics (there is an issue raised there though flutter/flutter#19532 so sounds like maybe that's coming, altho maybe what I did later actually works...)

If we need a quick hack (without hacking flutter_map itself...this _may_ work...
import 'package:flutter_map/src/layer/tile_layer/coords.dart';

class NetworkTileProvider extends TileProvider {
  Map<String,String> headers = const {};
  NetworkTileProvider({this.headers = const {}});

  @override
  ImageProvider getImage(Coords<num> coords, TileLayerOptions options) {
    return NetworkImage(getTileUrl(coords, options), headers: headers);
  }
}


in FlutterMap widget

TileLayerOptions(
  tileProvider: NetworkTileProvider(headers: { 'User-Agent' : 'flutter_map/1.1.1)' }),
  urlTemplate: 'https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png',
    subdomains: ['a', 'b', 'c'],
  ),
)

For FlutterMap itself, could we do something like (untested)...

class NetworkTileProvider extends TileProvider {
  @override
  ImageProvider getImage(Coords<num> coords, TileLayerOptions options) {
    return NetworkImageWithRetry(getTileUrl(coords, options), headers: const { 'User-Agent' : 'flutter_map/1.1.1)' });
  }
}

class NonCachingNetworkTileProvider extends TileProvider {
  const NonCachingNetworkTileProvider();

  @override
  ImageProvider getImage(Coords<num> coords, TileLayerOptions options) {
    return NetworkImage(getTileUrl(coords, options), headers: const { 'User-Agent' : 'flutter_map/1.1.1)' });
  }
}

in tile_provider/network_image_with_retry.dart

Not quite sure if this affects browser user_agents...or if I was doing something wrong, so above may be a load of rubbish!

[ibrierley]

(we should probably let users set any header they want actually if we can, at the same time)

[ibrierley]

Out of interest, are there any notification emails or anything for OpenStreetMaps, or work more closely with them somehow. Naturally it feels like it's in everyones interests to be more aware of changes, potential issues etc

[ibrierley]

Note, browser says "Refused to set unsafe header "User-Agent" when I try some previous suggestions, so for security I'm wondering if that's not possible in web mode...

[namanshergill]

Note, browser says "Refused to set unsafe header "User-Agent" when I try some previous suggestions, so for security I'm wondering if that's not possible in web mode...

Is this on a debug build? Because sometimes developer modes of browsers fail CORS checks by default for localhost, so it’s not necessary the issue would replicate in release builds if that’s the case

[JaffaKetchup]

@pnorman Would it be possible to have an agent like:
"flutter_map/1.2.3 (com.example.app)"?

If so, I like the version given above. We can provide a default in case they don't change it: we can use package_info_plus to get the application/package name.

[JaffaKetchup]

If so, I'll make a PR ASAP to fix it.

[pnorman]

Yes that would work

[gdy77]

Hi

If I extend TileProvider without overriding Map<String, String> headers,
(tileProvider..headers.putIfAbsent('User-Agent', () => 'flutter_map ($userAgentPackageName)')), send the error Unsupported operation: Cannot modify unmodifiable map

Is it a normal behavior ?

[andreandersson]

Long time since working with OpenStreetMap; but isn't the main policy that you should generate the tiles yourself or use a cache-server of your own (for example a Varnish cache)? That is, the main policy is that the end users (i.e. 500k app users) should never go directly to OpenStreetMap.org due to load restrictions?

It should also be noted that we might not want to set that header at all, depending on which tiles the users (i.e. developers using the library) uses.

[JaffaKetchup]

@paul-gdy That issue should be resolved. Thanks for the heads up!

@andreandersson
https://operations.osmfoundation.org/policies/tiles sets out the OSM tile server rules. It does say that:

Heavy use (e.g. distributing a heavy-usage app that uses tiles from openstreetmap.org) is forbidden without prior permission from the Operations Working Group. See below for alternatives.

However, this doesn't really concern us here: we provide a service for all tile servers. Originally, this issue was started by the OSM admin, but we've implemented this feature because it applies to most servers.

In answer to your second question, if the ToS says to include a User-Agent header, you must - but if it doesn't, it's recommended too anyway (almost out of courtesy). If developers are testing with this library, that's still traffic to a tile server that should be counted. In any case, there is no requirement in the API to identify the application/package name, only the default implementation to identify this library with an unknown application.
If you really need to disable the header, you can send custom headers with a blank or other 'User-Agent', and this library will not override it.