Setting Up OpenStreetMap Tile Server with Official Styles on AWS EC2 with PostgreSQL RDS and PostGIS
These are some notes on setting up OpenStreetMap tile server with official OSM styles on an Amazon EC2 instance with a PostgreSQL RDS instance supporting PostGIS.
These notes will cover some technical details of the AWS EC2 + RDS stack, that are omitted or not present in the guides at https://switch2osm.org
NOTE: This is written after the fact, I may have forgotten or omitted something by accident.
- Amazon Linux AMI https://aws.amazon.com/amazon-linux-ami/2015.03-release-notes/
- m3.medium instance with 60G General Purpose SSD + 1 Elastic IP
- PostgreSQL RDS m3.medium instance with (default) 100G storage + PostGIS 2.1.5
- http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html
- http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html
Before configuring the PostgreSQL instance, prepare your AMI Linux instance first and refer to Configure PostgresSQL RDS Instance further below.
To continue with this step your EC2 instance should be up and running and you should be able to access it over SSH.
First, enable the EPEL repository for extra packages:
$ sudo yum-config-manager --enable epel
To make life easier on the command line install byobu and to be able to clone required OpenStreetMap software later, also install git
$ sudo yum install byobu git
Additional packages are required for building and configuring the OpenStreetMap software stack. Some of the packages listed below will trigger additional dependencies that are also to be installed.
$ sudo yum install \
apr apr-devel \
apr-util apr-util-devel \
autoconf automake \
boost boost-devel \
bzip2 bzip2-devel \
cairo cairo-devel \
cjkuni-fonts-common \
freetype-devel \
gcc gcc-c++ \
gdal gdal-devel \
geos geos-devel \
httpd24 httpd24-devel \
libcurl libucrl-devel \
libicu libicu-devel \
libjpeg-turbo libjpeg-turbo-devel \
libpng libpng-devel \
libtiff libtiff-devel \
libtool \
libuv libuv-devel \
libxml libxml-devel \
lua lua-devel \
mod24_ssl \
nodejs npm \
ogdi \
postgresql93 postgresql93-contrib postgresql93-devel postgresql93-libs postgresql93-server \
proj proj-devel \
protobuf protobuf-c protobuf-c-devel protobuf-compiler protobuf-devel \
python27 python27-PyYAML python27-pycairo python27-pycairo-devel \
sqlite sqlite-devel \
zlib zlib-devel
Phew, one last package, install CartoCSS which we will require later to create map styles for Mapnik.
$ sudo npm install -g carto
In the home directory of your AMI Linux user, the ec2-user, create a src directory
$ mkdir src
The ~/src directory will be used going forward to get (git clone) additional software required:
- OpenStreetMap Carto
- OpenStreetMap data into PostgreSQL converter (osm2pgsql)
- Mapnik
- mod_tile
The PostgreSQL instance should be set up with the user postgres with a password of your choosing and a default database named gis.
After logging in into your AMI Linux instance connect to PostgreSQL
$ psql -U postgres -H <your-server-zone>.rds.amazon.com gis
In the psql terminal follow the PostGIS set up instructions accordingly to your database instance username.
gis=> create extension postgis;
Should be the bare minimum (<- Office Space reference right here!)
In addition you could do
gis=> create extension fuzzystrmatch;
gis=> create extension postgis_tiger_geocoder;
gis=> create extension postgis_topology;
Give access and change ownership:
gis=> grant connect on database gis to postgres;
gis=> alter schema public owner to postgres;
and in addition to the bare minimum if you were feeling like it:
gis=> alter schema tiger owner to postgres;
gis=> alter schema tiger_data owner to postgres;
gis=> alter schema topology owner to postgres;
$ cd ~/src
$ git clone git://github.com/mapnik/mapnik
$ cd mapnik
$ git checkout -b v2.2.0 v2.2.0
$ python scons/scons.py configure INPUT_PLUGINS=all OPTIMIZATION=3 SYSTEM_FONTS=/usr/share/fonts/dejavu/
$ make
$ sudo make install
NOTE: the following optional dependencies could not be satisfied with default AMI Linux packages.
Note: will build without these OPTIONAL dependencies:
- ociei (Oracle database library | configure with OCCI_LIBS & OCCI_INCLUDES | more info: https://github.com/mapnik/mapnik/wiki/OCCI)
- rasterlite
The mapnik libraries are installed into /usr/local/lib, which for some reason is ignored by default by AMI Linux. A configuration file needs to be set up for ldconfig to fix this.
$ pushd .
$ cd /etc/ld.so.conf.d/
$ sudo echo -n "/usr/local/lib" > mapnik.conf
$ sudo ldconfig
$ popd
$ python
Python 2.7.9 (default, Apr 1 2015, 18:18:03)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-16)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import mapnik
>>> quit()
Between the import mapnik and quit() commands there should be no output, indicating that the libmapnik.so.2.2.0 was properly found from /usr/local/lib
These are the CartoCSS map style sheets for the Standard map layer on https://www.openstreetmap.org/
$ cd ~/src
$ git clone https://github.com/gravitystorm/openstreetmap-carto
As of writing this the latest stable in the v2.x is v2.29.1
$ cd openstreetmap-carto
$ git checkout -b v2.29.1 v2.29.1
With a text editor modify the osm2pgsql: part in _parts section in the project.yaml file by adding host, user and password key value pairs
31 osm2pgsql: &osm2pgsql
32 type: "postgis"
33 dbname: "gis"
34 host: "<your-server-zone>.rds.amazonaws.com"
35 user: "postgres"
36 password: "<your-postgres-password>"
37 key_field: ""
38 geometry_field: "way"
39 extent: "-20037508,-20037508,20037508,20037508"
Now as described in Editing Layers at https://github.com/gravitystorm/openstreetmap-carto/blob/master/CONTRIBUTING.md populate the new database configuration to (a new) project.mml
$ ./scripts/yaml2mml.py < project.yaml > project-1.mml
$ ./get-shapefiles.sh
After which the data/ sub-directory is created into the openstreetmap-carto directory.
$ carto project-1.mml > OSMDefault.xml
That is it for now, we will follow up on this a bit later, after having set up the data set via osm2pgsql and built Mapnik and mod_tile.
$ cd ~/src
$ git clone git://github.com/openstreetmap/osm2pgsql.git osm2pgsql
$ cd osm2pgsql
$ ./autogen.sh
$ ./configure
$ make
$ sudo make install
As of writing this the osm2psql version is 0.87.4-dev.
Importing the whole planet data will take forever on this m3.medium instance used here, if that is your requirement you should upgrade your instance.
One of the sites offering OSM data sets is at http://download.geofabrik.de/
For demonstatration purposes we will use a single country from http://download.geofabrik.de/europe.html You can pick yours :) Depending on the file size this can take a while.
$ cd ~/src
$ mkdir osm-data; cd osm-data
$ wget http://download.geofabrik.de/europe/estonia-latest.osm.pbf
As the m3.medium is a single CPU instance with only 3.7GB of RAM the import will utilize the slim mode of osm2pgsql. For additional osm2pgsql help and options do:
$ osm2pgsql -h -v | less
To import the data set in slim mode with a cache of 2500MB, forcing a Postgres password prompt and utlizing the style from OpenStreetMap Carto:
$ osm2pgsql -c -s -C 2500 -G -K -W -d gis -U postgres -H <your-server-zone>.rds.amazonaws.com estonia-latest.osm.pbf -S ~/src/openstreetmap-carto/openstreetmap-carto.style
After a successful import the the table listing in psql should look something like this:
gis=> \dt
List of relations
Schema | Name | Type | Owner
----------+--------------------+-------+----------
public | planet_osm_line | table | postgres
public | planet_osm_nodes | table | postgres
public | planet_osm_point | table | postgres
public | planet_osm_polygon | table | postgres
public | planet_osm_rels | table | postgres
public | planet_osm_roads | table | postgres
public | planet_osm_ways | table | postgres
public | spatial_ref_sys | table | rdsadmin
topology | layer | table | rdsadmin
topology | topology | table | rdsadmin
(10 rows)
A program to efficiently render and serve map tiles for https://www.openstreetmap.org map using Apache and Mapnik.
The mod_tile will provide the following utilities that will be installed into /usr/local/bin
- renderd - Mapnik rendering daemon
- render_list - for pre-rendering the map tiles
$ cd ~/src
$ git clone git://github.com/openstreetmap/mod_tile.git
$ cd mod_tile
$ ./autogen.sh
$ ./configure
$ make
$ sudo make install
$ sudo make install-mod_tile
$ sudo ldconfig
This will set up the OpenStreetMap Carto files to be used for Mapnik and renderd.
$ cd ~/src/openstreetmap-carto
$ sudo cp OSMDefault.xml /usr/local/share/maps/style/
$ sudo cp -R data/ /usr/local/share/maps/style/
$ sudo cp -R symbols/ /usr/local/share/maps/style/
Locate the configuration file at /usr/local/etc/renderd.conf and fire up your text editor (sudo is required).
Locate the listed key/value pairs and un-comment them by removing the leading ; (semi-colon) and altering the values accordingly:
- socketname=/var/run/renderd/renderd.sock
- plugins_dir=/usr/local/lib/mapnik/input
- font_dir=/usr/share/fonts/dejavu
- XML=/usr/local/share/maps/style/OSMDefault.xml
- HOST=<your-ec2-dashed-ip-zone>.compute.amazonaws.com
So that the end result (with your editor's line numbers) might look something like this:
1 [renderd]
2 socketname=/var/run/renderd/renderd.sock
.
.
.
21 [mapnik]
22 plugins_dir=/usr/local/lib/mapnik/input
23 font_dir=/usr/share/fonts/dejavu
24 font_dir_recurse=1
25
26 [default]
27 URI=/osm_tiles/
28 TILEDIR=/var/lib/mod_tile
29 XML=/usr/local/share/maps/style/OSMDefault.xml
30 HOST=<your-ec2-dashed-ip-zone>.compute.amazonaws.com
31 TILESIZE=256
.
.
.
Set up the directories required for the mod_tile system to run:
$ sudo mkdir /var/run/renderd
$ sudo chown ec2-user:apache /var/run/renderd
$ sudo chmod 0775 /var/run/renderd
$ sudo mkdir /var/lib/mod_tile
$ sudo chown ec2-user:apache /var/lib/mod_tile
$ sudo chmod 0755 /var/lib/mod_tile
The Apache HTTPD on AMI Linux is configured via several configuration files at /etc/httpd/
NOTE: Do not forget to configure port 80 (and 443 for TLS) for your EC2 instance to make the httpd server available for the world.
The mod_tile.so file has already been deployed in /etc/httpd/modules, to configure mod_tile for httpd:
$ cd /etc/httpd/conf.modules.d/
$ sudo echo - n "LoadModule tile_module modules/mod_tile.so" > 00-mod_tile.conf
The renderd configuration can be added into httpd's main configuration file at /etc/httpd/conf/httpd.conf.
- open the httpd.conf with sudo in your text editor and locate the ServerName configuration option
- change ServerName to match <your-ec2-dashed-ip-zone>.compute.amazonaws.com
- insert the renderd configuration below
# renderd
LoadTileConfigFile /usr/local/etc/renderd.conf
ModTileRenderdSocketName /var/run/renderd/renderd.sock
# Timeout before giving up for a tile to be rendered
ModTileRequestTimeout 0
# Timeout before giving up for a tile to be rendered that is otherwise missing
ModTileMissingRequestTimeout 30
To run httpd:
$ sudo service httpd start
The command line outuput should look something like:
Starting httpd: [Thu Jun 18 06:03:12.519392 2015] [tile:notice] [pid 23221] Loading tile config default at /osm_tiles/ for zooms 0 - 20 from tile directory /var/lib/mod_tile with extension .png and mime type image/png
[ OK ]
First start renderd (foreground)
$ sudo -u ec2-user renderd -f -c /usr/local/etc/renderd.conf
Open a browser in your computer and navigate to:
http://<your-ec2-dashed-ip-zone>.compute.amazonaws.com/osm_tiles/0/0/0.png
If all goes well you should see a small tile of Earth. Also, look at the renderd output for troubleshooting issues.