no space left on device

[hugokoopmans]

I have used osmconvert to merged 6 countries to one pbf, imported this files into my test setup, all fine and running ok.

Now I replicate this on the production server, exact same setup, exact same procedure and I bump into this:

renderd[121]: ERROR: failed to render TILE ajt 2 0-3 0-3
renderd[121]:    reason: Postgis Plugin: ERROR:  could not resize shared memory segment "/PostgreSQL.790133961" to 12615680 bytes: No space left on device
in executeQuery Full sql was: 'SELECT ST_AsBinary("way") AS geom,"feature","name","way_pixels" FROM (SELECT
    way,
    way_area/NULLIF(39135.8::real*39135.8::real,0) AS way_pixels,
    COALESCE(
      'landuse_' || CASE WHEN landuse IN ('forest', 'military', 'farmland') THEN landuse ELSE NULL END,
      'natural_' || CASE WHEN "natural" IN ('wood', 'glacier', 'sand', 'scree', 'shingle', 'bare_rock',
                                            'water', 'bay') THEN "natural" ELSE NULL END,
      'place_' || CASE WHEN place IN ('island') THEN place ELSE NULL END,
      'boundary_' || CASE WHEN boundary IN ('national_park') THEN boundary ELSE NULL END,
      'leisure_' || CASE WHEN leisure IN ('nature_reserve') THEN leisure ELSE NULL END
    ) AS feature,
�
,
    CASE WHEN building = 'no' OR building IS NULL THEN 'no' ELSE 'yes' END AS is_building -- always no with the where conditions
  FROM planet_osm_polygon
  WHERE (landuse IN ('forest', 'military', 'farmland')
      OR "natural" IN ('wood', 'glacier', 'sand', 'scree', 'shingle', 'bare_rock', 'water', 'bay')
      OR "place" IN ('island')
      OR boundary IN ('national_park')
      OR leisure IN ('nature_reserve'))
    AND building IS NULL
    AND name IS NOT NULL
  ORDER BY way_area DESC
) AS text_poly_low_zoom WHERE "way" && ST_SetSRID('BOX3D(-20037508 -20037508,20037508 20037508)'::box3d, 3857)'
renderd[121]: DEBUG: DONE TILE ajt 2 0-3 0-3 in 135.980 seconds

I copied the merged file over to the production server, checked de md5sum to make sure the file is not corrupted.

Import ran without issues, container stopped when done, same as on test setup.
I checked disk usage on the server space enough according to to OS:

df -h
Filesystem                            Size  Used Avail Use% Mounted on
udev                                  3.9G     0  3.9G   0% /dev
tmpfs                                 798M  1.4M  797M   1% /run
/dev/mapper/37--97--158--90--vg-root  288G  103G  170G  38% /
tmpfs                                 3.9G  3.1M  3.9G   1% /dev/shm
tmpfs                                 5.0M     0  5.0M   0% /run/lock
tmpfs                                 3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/vda1                             236M  127M   98M  57% /boot

What could be the case?

[Istador]

Please run the df -h command inside the container and not on your host system.

I highly suspect that /dev/shm (shared memory) inside the container is full.

This happens because the docker default is 64M, but the postgres default setting for shared memory is 128M. Therefore postgres tries to allocate more than is available and fails.

Solutions:

• docker run option: --shm-size=128M
• docker-compose option: shm_size: 128M

Better use more than 128M - just to be sure.

Because you have a productive environment, you might want to take a look at my post in issue #5

[Istador]

According to your posts in issue #5 you seem to have fixed your issue now.

---

I suggested using docker-compose because it's easy to mess up calling docker directly with a lot of arguments, and it's hard to figure out what is currently running and with what specific arguments.

The idea behind the docker-compose.yml file is that you define in it your "services" that should run, what they are composed of (image, volumes, networks, etc.), their configuration, with which other services they work together or what ports they expose to the outside. Once that is defined in a file the docker-compose command is used to start (e.g. with up -d), stop, restart, rebuild, monitor, etc. the services.

In a nutshell docker-compose is just a python script that reads your docker-compose.yml configuration file and calls the docker commands for you with the right arguments according to what you have defined in it. It's a convenience tool. Like, without it you'd need to pass --shm-size=128M (together with all volumes and environment variables) to your docker command every time you want to (re)start the tileserver. (Yes, the shared memory will still be a problem while running the tileserver, not just during the initial import - because while running it, it'll render new tiles and uses the postgres database to do that.)

One suggestions for your docker-compose service definition posted in issue #5 would therefore be to change the command from import to run, because running the tileserver is what the service is usually for and importing the database is something you only want to do manually once (and only then the data.osm.pbf needs to be mounted).

I gave two examples for docker-compose a few days ago: one in issue #18, and one in combination with a https proxy in issue #19.

[stevo01]

Please run the df -h command inside the container and not on your host system.

I highly suspect that /dev/shm (shared memory) inside the container is full.

I can confirm that usage of your recommented option "--shm-size" solves the problem. I got the "no space left on device" message during request of tile in zoom level "8". I increased shm-size to value of 1GB because 128MB was still to small.

[Overv]

@stevo01 So the lack of shared memory can cause both "no space left on device" and "could not resize shared memory segment" errors?

[stevo01]

@stevo01 So the lack of shared memory can cause both "no space left on device" and "could not resize shared memory segment" errors?

Yes, because it is the same error. See error message from @hugokoopmans first comment:

renderd[121]: ERROR: failed to render TILE ajt 2 0-3 0-3
renderd[121]: reason: Postgis Plugin: ERROR: could not resize shared memory segment "/PostgreSQL.790133961" to 12615680 bytes: ### No space left on device

[hugokoopmans]

ok I have moved the tileserver to a dedicated virtual machine with better resources ( 4 cpu 16 GB RAM). In the setup above there where 8 more containers hungry for resources. Not sure if that caused issues.

I included these options in docker-compose:

THREADS: 8
shm_size: 256M

Seems to run fine now.
Closing, thx.

[Istador]

I included these options in docker-compose:

THREADS: 8
shm_size: 256M

Reading that, I just want to make sure to point out, that these go to different places inside the docker-compose.yml file.

THREADS is an arbitrary environment variable introduced by this docker image that is listed under the environment key (or in another file referenced with env_file) and shm_size is a docker-compose specific service option.

services:
  osm:
    [...]
    environment:
    - THREADS=8
    shm_size: 256M
    [...]

[hugokoopmans]

on docker-compose versio 3 the - THREADS=8 gives me:

root@sensa-7:/docker/s7# docker-compose -f tileserver.yml up -d
ERROR: yaml.parser.ParserError: while parsing a block mapping
  in "./tileserver.yml", line 14, column 9
expected <block end>, but found '-'
  in "./tileserver.yml", line 20, column 9

[Istador]

on docker-compose versio 3 the - THREADS=8 gives me:

root@sensa-7:/docker/s7# docker-compose -f tileserver.yml up -d
ERROR: yaml.parser.ParserError: while parsing a block mapping
  in "./tileserver.yml", line 14, column 9
expected <block end>, but found '-'
  in "./tileserver.yml", line 20, column 9

It's hard to tell what's wrong without the tileserver.yml, because the error message refers to a specific line and column in it.

Based on the error message the file doesn't contain valid YAML. I'd suspect that the indentation is wrong.

Example that results in a similar error:

version: '3'
services:
  myservice:
    image: alpine
    environment:
  - THREADS=8 # <- indentation error
    shm_size: 256M

$ docker-compose run myservice sh -c 'echo $THREADS'
ERROR: yaml.parser.ParserError: while parsing a block mapping
  in "./docker-compose.yml", line 3, column 3
expected <block end>, but found '-'
  in "./docker-compose.yml", line 6, column 3

Correct indentation:

version: '3'
services:
  myservice:
    image: alpine
    environment:
    - THREADS=8
    shm_size: 256M

$ docker-compose run myservice sh -c 'echo $THREADS'
8

[suneet-nokia]

I'm facing this same error, on kubernetes. Any help will be deeply appreciated
https://gis.stackexchange.com/questions/382093/pre-rendering-tiles-fail-only-once-in-a-while

[finisher0]

Looking for some tips to resolve the "no space left" issue as well.

Setup

CentOS 7 VM with
62GB RAM
16GB Swap
2T SSD

Docker Commands:

docker run \
  -v "<absolute path>/planet-latest.osm.pbf":/data.osm.pbf \
  -e THREADS=4 \
  -v planet-osm-data:/var/lib/postgresql/12/main \
  -v planet-osm-rendered-tiles:/var/lib/mod_tile \
  -v planet-osm-nodes:/nodes \
  -e "OSM2PGSQL_EXTRA_ARGS=--flat-nodes /nodes/flat_nodes.bin" \
  -e "OSM2PGSQL_EXTRA_ARGS=-C 16107 \
  planet-osm-tile-server \
  import

docker run \
  -p 8080:80 \
  -p 5550:5432 \
  -e THREADS=8 \
  --shm-size="256m" \
  -e "OSM2PGSQL_EXTRA_ARGS=-C 16107" \
  -v planet-osm-data:/var/lib/postgresql/12/main \
  -v planet-osm-rendered-tiles:/var/lib/mod_tile \
  -d planet-osm-tile-server \
  run

Dockerfile:

NOTE: I am using an edited version of the mapnik.xml file in which I request labels in English when they are available. The edits have been tested several times and so I don't have any reason to think they are causing the issue.

FROM overv/openstreetmap-tile-server

# Use modified Mapnik file to prefer English over native labels
COPY mapnikPreferEnglish.xml /home/renderer/src/openstreetmap-carto/mapnik.xml   # This file is tested and works
COPY postgresql.custom.conf.tmpl /etc/postgresql/12/main/postgresql.custom.conf.tmpl   # This file probably needs some edits

Contents of my custom PostgreSQL config (i.e. postgresql.custom.conf.tmpl):

# Suggested minimal settings from
# https://ircama.github.io/osm-carto-tutorials/tile-server-ubuntu/

shared_buffers = 128MB
min_wal_size = 1GB
max_wal_size = 2GB
maintenance_work_mem = 256MB

autovacuum_vacuum_scale_factor = 0.05
autovacuum_analyze_scale_factor = 0.02

# Suggested settings from
# https://github.com/openstreetmap/chef/blob/master/roles/tile.rb#L38-L45

max_connections = 250
temp_buffers = 32MB
work_mem = 32MB
wal_buffers = 1024kB
wal_writer_delay = 500ms
commit_delay = 10000
# checkpoint_segments = 60 # unrecognized in psql 10.7.1
max_wal_size = 2GB
random_page_cost = 1.1
track_activity_query_size = 16384
autovacuum = off

effective_io_concurrency = 50 # This should probably have been less than 10 since we are using an HHD (https://www.postgresql.org/docs/10/runtime-config-resource.html)
checkpoint_timeout = 1h
checkpoint_completion_target = 0.9

fsync = off
synchronous_commit = off


listen_addresses = '*'

Result

Planet import fails with ERROR: could not extend file "base/16385/6348049.7": No space left on device.

Here's an excerpt from the import container log:

...
2022-01-02 19:50:33  Done postprocessing on table 'planet_osm_nodes' in 0s
2022-01-02 19:50:33  Building index on table 'planet_osm_ways'
2022-01-02 19:50:34  Building index on table 'planet_osm_rels'
2022-01-03 04:42:25  ERROR: Database error: ERROR:  could not extend file "base/16385/7042294.254": No space left on device
HINT:  Check free disk space.

And from the postgresql log inside the container (/postgresql-12-main.log):

...
2022-01-02 20:06:23.955 UTC [29] LOG:  checkpoints are occurring too frequently (18 seconds apart)
2022-01-02 20:06:23.955 UTC [29] HINT:  Consider increasing the configuration parameter "max_wal_size".
2022-01-02 20:06:36.007 UTC [29] LOG:  checkpoints are occurring too frequently (13 seconds apart)
2022-01-02 20:06:36.007 UTC [29] HINT:  Consider increasing the configuration parameter "max_wal_size".
2022-01-02 20:06:40.740 UTC [117] renderer@gis ERROR:  could not extend file "base/16385/6348049.7": No space left on device
2022-01-02 20:06:40.740 UTC [117] renderer@gis HINT:  Check free disk space.
2022-01-02 20:06:40.740 UTC [117] renderer@gis STATEMENT:  CREATE TABLE "planet_osm_polygon_tmp"  AS SELECT * FROM "planet_osm_polygon" ORDER BY way
2022-01-02 20:06:40.740 UTC [116] renderer@gis ERROR:  could not extend file "base/16385/6348051.28": No space left on device
2022-01-02 20:06:40.740 UTC [116] renderer@gis HINT:  Check free disk space.
2022-01-02 20:06:40.740 UTC [116] renderer@gis STATEMENT:  CREATE TABLE "planet_osm_line_tmp"  AS SELECT * FROM "planet_osm_line" ORDER BY way
2022-01-03 04:42:11.087 UTC [7907] renderer@gis ERROR:  could not extend file "base/16385/7042294.254": No space left on device
2022-01-03 04:42:11.087 UTC [7907] renderer@gis HINT:  Check free disk space.

I'll go ahead and increase my wal_max_size, but I checked the storage usage, and it looks like I still have over 500GB available.

Output of df -h:

    Filesystem      Size  Used Avail Use% Mounted on
    devtmpfs      32G     0   32G   0% /dev
    tmpfs            32G     0   32G   0% /dev/shm
    tmpfs            32G  1.1G   31G   4% /run
    tmpfs            32G     0   32G   0% /sys/fs/cgroup
    /dev/sda1    2.0T  1.5T  536G  74% /
    tmpfs           6.3G     0  6.3G   0% /run/user/1000
    tmpfs           6.3G     0  6.3G   0% /run/user/7361

In case it's help, here's the output of lsblk as well:

NAME                                                                                       MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
fd0                                                                                           2:0    1    4K  0 disk
sda                                                                                           8:0    0    2T  0 disk
└─sda1                                                                                    8:1    0    2T  0 part /
sdb                                                                                           8:16   0   16G  0 disk
└─sdb1                                                                                    8:17   0   16G  0 part [SWAP]
sr0                                                                                          11:0    1 1024M  0 rom
loop0                                                                                        7:0    0  100G  0 loop
└─docker-8:1-536874549-pool                                           253:0    0  100G  0 dm
loop1                                                                                        7:1    0    2G  0 loop
└─docker-8:1-536874549-pool                                           253:0    0  100G  0 dm
loop2                                                                                        7:2    0  100G  0 loop
└─docker-8:1-1615076197-pool                                         253:1    0  100G  0 dm
  └─docker-8:1-1615076197-f9b5c4bcaa7cc468c39935d86b5edad96a61f848d3cc64bcd6d25eb2f493e914 253:2    0   10G  0 dm   /var/lib/docker/devicemapper/mnt/f9b5c4bcaa7cc468c39935d86b5edad96a61f848d3cc64bcd6d25eb2f493e914
loop3                                                                                        7:3    0    2G  0 loop
└─docker-8:1-1615076197-pool                                         253:1    0  100G  0 dm
  └─docker-8:1-1615076197-f9b5c4bcaa7cc468c39935d86b5edad96a61f848d3cc64bcd6d25eb2f493e914 253:2    0   10G  0 dm   /var/lib/docker/devicemapper/mnt/f9b5c4bcaa7cc468c39935d86b5edad96a61f848d3cc64bcd6d25eb2f493e914

Any tips on how to resolve this space issue? Or on tuning my the configs?

If necessary, I can increase my disk size to 3 of 4 T, but I'm not certain that's the issue. I should mention that I've completed a couple of full-planet imports using this same machine. However, with previous imports I was running into issues attempting to render several zoom levels of the planet. Hoping to resolve the render issues, I increased the shared memory and cache sizes for the import and run commands. To support these changes I am required tuning the PostgreSQL config. I've attempted to do so using using several suggestions given throughout this repository (and links provided in various issues). However, I'm not confident that my new config settings are appropriate.

Thanks in advance for you assistance.