rendering issues on full planet import

[cstaerkel]

Hi,

I have some rendering issues on full planet import:
The import completes without errors, but some details are missing from rendered tiles. The tiles from my server lack at least lakes, rivers and building footprints, please take a look at the screenshot attached.

If I import the data for a smaller area, I don't get any render issues. Any pointers are really appreciated.

Thanks,
Chris

[Overv]

Do you only have this issue with lakes and not oceans?

[Istador]

I'm not working with a full planet import (only europe & africa) and I didn't run into your particular problem, but even at that size I ran into other import problems working with this project (postgres db size: ~400 GiB, 48 core system, 128 GB RAM, daily differential updates).

Judging from the image posted, the database seems to lack content. The import might not have been completed or there were errors that weren't printed / or that you didn't noticed.

Sadly I can't send in pull requests (yet), for performance issues, bugs and improvements, without the approval of my employer, but I can give you the following advise:

• you want to change the osm2pgsl import to use the flatnodes parameter to save time and space (the flatnodes file needs to be saved and kept within a volume).
• osm2pgsl has a -C parameter to increase the allowed memory during the import. It is hardcoded to 2 GiB inside this project's run.sh (I think, but am not sure right now, that there was a similar settings for the renderd.conf).
• the postgres config should be tuned (I had to increase max_connections to make the import possible at all - probably because of a higher THREADS value).
• the shared memory (shm-size) setting of the docker container needs to be increased, because the default (64 MB) isn't big enough for postgres (128 MB).
• use docker-compose.
• be aware that docker on linux saves volumes by default into /var/lib/docker/volumes/. You might want to change the paths on the host system manually (via local volume driver).
• check that the storage device that contains the database volume hasn't run full (extrapolating, from europe & africa 20.8 GiB PBF to 400 GiB, planet.osm with 41 GiB PBF might be around 790 GiB imported into postgres?).
• try to delete the contents of the tiles volume, so that you get freshly rendered tiles and aren't served with tiles that might have been created while the import was still in progress.

Outdated but still useful (2012):

• http://www.geofabrik.de/media/2012-09-08-osm2pgsql-performance.pdf

[cstaerkel]

Thank you guys for your answers.
@Overv: the issue is only with lakes, oceans are rendered correctly. I did an import of Europe yesterday and everything seems to render fine and the server is very responsive.
@Istador
My machine has an i7-8700 with 6 physical/12 hyperthreading cores 64 GB of RAM and 1 TB of NVMe Storage. I'm running the import with the -e THREADS=12 parameter. I did not notice any error messages on import, and I don't think I ran out of space. I will try to follow your suggestions and adjust the memory usage on import, the shm-size value and the postgres config. Again, thanks for the pointers

I'll keep you posted

[hugokoopmans]

#5 (comment)

Her you say:

• the shared memory (shm-size) setting of the docker container needs to be increased, because the default (64 MB) isn't big enough for postgres (128 MB).
• use docker-compose.

I cant seem to get this working?

in docker compose I do this:

version: '3'
services:
  tileserver:
    container_name: tileserver
    image: overv/openstreetmap-tile-server
    volumes:
        - '/data/docker/tileserver/merged.osm.pbf:/data.osm.pbf'
        - 'openstreetmap-data:/var/lib/postgresql/10/main'
    environment:
        shm_size: '256M'
    command: import

volumes:
  openstreetmap-data:
    external: true

But inside the container df -hT /dev/shm still says:

root@2dc9518b5eb9:/home/renderer/src/openstreetmap-carto# df -hT /dev/shm
Filesystem     Type   Size  Used Avail Use% Mounted on
shm            tmpfs   64M  8.0K   64M   1% /dev/shm

Any suggestions?

[hugokoopmans]

Docker version 18.09.2, build 6247962
docker-compose version 1.24.0-rc1, build 0f3d4dda

If using the build option:

    build:
        context: .
        shm_size: '256M'

i get :

ERROR: The Compose file './tileserver-import.yml' is invalid because:
services.tileserver.build contains unsupported option: 'shm_size'

EDIT1:
updated docker-compose to 1.24, still the above error
(Apperently I need to update docker-compose for this to work?)

FIXED:
make sure the file version is '3.5' (I had still '3')

see: https://stackoverflow.com/questions/51517189/docker-compose-error-build-contains-unsupported-option-shm-size

But still the default SHM size of 64 MB ...?

EDIT2:

I toltally messed up sorry ofcourse this is the right complete compose file:

version: '3.5'
services:
  tileserver:
    container_name: tileserver
    image: overv/openstreetmap-tile-server
    volumes:
        - '/data/docker/tileserver/merged.osm.pbf:/data.osm.pbf'
        - 'openstreetmap-data:/var/lib/postgresql/10/main'
    shm_size: '256mb'
    command: import

volumes:
  openstreetmap-data:
    external: true

Somehow I thought the option needed to be in 'environment' or 'build' or something ....

[stevo01]

I have some rendering issues on full planet import:
The import completes without errors, but some details are missing from rendered tiles. The tiles from my server lack at least lakes, rivers and building footprints, please take a look at the screenshot attached.

If I import the data for a smaller area, I don't get any render issues. Any pointers are really appreciated.

I observed semilar problem of my full planet import. The attached sample shows a comparsion with osm map in zoom level 8 and 9. In zoom level 8 the lake is visible in zoom level 9 not.

[stevo01]

another sample from city of berlin. In zoom level 13 some parts of the highway are missing. In zoom level 12 everything looks fine.

[raphael-sch]

I had a similar problem and tracked it down to an issue with too many database connections.

Connection to database failed: FATAL: sorry, too many clients already

On default postgres allows 100 simultaneously connections. When running osm2pgsql on a server with multiple cores, it can hapen that more connections are open at the same time. to solve this problem just start postgres with
command: postgres -c 'max_connections=300'
in your docker-compose file

[Istador]

I had a similar problem and tracked it down to an issue with too many database connections.

Connection to database failed: FATAL: sorry, too many clients already

On default postgres allows 100 simultaneously connections. When running osm2pgsql on a server with multiple cores, it can hapen that more connections are open at the same time. to solve this problem just start postgres with
command: postgres -c 'max_connections=300'
in your docker-compose file

Increasing max_connections was already mentioned here in this issue #5 (comment). If you're using this docker image as it is, it is using the postgresql.custom.conf.tmpl file, which increases it from 100 to 250. But that might even be too less for some setups (see #32 (comment) for details).

If you try to separate postgres into an own docker-compose service, you might also want to consider the other settings from postgresql.custom.conf.tmpl.

[raphael-sch]

You are correct, I forgot that I heavily modified this project. However as you noted, 250 can still be too few connections. osm2pgsql opens multiple connections per process and people with big machines could set the processes too high. Maybe max_connections should scale with number of processes for osm2pgsql.