Does not work with official postgres:10 due to shutdown issues

[EugenMayer]

First off, thank you SO much for your work here - incredible!

Today i tried to upgrade from postgres:10 to postgres:11 - so using the official images. The problem is actually, that there seem to be no way to "cleanly" shutdown the "old" docker container so that this "counts" for pg_upgrade to be a clean shutdown and thus allow the migration.

What i had to do is (pg10 stands for your current docker container running pg old):

docker exec -it pg10 bash
su - postgres
export PGDATA=/var/lib/postgresql/data
/usr/lib/postgresql/10/bin/pg_ctl stop

This will now properly shutdown the container. Everything else goes just as you described in the README.md

docker run --rm  --name upgrade -u postgres -v /host/db/10:/var/lib/postgresql/10/data -v /host/db/11:/var/lib/postgresql/11/data tianon/postgres-upgrade:10-to-11

I am not sure we can fix the former (stop issue) here in the docker image automatically, but we can at least add it to the readme. What do you think?

Again, thank you so much!

[tianon]

I'd like to understand the shutdown issue better -- what's making docker stop not work properly?

[EugenMayer]

well as usual we would have 2 possible options:
a) the default kill signal does not suit the official postgresql image so the server does not shutdown in the terms it would need for a upgrade
b) the kill-timeout is to short, pg is shutting down slower then the timeout so the docker engine kills the process - unclean shutdown
c) the "upgrade" prodcess needs the server to have a very "deep shutdown" - i have no clue if there is a thing or not, but maybe the usual shutdown is just not enough here

i could reproduce this issue with about 6-8 start / stops - all unclean in terms of upgrade.

You should be able to reproduce this with

mkdir /tmp/db10 /tmp/db11
docker run--name pg10 -d -v /tmp/db10:/var/lib/postgresql/data postgres:10
# wait for some time, like 60 seconds
docker stop pg10
docker rm pg10

docker run --rm  --name upgrade -u postgres -v /tmp/db10:/var/lib/postgresql/10/data -v /tmp/db11:/var/lib/postgresql/11/data tianon/postgres-upgrade:10-to-11

[tianon]

So that sounds like we need to recommend a --stop-timeout on PostgreSQL instances (which IMO is a good idea for databases in general, given that stopping them uncleanly is Bad), and here perhaps we should recommend using docker stop's --time flag directly to increase the time from 10 seconds to something much larger?

I honestly always run my PostgreSQL instances with --stop-timeout 120 because I want to make sure that it's always got as much time as I can reasonably give it to shut down cleanly.

[tianon]

Better yet, maybe we ditch docker stop entirely and instead recommend users directly do docker kill --signal and wait for the database to shut down? (Although that won't overcome a restart policy like docker stop will, so probably not a great plan.)

[cmattoon]

FYI, I ended up having to run docker --update restart=no $OLD_CONTAINER_ID before exec-ing into the container to do these steps, but can confirm that stopping postgres this way works.

The error I was getting was:

There seems to be a postmaster servicing the old cluster.
Please shutdown that postmaster and try again.
Failure, exiting

[snspinn]

With docker-compose running docker-compose down -t 120 appears to have worked (as did some more 'creative' solutions).

As tianon mentioned, this avoids running into restart issues.

[tianon]

Adding a suitably large stop timeout is definitely good, but I think the core issue of the OP here is mostly resolved by docker-library/postgres#763. 👍