How to determine when new container is truly "ready"

[rarkins]

To wait for a postgres container to be "ready", I've been using a script like this:

# Wait for PG to be ready
until $PSQL -c "select version()" &> /dev/null
do
    echo "waiting for postgres container..."
    sleep 2
done

But it seems even this is too early. Is there any unequivocal way that another container can tell once the postgres container is ready to be provisioned, without resorting to more arbitrary wait timers?

[spmacdonald]

I have done this:

while ! pg_isready 
do
    echo "$(date) - waiting for database to start"
    sleep 10
done

and it seems to work, but I am still learning...

[rarkins]

Hi @spmacdonald,
Your sleep 10 is still what I'd call an "arbitrary wait timer". The challenge (problem?) with the current postgres image is that it is essentially "ready" twice - so the first time it reports ready it's not really ready:

Here's the first time:

postgres          | LOG:  database system is ready to accept connections
postgres          | LOG:  autovacuum launcher started
postgres          |  done
postgres          | server started
postgres          | ALTER ROLE

Then it runs its entrypoints, e.g. here is an example if adding postgis:

postgres          | /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/postgis.sh
postgres          | CREATE DATABASE
postgres          | UPDATE 1
postgres          | Loading PostGIS extensions into template_postgis
postgres          | CREATE EXTENSION
postgres          | CREATE EXTENSION
postgres          | CREATE EXTENSION
postgres          | CREATE EXTENSION
postgres          | Loading PostGIS extensions into postgres
postgres          | CREATE EXTENSION
postgres          | CREATE EXTENSION
postgres          | CREATE EXTENSION
postgres          | CREATE EXTENSION

And then the restart:

postgres          | LOG:  received fast shutdown request
postgres          | LOG:  aborting any active transactions
postgres          | LOG:  autovacuum launcher shutting down
postgres          | waiting for server to shut down....LOG:  shutting down
postgres          | LOG:  database system is shut down
postgres          |  done
postgres          | server stopped
postgres          |
postgres          | PostgreSQL init process complete; ready for start up.
postgres          |
postgres          | LOG:  database system was shut down at 2016-04-29 03:31:41 UTC
postgres          | LOG:  MultiXact member wraparound protections are now enabled
postgres          | LOG:  database system is ready to accept connections
postgres          | LOG:  autovacuum launcher started

And only now is it truly ready. If you kicked off a seeding/provisioning script once you saw PG "ready" the first time, you're likely to hit a failure when it restarts. I think all containers need to have a deterministic way to indicate when they're actually-truly "won't restart on you" ready without resorting to sleep timers.

[md5]

@rarkins What is the $PSQL in your command? Is it connecting to a separate container? If so, then I'm surprised that it's "too early". The initialization process you're talking about in this container should not be exposing ports in a way that anything outside the container can connect.

[rarkins]

@md5 sorry, $PSQL is just an alias defined earlier in my file:

PSQL="psql -h localhost -p 5432 -U postgres -v ON_ERROR_STOP=1"

I am using host networking in docker, which might explain the difference?

[md5]

@rarkins You're going to have a hard time with a lot of assumptions in various Docker images if you're using host networking.

[yosifkit]

The way the container is designed is that when it is listening on the network port via its non-loopback interface, it is ready to accept connections. Accepting connections on the loopback connection is for the scripts in /docker-entrypoint-initdb.d/; once the scripts are done, the server is killed and then started as pid 1 of the container.

[gravis]

The best way to wait for your container to be up and ready is this:

until docker run -it --rm --link [YOUR-PG-CONTAINER-NAME]:pg postgres:9.5 psql -U postgres -h pg -c "select 1" -d postgres; do sleep 1; done

That's what we're using at Gemnasium, it avoid the "double start" issue.

[sydcanem]

We do sleep 2 after done to accomodate the next restart.

until pg_isready
do
    echo "."
    sleep 1
done
sleep 2

[yelizariev]

improved version of @gravis's solution

 until docker run --rm --link [YOUR-PG-CONTAINER-NAME]:pg postgres:9.5 pg_isready -U postgres -h pg; do sleep 1; done

it works even if password is required

Update as noted by @enumag if you use network, then you need to add --net too:

until docker run --rm --link [CONTAINER]:pg --net [NETWORK] postgres:9.5 pg_isready -U postgres -h pg; do sleep 1; done

[jasonkeene]

@gravis @yelizariev Thank you for your solutions!

[fdietze]

Is the -w option for pg_ctl helpful here?
https://www.postgresql.org/docs/current/static/app-pg-ctl.html

-w
Wait for the startup or shutdown to complete. Waiting is the default option for shutdowns, but not startups. When waiting for startup, pg_ctl repeatedly attempts to connect to the server. When waiting for shutdown, pg_ctl waits for the server to remove its PID file. This option allows the entry of an SSL passphrase on startup. pg_ctl returns an exit code based on the success of the startup or shutdown.

[valk8]

As @rarkins I've struggled with the server restart too. pg_isready is not helpful if you apply a schema, seed a data right away (e.g CI workflow).
I got inspired by https://github.com/vishnubob/wait-for-it
which Docker mentions in its docs here https://docs.docker.com/compose/startup-order/

HOST=db
PORT=5432
RETRIES=10

until docker-compose exec container bash -c '(echo > /dev/tcp/$HOST/$PORT) > /dev/null 2>&1' || [ $RETRIES -eq 0 ]; do
    echo "Waiting for Postgres server, $((RETRIES--)) remaining attempts..."
        sleep 1
    done

[cjlint]

The simplest solution for me was watching the logs for PostgreSQL init process complete; ready for start up, and then watching the logs for database system is ready to accept connections

[vitorreis]

thanks for the solution @gravis

[enumag]

I tried to use the solution posted by @gravis and later improved by @yelizariev. In my case there was one more problem:

docker: Error response from daemon: Cannot link to /[CONTAINER], as it does not belong to the default network.

I found a fix here. I just had to add --net [NETWORK] to the command, where network name can be found using docker network ls.

Here is the final version:

until docker run --rm --link [CONTAINER]:pg --net [NETWORK] postgres:9.5 pg_isready -U postgres -h pg; do sleep 1; done

[Mobe91]

This should be solved if #282 is implemented, right?

[enumag]

The problem is that you can't use healthcheck when using docker-compose.yml version 3 since depends_on no longer supports condition.

https://docs.docker.com/compose/compose-file/#depends_on

[mausch]

As explained in this comment, solutions that merely check for connectivity (using psql or pg_isready or anything else) will not work in the general case with this image since it does a postgres restart after the initial setup (if you are mounting an existing database you're fine though).
You need a solution like the one outlined by cjlint.
Here's a bash snippet that does that:

for i in {1..7}; do
    docker logs <YOUR_CONTAINER> 2>&1  | grep -Pzl '(?s)init process complete.*\n.*ready to accept connections'
    if [ $? -eq 0 ]; then
        break
    fi
    if [ $i -eq 7 ]; then
        echo "Postgres did not start up successfully"
        exit 1
    fi
    sleep 2
done

[enumag]

@mausch Unfortunately your solution is can't be used it from inside of another container - which is what you want most of the time. Any tips for that situation?

[MaerF0x0]

For future google hits:

I'm using /usr/local/bin/dockerize -wait tcp://localhost:5432 -timeout 1m in a CI job .

[sizief]

If using docker version 2.1

healthcheck:                                                                 
  test: ["CMD-SHELL", "psql -h localhost -p 5432 -U postgres -v ON_ERROR_STOP=1 -c 'select version()' &> /dev/null"]
   test: ["CMD-SHELL", "pg_isready -U postgres"] #Are you really up?    
    interval: 2s                                                            
    timeout: 30s                                                            
    retries: 15    

and then

depends_on:                                                                 
  db:                                                                       
     condition: service_healthy   

[dliff]

@enumag
Been struggling with this as I cannot access the logs for my db container from web which @mausch 's solution seems to require (unless there is a way to, I'm actually new to Docker).

The other solutions aren't effective because my docker image, the kartoza/postgis with Postgis, seems to actually fully start Postgres once, then restart. So it will pass health check one time, then restart.

For those of us whose Postgres image needs to start x amount of times, I have this shell script that uses a pg function to get the pid and keep track of how many distinct times Postgres has started.

Have not tested this on platform besides Linux, or with a goal other than 2.

count=0
goal=2

lastpid=""
pid=""

function getpid()
{
    lastpid="$pid"
    pid=$(PGPASSWORD=$POSTGRES_PASSWORD psql "$DB_NAME" -At -h "$HOST" -U "$POSTGRES_USER" -c 'SELECT pg_backend_pid()')
}

until getpid && [ "$count" -eq "$goal" ]; do
    >&2 echo "Postgres has started $count out of $goal time(s)."
    if [ "$pid" ] && [ "$pid" != "$lastpid" ];
    then
       count=$((count+1))
    fi
    sleep 1
done

>&2 echo "Postgres has started $goal time(s)... executing command"

[ezpuzz]

Use this:
( docker-compose logs -f postgres & ) | grep -m2 'ready to accept'
or
( docker logs -f postgres & ) | grep -m2 'ready to accept'

[tianon]

As noted back in #146 (comment), the best way to test if the container is "ready" is to connect to it using its external IP address (it does not listen externally until the initialization process is fully complete).

In the future, these sorts of questions/requests would be more appropriately posted to the Docker Community Forums, the Docker Community Slack, or Stack Overflow.

[mboyea]

I came across this while building a devtool that starts a postgres database inside a Docker container.

Some of these solutions don't deal with the "first start" "second start" issue.
My solution was to store the output of the container using tee, and then wait for that file to contain a line with [1] followed by database system is ready to accept connections.
I'm not quite sure what [1] means, but I know this text is reliably output to the log, only at the "second start" for the version of postgres (15) that my devtool uses.
And it still behaves this way even if the container skips the "first start" (/var/run/postgresql/data has a database).
In this script excerpt, the database is started as a background process that is reliably destroyed when the script exits.

process_ids=()
kill_processes() {
  # send kill signal to each process
  for process_id in "${process_ids[@]}"; do
    kill "$process_id" > /dev/null 2>&1
  done
  # wait for each process to exit
  for process_id in "${process_ids[@]}"; do
    wait "$process_id" 2>/dev/null
  done
}
trap kill_processes EXIT

# start database process
database_log_file="$(mktemp)"
"$START_DATABASE" 2>&1 | tee "$database_log_file" & process_ids+=($!)
database_process_id="${process_ids[-1]}"
# wait for the database to initialize and be ready to accept connections
until \
  grep -q "^.*\[1\].*database system is ready to accept connections" "$database_log_file"
do
  # check that the database process still exists
  if ! ps -p "$database_process_id" > /dev/null; then
    echo_error "The database failed to start"
    exit 1
  fi
  sleep 0.1
done
# check that the database is accessible at port 5432
if ! netstat -tulpn 2>&1 | grep ":5432 " > /dev/null 2>&1; then
  echo_error "The database did not bind to port 5432"
  exit 1
fi

# TODO: start the server that connects to the database

$START_DATABASE can be docker container run [OPTIONS] IMAGE [COMMAND] [ARG...]

In my case, it is podman container run --publish 5432:5432 --env 'POSTGRES*' --volume lacuna-postgres-docker-image-0.0.0:/var/lib/postgresql/data localhost/lacuna-postgres-docker-image:0.0.0

[tianon]

I'm not quite sure what [1] means, but I know this text is reliably output to the log, only at the "second start" for the version of postgres (15) that my devtool uses.

It's the PID -- so, if you use --init either on docker run or on dockerd (or in daemon.json), it will probably be ~[7] instead.

[calebpitan]

Tested this works, you can thank me later if it works for you too ✌🏼

I only tested for postgis/postgis:17-3.5 image.

def wait_for_psql_container(timeout: int = 30):
    """A blocking function that runs until a polled, executed command against the container
    to check that the postgresql instance is ready to accept connections succeeds. Otherwise
    raises a :py:class:`TimeoutError` when the set :param:`timeout` elapses without a positive
    result from the executed command.

    :param timeout: The timeout in seconds after which the waiting cancels and fails, defaults to 30
    :raises TimeoutError: When :param:`timeout` is elapsed before the postgresql instance is ready
    :return: `None`
    """

    logger.info("[+] Waiting for PostgreSQL to be ready...")
    commands = [
        [
            "docker",
            "exec",
            PSQL_CONTAINER_NAME,
            "pg_isready",
            "-U",
            "postgres",
        ],
        [
            "docker",
            "exec",
            PSQL_CONTAINER_NAME,
            "psql",
            "-U",
            "postgres",
            "-d",
            "postgres",
            "-c",
            "SELECT PostGIS_version();",
        ],
    ]
    command = commands[0]
    times_ready = 0

    while (timeout := timeout - 1) > -1:
        result = subprocess.run(command, capture_output=True)
        logger.info("%s", result.stdout.decode())
        if result.returncode == 0 and "accepting connection" in result.stdout.decode():
            # Postgres restarts after PostGIS is loaded, and causes ConnectionDoesNotExistError:
            # "connection was closed in the middle of operation"
            # https://github.com/docker-library/postgres/issues/146#issuecomment-215619771
            #
            # ============================================================================
            command = commands[1]
            if (times_ready := times_ready + 1) < 2:
                logger.info("[+] Checking PostGIS is loaded...")
                continue
            time.sleep(1)
            return logger.info("[✓] PostgreSQL is ready.")
        elif result.returncode == 0 and "postgis_version" in result.stdout.decode():
            logger.info("[+] Checking PostgreSQL is accepting connections...")
            command = commands[0]
            time.sleep(1)
            continue
        time.sleep(1)
    raise TimeoutError("PostgreSQL container could not start up in time.")

[mboyea]

Tested this works, you can thank me later if it works for you too ✌🏼

I only tested for postgis/postgis:17-3.5 image.

def wait_for_psql_container(timeout: int = 30):
"""A blocking function that runs until a polled, executed command against the container
to check that the postgresql instance is ready to accept connections succeeds. Otherwise
raises a :py:class:TimeoutError when the set :param:timeout elapses without a positive
result from the executed command.

:param timeout: The timeout in seconds after which the waiting cancels and fails, defaults to 30
:raises TimeoutError: When :param:`timeout` is elapsed before the postgresql instance is ready
:return: `None`
"""

logger.info("[+] Waiting for PostgreSQL to be ready...")
commands = [
    [
        "docker",
        "exec",
        PSQL_CONTAINER_NAME,
        "pg_isready",
        "-U",
        "postgres",
    ],
    [
        "docker",
        "exec",
        PSQL_CONTAINER_NAME,
        "psql",
        "-U",
        "postgres",
        "-d",
        "postgres",
        "-c",
        "SELECT PostGIS_version();",
    ],
]
command = commands[0]
times_ready = 0

while (timeout := timeout - 1) > -1:
    result = subprocess.run(command, capture_output=True)
    logger.info("%s", result.stdout.decode())
    if result.returncode == 0 and "accepting connection" in result.stdout.decode():
        # Postgres restarts after PostGIS is loaded, and causes ConnectionDoesNotExistError:
        # "connection was closed in the middle of operation"
        # https://github.com/docker-library/postgres/issues/146#issuecomment-215619771
        #
        # ============================================================================
        command = commands[1]
        if (times_ready := times_ready + 1) < 2:
            logger.info("[+] Checking PostGIS is loaded...")
            continue
        time.sleep(1)
        return logger.info("[✓] PostgreSQL is ready.")
    elif result.returncode == 0 and "postgis_version" in result.stdout.decode():
        logger.info("[+] Checking PostgreSQL is accepting connections...")
        command = commands[0]
        time.sleep(1)
        continue
    time.sleep(1)
raise TimeoutError("PostgreSQL container could not start up in time.")

I think this won't work for the case where there is only 1 start. So when database initialization is skipped (let's say you persist your PG_DATA directory), your script will timeout.

[calebpitan]

This won't work for the case where there is only 1 start. So when database initialization is skipped (let's say you persist your PG_DATA directory), your script will timeout.

I believe it would work!

It only checks pg_isready twice, which irrespective of whether it has started once or twice, would return a 0 exit code and "accepting connection". If you execute pg_isready an arbitrary amount of time after the db has started and is ready, it still exits with 0 and logs "accepting connection".

In fact it would work for an already running container and a stopped container just as it would for a newly created container. I've used it in all those listed situations. The only difference I've made so far is move the script into a bash script rather than a python script.