Problem setting environment variables through docker-compose

[datracka]

I have problems to create a custom DB in docker-compose using the environment variables and postgres docker image.

having this docker.compose.yml file and running docker-compose up -d

postgres:
  restart: 'true'
  image: postgres
  ports:
    - "5432:5432"
  environment:
    - POSTGRES_DB:test_db 

the container created does not contain the test_db database

But if I run the following command:

docker run -e POSTGRES_DB=test_db --name postgres-test -d -p 5432:5432 postgres

the created container DO have the test_db

I was reading the following links and everything looks good.

https://hub.docker.com/_/postgres/
https://docs.docker.com/engine/reference/run/#/env-environment-variables
https://docs.docker.com/compose/environment-variables/

Sure I am missing something because it should be straightforward. But what?

thank for your time and help!

[mrafayaleem]

I suspect this is happening because you have a previous image of this in your docker where you ran docker-compose up without providing the envs and hence it is not able to overwrite. Check and delete postgres image by doing docker ps -a and docker rm <image> respectively.

[datracka]

Hi,

it did not work :( I removed all the images in my local machine and regardless still not working.

Below the log output when I run $ docker-compose up

postgres_1  | The files belonging to this database system will be owned by user "postgres".
postgres_1  | This user must also own the server process.
postgres_1  | 
postgres_1  | The database cluster will be initialized with locale "en_US.utf8".
postgres_1  | The default database encoding has accordingly been set to "UTF8".
postgres_1  | The default text search configuration will be set to "english".
postgres_1  | 
postgres_1  | Data page checksums are disabled.
postgres_1  | 
postgres_1  | fixing permissions on existing directory /var/lib/postgresql/data ... ok
postgres_1  | creating subdirectories ... ok
postgres_1  | selecting default max_connections ... 100
postgres_1  | selecting default shared_buffers ... 128MB
postgres_1  | selecting dynamic shared memory implementation ... posix
postgres_1  | creating configuration files ... ok
postgres_1  | running bootstrap script ... ok
postgres_1  | performing post-bootstrap initialization ... ok
postgres_1  | syncing data to disk ... ok
postgres_1  | 
postgres_1  | WARNING: enabling "trust" authentication for local connections
postgres_1  | You can change this by editing pg_hba.conf or using the option -A, or
postgres_1  | --auth-local and --auth-host, the next time you run initdb.
postgres_1  | 
postgres_1  | Success. You can now start the database server using:
postgres_1  | 
postgres_1  |     pg_ctl -D /var/lib/postgresql/data -l logfile start
postgres_1  | 
postgres_1  | ****************************************************
postgres_1  | WARNING: No password has been set for the database.
postgres_1  |          This will allow anyone with access to the
postgres_1  |          Postgres port to access your database. In
postgres_1  |          Docker's default configuration, this is
postgres_1  |          effectively any other container on the same
postgres_1  |          system.
postgres_1  | 
postgres_1  |          Use "-e POSTGRES_PASSWORD=password" to set
postgres_1  |          it in "docker run".
postgres_1  | ****************************************************
postgres_1  | waiting for server to start....LOG:  database system was shut down at 2016-10-17 08:45:42 UTC
postgres_1  | LOG:  MultiXact member wraparound protections are now enabled
postgres_1  | LOG:  database system is ready to accept connections
postgres_1  | LOG:  autovacuum launcher started
postgres_1  |  done
postgres_1  | server started
postgres_1  | ALTER ROLE
postgres_1  | 
postgres_1  | 
postgres_1  | /docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
postgres_1  | 
postgres_1  | LOG:  received fast shutdown request
postgres_1  | LOG:  aborting any active transactions
postgres_1  | LOG:  autovacuum launcher shutting down
postgres_1  | LOG:  shutting down
postgres_1  | waiting for server to shut down....LOG:  database system is shut down
postgres_1  |  done
postgres_1  | server stopped
postgres_1  | 
postgres_1  | PostgreSQL init process complete; ready for start up.
postgres_1  | 
postgres_1  | LOG:  database system was shut down at 2016-10-17 08:45:43 UTC
postgres_1  | LOG:  MultiXact member wraparound protections are now enabled
postgres_1  | LOG:  database system is ready to accept connections
postgres_1  | LOG:  autovacuum launcher started

has something to do this line? /docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/* I am pointing in this direction: I assume my docker configuration should be tweak in some way... (I am using Docker for Mac Version 1.12.1 (build: 12133))

at the moment and as workaround I create the DB / user / pass manually.

[mrafayaleem]

I still don't get it but I am using the following docker-compose.yml and it works perfectly fine. I was in a similar problem that you have mentioned and the trick was to just remove the previous docker image and data dir and re-run the container from scratch.

version: '2'
services:
  postgres:
    image: postgres:9.3
    ports:
    - "5432:5432"
    environment:
      - POSTGRES_USER=test
      - POSTGRES_PASSWORD=test
      - POSTGRES_DB=test_db
    volumes:
      - ./data/postgres:/var/lib/postgresql/data

[datracka]

your script works like a charm! Still I do not get exactly what I was doing wrong (maybe it was the version:'2' or nest the services under 'services' tag..), but it does not mind, I move forward. Thanks for your help!

[Chrisissorry]

Had the exact same problem. Removed all images and deleted the data directory as @mrafayaleem suggested - works!

[yosifkit]

Yeah, the problem is not the existing images on your host, but it stems from the volume (or bind mount directory) being already initialized after your first start. The postgres user, and database creation only happens on the first start (ie, /var/lib/postgresql/data must not already contain database files).

Here is an example flow

1. declare volume for my image via compose: ./data/postgres:/var/lib/postgresql/data
2. set env in yaml: POSTGRES_PASSWORD=test
3. start containers docker-compose up -d
4. decide to add database to postgres env POSTGRES_DB=test_db or change the password
5. restart my containers docker-compose up -d
6. database test_db does not exist or password is unchanged 😢
   1. db files must be removed in order for postres entrypoint to re-initialize: docker-compose stop; sudo rm -rf ./data/postgres/
   2. now it can be restarted docker-compose up -d 😃

If you do not declare a volume mount point, then the VOLUME declared in the postgres image will apply and docker will create and manage the directory independent of the life-cycle of the container. This gets more complicated when using compose, since it will keep the volume to re-use later even when you docker-compose rm -f all of your running containers.

An example without a bind mounted volume:

1. say we do steps 2-5 above
2. we still have the same problem of the test_db database not existing (or the password not changing) since the data from the volume still exists 😞
   1. so, get rid of all containers and their volumes docker-compose rm -fv
   2. or docker-compose rm -fv postgres to get rid of just the postres service and its volumes
   3. now we can start up a new postgres container with a new empty volume docker-compose up -d 😃

You can see what volumes you have on your host by doing a docker volume ls (bind mounts will not show up in this list). There is currently no easy way to see what containers are attached to a volume, so you would have to docker inspect the container to see which ones are attached to it.

If you want to clean up all local volumes that are not attached to containers (WARNING this could delete important data 😲): docker volume ls | awk '$1 == "local" { print $2 }' | xargs --no-run-if-empty docker volume rm. On my development machine I usually precede this by removing stopped containers.

Also, in case it is not obvious, do not delete your postres data directory or volume if you have important data stored there. 😱

[erikj]

I do not know for sure if it was the root cause, but the syntax used to specify the test_db environment variable appears to be incorrect and will not set POSTGRES_DB:

  environment:
    - POSTGRES_DB:test_db 

You can either specify it as a hash (no -):

  environment:
    POSTGRES_DB:test_db 

Or as an array, -, w/ = assignment rather than key:value

  environment:
    - POSTGRES_DB=test_db