docker-compose does not set environment variables #7423

Kytech · 2020-05-04T03:52:06Z

Description of the issue

When using docker-compose up or docker-compose run -e ENVIRONMENT_VAR=value, environment variables are not being applied either through the -e command line parameters or from the docker-compose.yml file.

The docker-compose.yml file I am using is as follows:

version: '3'
services:
    nginx-reverse-proxy:
        image: nginx
        volumes:
         - C:/Users/<Username>/nginx-reverse-proxy/configdir:/etc/nginx/conf.d/
        ports:
         - "80:80"
        environment:
            NGINX_SERVER_NAME: "test.mysite.org"
            NGINX_REMOTE_URL: "http://remote.mysiteengine.somehost.com:13345"

Context information (for bug reports)

Output of docker-compose version

docker-compose version 1.25.5, build 8a1c60f6

Output of docker version

Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b
 Built:             Wed Mar 11 01:23:10 2020
 OS/Arch:           windows/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       afacb8b
  Built:            Wed Mar 11 01:29:16 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker-compose config

WARNING: The NGINX_SERVER_NAME variable is not set. Defaulting to a blank string.
WARNING: The NGINX_REMOTE_URL variable is not set. Defaulting to a blank string.
services:
  nginx-reverse-proxy:
    environment:
      NGINX_SERVER_NAME: "test.mysite.org"
      NGINX_REMOTE_URL: "http://remote.mysiteengine.somehost.com:13345"
    image: nginx
    ports:
    - 80:80/tcp
    volumes:
    - C:\Users\<Username>\nginx-reverse-proxy\configdir:/etc/nginx/conf.d/:rw
version: '3.0'

Steps to reproduce the issue

Create a docker-compose.yml file that sets environment variables
Run docker-compose up

Observed result

Environment Variables are not applied

Expected result

Environment variables should be applied to containers started by docker-compose

Stacktrace / full error message

WARNING: The NGINX_SERVER_NAME variable is not set. Defaulting to a blank string.
WARNING: The NGINX_REMOTE_URL variable is not set. Defaulting to a blank string.
Creating network "nginx-reverse-proxy_default" with the default driver

Additional information

OS Version: Windows 10 Pro v1909,
docker-compose install method: Docker Desktop for Windows Installer, followed by manual upgrade of docker-compose (Issue occurred on docker-compose version 1.25.4 which was included with Docker Desktop and with manual upgrade to 1.25.5)

The text was updated successfully, but these errors were encountered:

hholst80 · 2020-05-11T17:57:48Z

???
This yaml is broken. Please post the actual yaml you are deploying with.

        volumes:
         - C:/Users/<Username>/nginx-reverse-proxy/configdir:/etc/nginx/conf.d/

Kytech · 2020-05-11T20:08:51Z

???
This yaml is broken. Please post the actual yaml you are deploying with.
        volumes:
         - C:/Users/<Username>/nginx-reverse-proxy/configdir:/etc/nginx/conf.d/

This is the yaml I am using. It works exactly as expected on my local machine when I don't have environment variables included. I am using docker for Windows, so the file path format is different than with linux. I have not included my PC username for privacy purposes.

kravchenkoloznia · 2020-05-28T08:52:52Z

Facing the same issue.

And I found that environment variables are working only from .env file placed in the folder where the docker-compose command is executed (current working directory).

Providing environment variables from command line giving the same warnings as @Kytech mentioned.

$ docker-compose run -e PRODUCT=1 all
WARNING: The PRODUCT variable is not set. Defaulting to a blank string.

Providing environment variables from user-specific docker-compose.env file still resulting with the same warnings

$ cat docker-compose.env 
PRODUCT=1
RELEASE=0
FLASH=0
DEBUG=3

$ cat docker-compose.yml 
version: '3'
services:
  all:
    image: 82100_src:latest
    env_file: ./docker-compose.env
    volumes:
      - .:/home/user
    command: make PRODUCT=${PRODUCT} all RELEASE=${RELEASE} FLASH=${FLASH} DEBUG=${DEBUG}

$ docker-compose run all
WARNING: The PRODUCT variable is not set. Defaulting to a blank string.
WARNING: The RELEASE variable is not set. Defaulting to a blank string.
WARNING: The FLASH variable is not set. Defaulting to a blank string.
WARNING: The DEBUG variable is not set. Defaulting to a blank string.

I'm using Linux Debian 10 with next versions of Docker and Docker-compose:

$ docker-compose version
docker-compose version 1.23.2, build 1110ad01
docker-py version: 3.6.0
CPython version: 3.6.7
OpenSSL version: OpenSSL 1.1.0f  25 May 2017

$ docker version
Client: Docker Engine - Community
 Version:           19.03.9
 API version:       1.40
 Go version:        go1.13.10
 Git commit:        9d988398e7
 Built:             Fri May 15 00:25:25 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.9
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       9d988398e7
  Built:            Fri May 15 00:23:57 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

hholst80 · 2020-05-28T10:46:08Z

.env file and env_file have different semantics. If you want to use variable interpolation in your compose file you have to use the .env file.

hholst80 · 2020-05-28T10:50:25Z

My conclusion here is that the -e environment variables are treated with the same semantics as env_file. Consistent with docker cli run cmd.

kravchenkoloznia · 2020-05-28T11:06:34Z

@hholst80 indeed, my mistake.

After reading docker documentation one more time and few additional articles, it's now obvious for me that for substitution in docker-compose file itself used environment variables of my current shell, and to send environment variables inside docker container I may use either docker run CLI or env_file.

Thanks!

Kytech · 2020-05-28T18:06:11Z

@kravchenkoloznia that should be a good workaround in the meantime.

I hope the docker team fixes this since it's broken just about every docker-compose file I've been using that specifies environment variables in the docker-compose.yml

azryelryvel · 2020-06-04T20:23:37Z

Nothing is working for me, I can't use any environment variable :

/tmp/test$ curl -SsL "https://github.com/docker/compose/releases/download/1.26.0/docker-compose-$(uname -s)-$(uname -m)" -o docker-compose
/tmp/test$ ls -a
.  ..  docker-compose  docker-compose.yml
/tmp/test$ cat docker-compose.yml
version: '3'
services:
  test:
    image: 'busybox'
    command: env
    environment:
      - HELLO=WORLD
/tmp/test$ sudo ./docker-compose down
Removing network test_default
WARNING: Network test_default not found.
/tmp/test$ sudo ./docker-compose up -d
Creating network "test_default" with the default driver
Creating test_test_1 ... done
/tmp/test$ sudo docker logs test_test_1
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=b68e605e609d
HOME=/root

It doesn't work with env_file, .env or docker run -e either

EDIT: The problem disappeared after updating docker to 18.09.8

Mahima-ai · 2020-07-31T10:46:46Z

Hey, While working on docker with Celery and Rabbitmq integration, I also faced the issue where the environment variables were not used up (set) though they can be seen using the env command in the container. I solved the issue by taking the below two steps.

I uninstalled the older docker-compose and installed the latest by following https://docs.docker.com/compose/install/
I renamed the service for which the environment variables were defined in the docker-compose.yml.

Later on, I rechecked with the old service name and found that the issue reappeared. Then I removed the older container and again did docker-compose up. This solved the issue. So I think, it might be the old container which does not let us see the new changes.

I am using Docker Engine v19.03.12 and docker-compose v 1.26.2

The contents (related to the issue) of my docker-compose.yml are:

version: "3.8"

services:  
  rabbit:
    hostname: rabbit
    image: rabbitmq:management
    environment:
      - RABBITMQ_DEFAULT_USER=user1
      - RABBITMQ_DEFAULT_PASS=password1
    ports:
      - "5672:5672"
      - "15672:15672"

agovindaraju · 2020-10-01T11:12:36Z

I am on the latest version of docker compose 1.27.4. I still see the same issue as other users have described above.

michaelpburt · 2020-10-08T15:26:54Z

I am also experiencing the same problem on the latest version of Docker (2.4.0.0).

kerolos-sss · 2020-10-11T00:57:25Z

I have encountered the same issue,
My issue was caused by running docker-compose in a different python virtual environment.

I hope this could help.

It seems that by running on a different python virtual env, I ruined the docker-compose dependency pythondotenv

zachsa · 2020-10-20T13:04:38Z

I ran into this problem trying to use docker-compose to recreate a Mongo container.

MONGO_USER=me MONGO_PSWD=somepswd docker-compose up -d --force-recreate --build

This works, providing volumes associated with the resultant container are recreated (delete the volumes that the container will use and recreate)

MatteoGioioso · 2021-02-08T02:09:23Z

I have the same exact bug with the latest version of Docker 20.10 and Docker-compose 1.27 on Mac OS

namighajiyev · 2021-03-30T17:01:31Z

In my case only newly added one environment variable is not set in running container others get set normally.

     environment:
           Api__ConnectionString: ${Api_Docker__ConnectionString}
           Api__Recaptcha__Secret: ${Api_Docker__Recaptcha__Secret}
           ASPNETCORE_URLS: 'https://+;http://+'
           ASPNETCORE_HTTPS_PORT: 10081
           ASPNETCORE_Kestrel__Certificates__Default__Password: ${Api__Kestrel__Certificates__Default__Password}
           ASPNETCORE_Kestrel__Certificates__Default__Path: /https/Api/api.pfx

I run printenv command inside running Linux container and verified that the Api__Recaptcha__Secret variable doesn't get set.

JonnyWaffles · 2021-10-02T01:44:34Z

This same incredibly confusing issue is occurring to me. Some of my environmental variables set in my yaml file are respected during the runtime execution, and others are missing and my runtime script validation functions terminate the container. Awkwardly, I can see all of the variables properly set in the PyCharm Environmental Variables Docker inspection. Like @namighajiyev reports above, it appears my more recent environmental variables are not being respected during runtime.

I tried docker-compose build --no-cache but that doesn't resolve the issue either.

JonnyWaffles · 2021-10-25T20:58:59Z

So while trying to debug this strange issue, I came up with this awkward workaround. I deleted the docker compose deployment using the docker dashboard so I can reuse the same compose definition and not have any container name conflicts.

I then copied the project to another directory. Executing docker-compose up now works! So I try the aforementioned workflow again, cp the working project folder to a new home, and now docker-compose fails like previous. I am completely stumped. It seems so random.

joma74 · 2021-11-17T21:37:52Z

Had some luck by starting docker-compose in non daemon mode after a change to environment setting(via environment section in docker-compose.yml). Then stop that and start in daemon mode again, environment settings are kept.

franklinchou · 2022-01-01T04:39:19Z

Any update on this? Seems that docker-compose is still not picking up .env files?

ndeloof · 2022-01-03T07:05:42Z

Please give Compose v2 a try and let me know if this issue persists with the new codebase

ubiuser · 2022-01-13T15:17:40Z

docker compose up --build <your-container> fixed the issue for me.

Lhamide00 · 2022-01-14T08:22:17Z

Any update on this?

I have the exact same bug with Docker version 20.10.11 and Docker-compose 1.29.2 on Mac OS.

Removing and re-building the containers did not help. Doesn't matter whether I docker-compose up with or without -d flag either; bug remains!

qalinn · 2022-02-10T16:21:58Z

Hello!
I have the same issue. Any update on this?

ubiuser · 2022-02-17T10:33:59Z

@Lhamide00 @qalinn As earlier said by @ndeloof, please try using v2, so call docker compose instead of docker-compose.

qalinn · 2022-03-01T09:37:41Z

Hello @ubiuser, I tried how you suggested but it didn't work. I don't have any clue what should I try.
Do you have another suggestion?

JonnyWaffles · 2022-03-14T16:32:42Z

Hi friends, we once again encountered this mysterious error even when using docker compose instead of docker-compose. Random environmental variables vanish.

palunel · 2022-04-19T17:31:46Z

Same here.

I have the following docker-compose.yml:

version: "3.9"

services:
  compute:
    build: .
    ports:
      - "27030:27030"
    environment:
      - COMPUTE_HOST=$COMPUTE_HOST
      - COMPUTE_PORT=$COMPUTE_PORT

In my .env file I have the following:

COMPUTE_HOST="0.0.0.0"
COMPUTE_PORT=27030

This is to start up a flask api, but what I get when I run the container with:

docker compose --env-file .env up --build

or

docker-compose --env-file .env up --build

is this:

apv_compute-compute-1  | APV server starting on port None ...
apv_compute-compute-1  |  * Serving Flask app 'api' (lazy loading)
apv_compute-compute-1  |  * Environment: production
apv_compute-compute-1  |    WARNING: This is a development server. Do not use it in a production deployment.
apv_compute-compute-1  |    Use a production WSGI server instead.
apv_compute-compute-1  |  * Debug mode: on
apv_compute-compute-1  |  * Running on http://127.0.0.1:5000 (Press CTRL+C to quit)
apv_compute-compute-1  |  * Restarting with stat
apv_compute-compute-1  |  * Debugger is active!
apv_compute-compute-1  |  * Debugger PIN: 483-995-326

The api insists on starting on 127.0.0.1:5000 suggesting that the environment variables are not set at all.

It also does not work if I change my compose file as follows:

    environment:
      - COMPUTE_HOST="0.0.0.0"
      - COMPUTE_PORT=27036

When running the app outside of docker it works perfectly fine. Does anyone have a sensible workaround?

Docker version 20.10.14, build a224086

ndeloof · 2022-04-19T17:50:22Z

@palunel can you check docker-compose exec <service> env ?

palunel · 2022-04-19T17:54:15Z

@ndeloof Interesting:

PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=42a08ea9f254
COMPUTE_HOST="0.0.0.0"
COMPUTE_PORT=27036
GPG_KEY=E3FF2839C048B25C084DEBE9B26995E310250568
PYTHON_VERSION=3.9.12
PYTHON_PIP_VERSION=22.0.4
PYTHON_SETUPTOOLS_VERSION=58.1.0
PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/38e54e5de07c66e875c11a1ebbdb938854625dd8/public/get-pip.py
PYTHON_GET_PIP_SHA256=e235c437e5c7d7524fbce3880ca39b917a73dc565e0c813465b7a7a329bb279a
HOME=/home/app

ndeloof · 2022-04-19T17:56:32Z

ok, so this is an application issue then :P

Kytech · 2022-06-02T03:05:07Z

Just to add my two cents as the original author if this issue, I've been able to successfully set env variables in docker compose for a little while now. Not sure what fixed it, but it seems newer versions resolved the issues I was having.

Bakanych · 2022-06-02T07:38:46Z

Having similar issue right now
Docker version 20.10.14, build a224086

Azbesciak · 2022-07-22T13:49:26Z

Same as me, casual docker-compose, after update on windows to 4.8.1 (docker for windows) env variables are not passed to the container, even if in inspect mode these are visible...
but printenv does not print anything like this.

michael-gates-techngs · 2022-08-10T20:40:32Z

Experiencing this issue on Docker 20.10.17 build 100c701 on an M1 Mac.

unRARed · 2022-08-10T21:31:05Z

Experiencing this issue on Docker 20.10.17 build 100c701 on an M1 Mac.

same build, same issue on an Intel Mac this afternoon.

andremarcondesteixeira · 2022-09-23T03:53:14Z

I have the same problem, using docker compose v2
extremely frustrated! :(

glours · 2022-09-23T08:10:24Z

@andremarcondesteixeira feel free to open a new issue with an example that allows us to reproduce the problem (compose file, dockerfile, try to use official images...)

gopar · 2023-01-31T01:59:56Z

Any updates on this? Running into this today as well.
Tried re-installing and did not work.

Extremely unfortunate

gopar$docker version

Client:
 Cloud integration: v1.0.29
 Version:           20.10.22
 API version:       1.41
 Go version:        go1.18.9
 Git commit:        3a2c30b
 Built:             Thu Dec 15 22:28:41 2022
 OS/Arch:           darwin/arm64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.16.2 (95914)
 Engine:
  Version:          20.10.22
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.9
  Git commit:       42c8b31
  Built:            Thu Dec 15 22:25:43 2022
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.14
  GitCommit:        9ba4b250366a5ddde94bb7c9d1def331423aa323
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

gopar$ docker-compose version
Docker Compose version v2.15.1

glours · 2023-01-31T07:14:02Z

Hello @gopar
Can you open a new issue with a reproducible example?

HansBambel · 2023-04-03T06:53:14Z

I had a similar issue, but substituting RUN with CMD for the last command made the environment variables discoverable. This Stackoverflow post helped me with this.

phutaneVinayak · 2024-01-22T14:54:16Z

Any update on this ?

ndeloof · 2024-02-06T18:40:27Z

@phutaneVinayak :

Can you open a new issue with a reproducible example?

ihass · 2024-02-07T09:23:42Z

Please also note environment variables concatenation in docker-compose is not working. Expl below, KC_HOSTPORT and KC_DUMMY can never be reused in the following environment variables (KC_COLON_HOSTPORT, KC_DUMMY2, KC_DUMMY3, ...). I tried a lot of combination. (My tech environment : Windows 11, Spring Boot, Gradle/Kotlin, Docker Desktop, IntelliJ)

version: "3.8"

x-common-variables:
  &common-variables
  KC_HOSTNAME: localhost
  KC_HOSTPORT: 8080

services:
  my-backend:
    environment:
      <<: *common-variables
      KC_PROTOCOL: http
      KC_DUMMY: dummy
      KC_DUMMY2: "$KC_DUMMY" #KO
      KC_DUMMY3: "${KC_DUMMY}" #KO
      KC_DUMMY4: "$${KC_DUMMY}" #KO
      KC_COLON_HOSTPORT: ":${KC_HOSTPORT}" #KO
       ...

ndeloof · 2024-02-07T09:32:12Z

@ihass you expect compose to replace ${XX} with other variables set by environment. Variable interpolation does not relies on this, but only uses variables set in your local environment (os.Env) and local .env file.

phutaneVinayak · 2024-02-07T09:35:35Z

Sure I will give it a try. vinayak Phutane

…

On Wed, 7 Feb 2024 at 12:10 AM, Nicolas De loof ***@***.***> wrote: @phutaneVinayak <https://github.com/phutaneVinayak> : Can you open a new issue with a reproducible example? — Reply to this email directly, view it on GitHub <#7423 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGE5QTMWMR2VPMV4JZCXHYDYSJ2KTAVCNFSM4MYNU2XKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJTGA2TIMZUGYYQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

asdf8601 · 2024-09-19T21:08:10Z

This worked for me:

services:
  test:
    image: alpine
    # command: echo MYVAR=$MYVAR  # Not working
    # command: echo MYVAR=$$MYVAR  # Not working
    command: sh -c "echo MYVAR=$$MYVAR"
    environment:
      MYVAR: "asfd"

justinabrahms · 2024-09-23T17:44:00Z

Just noting for others, docker-compose restart $mycontainer isn't adequate to have it pick up envvars. docker-compose up --force-recreate --build $mycontainer does appear to be adequate.

Kytech added the kind/bug label May 4, 2020

Mahima-ai mentioned this issue Jul 31, 2020

Environment variable expansion: environment variable in default fails #7256

Closed

georglauterbach mentioned this issue Sep 7, 2020

Complains environment variables are not set docker-mailserver/docker-mailserver#1598

Closed

ndeloof added the Docker Compose V1 label Sep 16, 2021

ndeloof closed this as completed Apr 19, 2022

zaneselvans mentioned this issue May 3, 2022

Local Docker ETL with local inputs/outputs catalyst-cooperative/pudl#1606

Closed

4 tasks

docker-compose does not set environment variables #7423

docker-compose does not set environment variables #7423

Comments

Kytech commented May 4, 2020

Description of the issue

Context information (for bug reports)

Steps to reproduce the issue

Observed result

Expected result

Stacktrace / full error message

Additional information

hholst80 commented May 11, 2020

Kytech commented May 11, 2020

kravchenkoloznia commented May 28, 2020

hholst80 commented May 28, 2020

hholst80 commented May 28, 2020

kravchenkoloznia commented May 28, 2020

Kytech commented May 28, 2020

azryelryvel commented Jun 4, 2020 • edited Loading

Mahima-ai commented Jul 31, 2020 • edited Loading

agovindaraju commented Oct 1, 2020 • edited Loading

michaelpburt commented Oct 8, 2020

kerolos-sss commented Oct 11, 2020

zachsa commented Oct 20, 2020

MatteoGioioso commented Feb 8, 2021

namighajiyev commented Mar 30, 2021

JonnyWaffles commented Oct 2, 2021

JonnyWaffles commented Oct 25, 2021

joma74 commented Nov 17, 2021

franklinchou commented Jan 1, 2022

ndeloof commented Jan 3, 2022

ubiuser commented Jan 13, 2022

Lhamide00 commented Jan 14, 2022

qalinn commented Feb 10, 2022

ubiuser commented Feb 17, 2022

qalinn commented Mar 1, 2022

JonnyWaffles commented Mar 14, 2022

palunel commented Apr 19, 2022 • edited Loading

ndeloof commented Apr 19, 2022

palunel commented Apr 19, 2022 • edited Loading

ndeloof commented Apr 19, 2022

Kytech commented Jun 2, 2022

Bakanych commented Jun 2, 2022

Azbesciak commented Jul 22, 2022 • edited Loading

michael-gates-techngs commented Aug 10, 2022

unRARed commented Aug 10, 2022

andremarcondesteixeira commented Sep 23, 2022

glours commented Sep 23, 2022

gopar commented Jan 31, 2023 • edited Loading

glours commented Jan 31, 2023

HansBambel commented Apr 3, 2023

phutaneVinayak commented Jan 22, 2024

ndeloof commented Feb 6, 2024

ihass commented Feb 7, 2024 • edited Loading

ndeloof commented Feb 7, 2024

phutaneVinayak commented Feb 7, 2024 via email

asdf8601 commented Sep 19, 2024

justinabrahms commented Sep 23, 2024

azryelryvel commented Jun 4, 2020 •

edited

Loading

Mahima-ai commented Jul 31, 2020 •

edited

Loading

agovindaraju commented Oct 1, 2020 •

edited

Loading

palunel commented Apr 19, 2022 •

edited

Loading

palunel commented Apr 19, 2022 •

edited

Loading

Azbesciak commented Jul 22, 2022 •

edited

Loading

gopar commented Jan 31, 2023 •

edited

Loading

ihass commented Feb 7, 2024 •

edited

Loading