env_file variable values overwritten by .env values since v2.6.0

[b3n3d1k7]

Description

Environment variable values specified via env_file in a docker_compose.yml are overwritten by the values in a .env file since v2.6.0.

Steps to reproduce the issue:

1. Create the following files

# .env
TEST=.env

# .env.test
TEST=.env.test

# docker-compose.yml
version: '3'
services:
  test:
    image: 'node:6-alpine'
    env_file:
      - .env.test

3. Run docker-compose config with docker-compose v2.6.0

Describe the results you received:

name: docker-compose-test
services:
  test:
    environment:
      TEST: .env
    image: node:6-alpine
    networks:
      default: null
networks:
  default:
    name: docker-compose-test_default

Describe the results you expected:
This is the result of running the docker-compose config command with e.g. docker-compose v2.5.1:

name: docker-compose-test
services:
  test:
    environment:
      TEST: .test.env
    image: node:6-alpine
    networks:
      default: null
networks:
  default:
    name: docker-compose-test_default

Additional information you deem important (e.g. issue happens only occasionally):
If the environment variables are defined directly in the docker-compose.yml instead of via an env_file, they are not overwritten by the values in the .env file.

Output of docker compose version:

Docker Compose version v2.6.0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.8.2)
  compose: Docker Compose (Docker Inc., v2.6.0)

Server:
 Containers: 3
  Running: 2
  Paused: 0
  Stopped: 1
 Images: 375
 Server Version: 20.10.14
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc version: v1.0.3-0-gf46b6ba
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.11.0-49-generic
 Operating System: Ubuntu 21.04
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.44GiB
 Name: benedikt-ThinkPad-E15
 ID: 7NBC:LL2S:XCZE:FWIG:EFK7:NDKX:FUNR:OXTV:D7RE:EXUP:PRBV:SZCE
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Default Address Pools:
   Base: 192.168.0.0/16, Size: 24

Additional environment details:
We noticed this problem in our gitlab pipelines using the docker:git image.
For now we just fixed the image version to docker:20.10.15-git to circumvent the problem.

[bkonetzny]

Pretty sure this is caused by the changes from #9512

Also for Docker Desktop 4.9.0 is also broken as it contains docker-compose 2.6.0, downgrading to Docker Desktop 4.8.2 fixes the issue, as this bundles docker-compose 2.5.1

[pieterza]

This also affected us, and after quite a painful investigation implemented a workaround.

What's particularly concerning is how it was decided that defaults/external env would have PRECEDENCE over the configurable env-file option.

It should be reversed (as is the previous behavior)

I'm hopeful of this issue being acknowledged and fixed soon

For what it's worth I logged a similar (in logic, if not exact cause as this) issue over a year ago:

#8108

Despite all the feedback, it was ultimately auto-expired without any fix.

[ulyssessouza]

Hello.

This is a clear regression on 2.6.0. I'm currently working on it to make a patch release ASAP.
We will also work on the documentation to make the precedence chain more explicit for everyone.

[Matts154]

Worked around this bug by using the --env-file flag with docker compose.

e.g. docker compose --env-file .env.test up -d

[nickdelja]

I am having a similar issue. I have seen the precedence chain to be:

1. My local environment variables
2. .env (automatically loaded in 2.6)
3. env files specified through docker compose.

This is a bummer as we would want to overwrite local env vars and potentially .env vars as well.

@ulyssessouza would your fix apply to the env_file: setting the docker-compose.yaml file as well?

[machisuke]

For Docker Desktop users, you can use Docker Compose V1 which is not buggy by unchecking "Use Dcoker Compose V2".

not checked

$ docker-compose version
docker-compose version 1.29.2, build 5becea4c

checked

$ docker-compose version
Docker Compose version v2.6.0

[nickdelja]

For Docker Desktop users, you can use Docker Compose V1 which is not buggy by unchecking "Use Dcoker Compose V2".

not checked

$ docker-compose version
docker-compose version 1.29.2, build 5becea4c

checked

$ docker-compose version
Docker Compose version v2.6.0

NOTE: If you alias the compose call to something shorter like I did, make sure you are calling docker-compose and not docker compose. Unchecking Use Docker Compose V2 has no impact on the docker compose command.

[sdt]

Any updates on this one? It's about to fall off the first page of issues and I fear the next notification I get will be from the stalebot.

[ToshY]

@ulyssessouza As this doesn't seemed to be patched in 2.6.1, could you please give us a status update on your progress?

[digaoddc]

The same issue was described in this issue: #9608

[curry684]

This is a clear regression on 2.6.0. I'm currently working on it to make a patch release ASAP.

@ulyssessouza why is this taking 5 weeks to do "ASAP"? It's actively breaking production environments worldwide for goodness' sake...

[ulyssessouza]

Hello @curry684.

Sorry for the delay, but we had to rework and define the whole environment variable precedence.
A PR and its new documentation page are currently in review phase.
The fix is meant to be in the next release.

[malinoff]

I've experienced this issue as well, and it was painful to find what was causing AWS credentials defined in my shell to leak into containers using local, fake keys from a .env file.
I want to emphasize how bad this can be. I was leaking my real AWS credentials into containers.

Not me, but many people deploy their software on their single-server setups with compose. Some of these people use remote docker api instead of running compose via ssh. Some of these people would have some sort of credentials used in a .env file as well as defined in their shell. Please, do not make a security hole out of nowhere. Do not leak user credentials like this. Make it an explicit CLI option, to inherit shell variables, or just require users to explicitly pass variables with -e.