Referencing local image name fails on "docker compose build"

[neilyio]

Description

I'm getting an error from docker compose build with a setup that is consistently successful with docker-compose build.

I have a docker-compose.yml file with two services: base and extended. base.image gives a name to the image built by base, and I'd like to use that name as the FROM image in the Dockerfile for the extended service.

This works well with docker-compose build. It does not work with docker compose build.

Steps to reproduce the issue:

1. Re-create my docker/ folder with these three files, plus an empty README.md. cd into docker/.

# docker/docker-compose.yml
services:
  base:
    image: neilyio/base
    build:
      context: .
      dockerfile: base.Dockerfile

  extended:
    build:
      context: .
      dockerfile: extended.Dockerfile

# docker/base.Dockerfile
FROM scratch
COPY README.md /root/README.md

# docker/extended.Dockerfile
FROM neilyio/base
CMD cat /root/README.md

2. docker-compose build, expect a successful run.
3. Clear your cache and delete these new images so we have a clean comparison for the next step. I used these commands:

docker image rm neilyio/base docker_extended
docker system prune -f

4. docker compose build, expect a failure.

Describe the results you received:

docker compose build produces:

[+] Building 0.6s (8/8) FINISHED                                                                                                                                                                            
 => [docker_extended internal] load build definition from extended.Dockerfile                                                                                                                          0.0s
 => => transferring dockerfile: 88B                                                                                                                                                                    0.0s
 => [neilyio/base internal] load build definition from base.Dockerfile                                                                                                                                 0.0s
 => => transferring dockerfile: 86B                                                                                                                                                                    0.0s
 => [docker_extended internal] load .dockerignore                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                                        0.0s
 => [neilyio/base internal] load .dockerignore                                                                                                                                                         0.0s
 => => transferring context: 2B                                                                                                                                                                        0.0s
 => ERROR [docker_extended internal] load metadata for docker.io/neilyio/base:latest                                                                                                                   0.4s
 => [neilyio/base internal] load build context                                                                                                                                                         0.0s
 => => transferring context: 3.65kB                                                                                                                                                                    0.0s
 => [neilyio/base 1/1] COPY README.md /root/README.md                                                                                                                                                  0.0s
 => [neilyio/base] exporting to image                                                                                                                                                                  0.0s
 => => exporting layers                                                                                                                                                                                0.0s
 => => writing image sha256:e29ad2347eed9046148ed435ca66984c9421c1d39ae9f40004a62658e60640c3                                                                                                           0.0s
 => => naming to docker.io/neilyio/base                                                                                                                                                                0.0s
------
 > [docker_extended internal] load metadata for docker.io/neilyio/base:latest:
------
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

Describe the results you expected:

I expected docker compose build to find the locally-built neilyio/base image, instead it seems to try and load from docker.io/neilyio/base:latest. I expected docker compose build to have the same behaviour as docker-compose build, which successfully found the local image.

Additional information you deem important (e.g. issue happens only occasionally):

This can be a little tricky to reproduce because of Docker's caching. docker compose build will work fine if neilyio/base is already built. docker compose build will successfully find the local image, so it can give the impression that it's working. My step 3 above, clearing the cache, is important to accurately reproduce this. I found I needed to do both a system prune and image rm for this.

docker-compose build works every time, whether or not neilyio/base has been built before.

Output of docker version:

Client:
 Cloud integration: 1.0.14
 Version:           20.10.6
 API version:       1.41
 Go version:        go1.16.3
 Git commit:        370c289
 Built:             Fri Apr  9 22:46:57 2021
 OS/Arch:           darwin/arm64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.6
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       8728dd2
  Built:            Fri Apr  9 22:44:13 2021
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.4.4
  GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc:
  Version:          1.0.0-rc93
  GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker context show:

default

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  compose: Docker Compose (Docker Inc., 2.0.0-beta.1)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 20.10.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.25-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 1.942GiB
 Name: docker-desktop
 ID: OP3D:IHZS:FQCX:56ZP:HNOA:X4KO:2EF2:AOY2:URIC:5GF6:LUHX:Z7QD
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details (AWS ECS, Azure ACI, local, etc.):

Local run on M1 Macbook Air.

[neilyio]

I realize this is probably because of the parallelism that is part of the compose-cli project, but this seems to be a case where that parallelism is getting in the way.

[ndeloof]

It seems to me this indeed only works "by chance" as docker-compose build run sequentially, building first image with a tag, then using it as base image for the next service.

I'm not sure if/how we can support such a use-case, would need to check how buildkit can handle this.

[itchyny]

I'm facing this issue too. Is it possible to analyze the build dependency by marking possibly used as base image against the images which specify both image and build? Do we need a new option to specify build dependency in the compose config?

[calebickler]

Having this issue as well. I tried defining depends_on: on the other services but still didn't work. Went back to use docker-compose (2nd time I had to bail on using the new command)

[aleccool213]

Do we need a new option to specify build dependency in the compose config?

IMO this is what depends_on: was being used for previously. Docker Compose V1 was smart enough to see that an image in depends_on: didn't exist and that when defined with local build configuration in docker-compose.yml, to build that image first before being used.

A lot of people depend on this behaviour being the same when moving to V2.

A workaround I see is manually building the images in depends_on: in the docker-compose.yml first before building the rest of the images. This is a large maintenance burden though as developers would need to keep track of these image names in a static configuration/scripts at the very least. For this reason, I'm holding off on V2.

@ndeloof This issue is breaking behaviour in a lot of repos I'm seeing and should be addressed.

[ndeloof]

Same issue applies to compose v1 as long as buildkit is enabled: #8449

[ndeloof]

depends on docker/buildx#447

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ndeloof]

Still on it

[stale]

This issue has been automatically marked as not stale anymore due to the recent activity.

[mbrodala]

Apparently similar to #8805

[curlybeast]

Are there any temporary work arounds for this problem?

[thaJeztah]

Workarounds depend a bit on your exact use-case;

You can run the builds for each service manually to make sure to build the base image first (docker compose build base), but this depends on what "builder" you use; as it won't work if you use a remote or "container" builder (such builders store build-cache, but not images).

The other workaround (this would usually be the recommended approach) is to use a multi-stage build. However, this assumes the situation as outlined in this ticket's description where both images share the same build-context.

Rewriting the example to have both services use the same Dockerfile, but a different target (stage). The second (extended) stage depends on the first (base) stage, which means that building extended will also build the base stage.

# docker/docker-compose.yml
services:
  base:
    image: neilyio/base
    build:
      context: .
      target: base

  extended:
    build:
      context: .
      target: extended

# syntax=docker/dockerfile:1

FROM scratch AS base
COPY README.md /root/README.md

FROM base AS extended
CMD cat /root/README.md

It's worth noting that;

When building only extended, the layer(s) for base will be built, but no image (neilyio/base) is tagged for the base image.

docker compose build extended
[+] Building 2.9s (9/9) FINISHED
 => [internal] load build definition from Dockerfile                                                                              0.2s
 => => transferring dockerfile: 32B                                                                                               0.0s
 => [internal] load .dockerignore                                                                                                 0.3s
 => => transferring context: 2B                                                                                                   0.0s
 => resolve image config for docker.io/docker/dockerfile:1                                                                        2.0s
 => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:443aab4ca21183e069e7d8b2dc68006594f40bddf1b15bbd83f5137bd93e80e2   0.0s
 => [internal] load build definition from Dockerfile                                                                              0.0s
 => [internal] load .dockerignore                                                                                                 0.0s
 => [internal] load build context                                                                                                 0.1s
 => => transferring context: 30B                                                                                                  0.0s
 => CACHED [base 1/1] COPY README.md /root/README.md                                                                              0.0s
 => exporting to image                                                                                                            0.0s
 => => exporting layers                                                                                                           0.0s
 => => writing image sha256:ee813e9db95b2b2d222f4cf72b4507ece85773201a26634f5066ff7d2d4bec65                                      0.0s
 => => naming to docker.io/library/compose-multibuild_extended                                                                    0.0s

docker image ls --filter reference=neilyio/base
REPOSITORY   TAG       IMAGE ID   CREATED   SIZE


docker compose build base
[+] Building 1.4s (9/9) FINISHED
 => [internal] load build definition from Dockerfile                                                                              0.1s
 => => transferring dockerfile: 32B                                                                                               0.0s
 => [internal] load .dockerignore                                                                                                 0.1s
 => => transferring context: 2B                                                                                                   0.0s
 => resolve image config for docker.io/docker/dockerfile:1                                                                        0.8s
 => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:443aab4ca21183e069e7d8b2dc68006594f40bddf1b15bbd83f5137bd93e80e2   0.0s
 => [internal] load build definition from Dockerfile                                                                              0.0s
 => [internal] load .dockerignore                                                                                                 0.0s
 => [internal] load build context                                                                                                 0.1s
 => => transferring context: 30B                                                                                                  0.0s
 => CACHED [base 1/1] COPY README.md /root/README.md                                                                              0.0s
 => exporting to image                                                                                                            0.0s
 => => exporting layers                                                                                                           0.0s
 => => writing image sha256:67e08e818eb01fd3b97ad637b5e99762d34c766e2bb11f2c501a716ef183de62                                      0.0s
 => => naming to docker.io/neilyio/base                                                                                           0.0s

Generally, building extended, then building base should produce the same image (common layers shared between both images), but there may be some improvements to be made in compose here; if both builds would run in parallel, they're not guaranteed to produce the same layer-digests (if changes are made in between within the build-context).

docker inspect --format '{{json .RootFS.Layers}}' neilyio/base 
["sha256:3b1be5298307e60029517bf38caa3f1b58121ef39d56d832b4a62dbabc56c3d3"]

docker inspect --format '{{json .RootFS.Layers}}' compose-multibuild_extended
["sha256:3b1be5298307e60029517bf38caa3f1b58121ef39d56d832b4a62dbabc56c3d3"]

[curlybeast]

@thaJeztah thank you for the detailed response. From what I've been reading online, it was my conclusion too.

My situation is tad more dependency chain problematic, where I'm building a base and using that local image in 2 other local images (e.g. foo and bar) which are using FROM base. I suppose adopting your approach would mean quite some changes to the file system structure (all images are built from root/base, root/foo, root/bar, etc. currently). Also, when I looked at this earlier, it lead me to cache-from and target in docker-compose.yml and I was really hoping to avoid all that.

Any idea on ETA / priority for this issue?

Thanks again.

[ndeloof]

@curlybeast Recent docker compose release will build images with respect to the depends_on directive, so that you can declare the base image to be built first, then the other service images.

[rolies106]

Not sure if this valid or not, but I have same issue and I solve it by using this 2 commands :

export DOCKER_BUILDKIT=0
export COMPOSE_DOCKER_CLI_BUILD=0

After i run that I re-run docker-compose up/build and its solved.

[curlybeast]

@curlybeast Recent docker compose release will build images with respect to the depends_on directive, so that you can declare the base image to be built first, then the other service images.

Thanks for letting me know about this. Which version is this released in?

[kostya]

for me this is fixed when i start use docker compose instead of docker-compose. because docker-compose outdated.

[curlybeast]

for me this is fixed when i start use docker compose instead of docker-compose. because docker-compose outdated.

Just be careful you don't have another version in your PATH. I just realised I had an old docker-compose in /usr/local/bin from back in the days when it wasn't installed as standard with docker packages. This is why I couldn't see any of the new features at first 🙄 (like docker compose pull --ignore-pull-failures)

[ndeloof]

To better cover this scenario, it seems to be we should define a new depends_on condition dedicated to build requirements, i.e.

services:
  base:
    build: .

  extened:
    build:
      context: extended
   depends_on:
      base:
         condition: image_built

I'll experiment with this approach and prepare a proposal on https://github.com/compose-spec/compose-spec

[pebo]

Any updates on this issue? It seems like docker compose v2.18.1 doesn't respect depends_on for the build when build kit is enabled.

[ndeloof]

@pebo can you provide a simple example to reproduce this issue?
Docker Compose run builds in depends_on order to address this need (while dependency might not be required at runtime, so my comment)

[pebo]

@ndeloof It seems like the problem arises when the dependent service is using the image property rather than a build section.

Dockerfile

FROM alpine as base

RUN echo 'foo' > /foo.txt
RUN echo 'sleeping for 10s' && sleep 10


FROM base-image as sub

RUN echo 'bar' > /bar.txt && cat /foo.txt >> /bar.txt

docker-compose.ml

version: "3"

services:
  base:
    image: base-image
    build:
      context: .
      dockerfile: ./Dockerfile
      target: base
    deploy:
      replicas: 0

  sub-with-build:
    build:
      context: .
      dockerfile: ./Dockerfile
      target: sub
    command: echo 'sub-with-build' && cat /bar.txt
    depends_on:
      - base

  sub-with-image:
    image: base-image
    command: echo 'sub-with-image' && cat /bar.txt
    depends_on:
      - base

Error

docker compose up       
[+] Running 2/2
 ✘ sub-with-image Error 

Full Gist

The service sub-with-build waits for base-image to be built and then runs as expected but sub-with-image fails. Is this the expected outcome?

[MarioLiebisch]

@pebo Can confirm this. I had one base image and it was working. Then tried adding a second "stage" (since a few images share part of the features) and suddenly everything fell apart.

[arvindpdmn]

Some folks have noted that depends_on controls the order of the build. I am not so sure. Official docs says that depends_on is used to control the startup and shutdown sequence of the containers. So it's a runtime configuration, not a build configuration. See https://docs.docker.com/compose/compose-file/05-services/#depends_on
Apologies if my understanding is wrong.

[akors]

Some folks have noted that depends_on controls the order of the build. I am not so sure. Official docs says that depends_on is used to control the startup and shutdown sequence of the containers. So it's a runtime configuration, not a build configuration. See https://docs.docker.com/compose/compose-file/05-services/#depends_on Apologies if my understanding is wrong.

Even though undocumented, depend_on used to control the build order as well (haven't tested it recently). And unfortunately it's the only way to control the build order without using third-party tools. For some reason, docker-compose maintainers decided to exclude build ordering from the docker-compose feature set, despite most of the code being clearly present and working.

[typoworx-de]

Be careful when using buildkit features. Buildkit uses dockerized build-instances and therefore this instances have some isolation against host system (and host docker). I did not realize at first, what consequences that can have.

If you have two docker-compose services with services that depend on base-image build by another service this can break if you are using separate Dockerfiles!

moby/buildkit#4162 (comment)

[namedgraph]

export DOCKER_BUILDKIT=0 worked for me on WSL2. Looks like buildkit usage is recent? I would expect it to work as previously though.