Docker-compose V2 for aarch64 doesn't find local images

[deleugpn]

Description

When running docker-compose build using version 2 on an arm64 machine, the Dockerfile being built does not find local images.

Steps to reproduce the issue:

1. Create A.Dockerfile

FROM alpine:3.14

RUN echo "base image" > /docker-compose-v2-is-broken.txt

2. Create B.Dockerfile

FROM my-org/my-base-image

RUN cat /docker-compose-is-broken.txt

3. Create docker-compose.yml

version: "3.8"

services:
  service-a:
    image: my-org/my-base-image
    build:
      context: .
      dockerfile: ./A.Dockerfile

  service-b:
    image: my-org/my-specific-image
    build:
      context: .
      dockerfile: ./B.Dockerfile

4. run docker-compose build service-a && docker-compose build service-b.

Describe the results you received:

[root@ip-10-40-0-60 docker-compose-bug]# docker-compose build service-b
[+] Building 0.8s (3/3) FINISHED
 => [internal] load build definition from B.Dockerfile                                                                                                   0.0s
 => => transferring dockerfile: 104B                                                                                                                     0.0s
 => [internal] load .dockerignore                                                                                                                        0.0s
 => => transferring context: 2B                                                                                                                          0.0s
 => ERROR [internal] load metadata for docker.io/my-org/my-base-image:latest                                                                             0.7s
------
 > [internal] load metadata for docker.io/my-org/my-base-image:latest:
------
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

Describe the results you expected:

my-org/my-base-image should be loaded from the local repository of images built. This work fine on a x86 computer.

Additional information you deem important (e.g. issue happens only occasionally):

Issue only happens on aarch64 binary of docker-compose

Output of docker compose version:

Docker Compose version v2.0.1

Output of docker info:

Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 26
  Running: 1
  Paused: 0
  Stopped: 25
 Images: 136
 Server Version: 20.10.7
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc version: %runc_commit
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.14.246-187.474.amzn2.aarch64
 Operating System: Amazon Linux 2
 OSType: linux
 Architecture: aarch64
 CPUs: 2
 Total Memory: 7.754GiB
 Name: ip-10-40-0-60.eu-west-1.compute.internal
 ID: M3FZ:TIBF:74OL:XW2V:HNU5:T5AQ:7Y2E:LHMB:4XNV:AKCA:ANZJ:D4WP
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details:

EC2 instance t4g on AWS.

[georgeswani]

Hi, I am facing similar issue using buildkit with the docker version higher than 5:19.03.153-0ubuntu-focal. This is perfectly working with docker version 5:19.

The image arm64v8/<image_name>:latest exist in local docker images.

[internal] load metadata for docker.io/arm64v8/<image_name>:latest:

---

failed to solve with frontend dockerfile.v0: failed to solve with frontend gateway.v0: rpc error: code = Unknown desc = pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

[d3x0r]

Oh - it would be the specific architecture I picked? I just tried to redo what I was doing for x86 for arm64, and it fails to use the local file - I thought it might have been a cross-compile issue. The arm base image does run, and I can do uname -a; but trying to then use that base image in something else fails. (or it finds the similarly named other package)

failed to solve with frontend dockerfile.v0: failed to create LLB definition: docker.io/d3x0r/alpine:latest: not found 

[ndeloof]

Do you have the same issue running DOCKER_BUILDKIT=1 docker build -f A.Dockerfile . && docker build -f B.Dockerfile . ?
(in such case, this is a buildkit issue)

Compose v2.1.x allows to use the "classic" docker builder, by disabling buildkit (DOCKER_BUILDKIT=0), could you please give it a try?

[deleugpn]

I will give it a try again and report back later today, but my untrustworthy memory says I did manage to run the scenario with docker build normally.

[d3x0r]

I wasn't using builder specifically.

This is project I was working on; it's just some bat files and docker scripts.
https://github.com/d3x0r/d0x0r

mk-arm was the variant to build against an arm version.
base is just the alping linux core
build is the built on that to compile some code
extra builds some extra code
run uses base again + products from build/extra to make a final runtime image...

I could do mk-arm.bat in base but build wouldn't successfully use that; it would run, if the options were wrong and it used the wrong base; but basically just 'docker build -t d3x0r/node-alpine-build-arm64 -f Dockerfile.arm .' would fail.

[georgeswani]

Hi, I tested this with DOCKER_BUILDKIT=0 and it seems to be working, so this confirm the issue is with buildkit for arm64 builds. Is there any workaround suggestion ?
Thanks

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jasonmccallister]

Still having some issues with this as well... Also, should the focus of changing/effort be on the docker-compose command or the new docker compose that ships with the new docker CLI?

[stale]

This issue has been automatically marked as not stale anymore due to the recent activity.

[toabi]

This is still an issue, it makes developing base images and testing them with docker-compose projects a bit annoying. But for now running docker compose with DOCKER_BUILDKIT=0 is a working workaround.

[nickcoad]

This is a major issue, genuinely surprised it's still ongoing more than a year later - is there some fix for this (other than DOCKER_BUILDKIT=0 which is so slow as to be unworkable) that I'm not aware of?

[ndeloof]

@nickcoad Do you have the same issue running DOCKER_BUILDKIT=1 docker build -f A.Dockerfile . && docker build -f B.Dockerfile . ?

I understand the frustration with this issue, but without more inputs, it's hard for me to diagnose...

[glours]

Hey @deleugpn @georgeswani @d3x0r @jasonmccallister @toabi @nickcoad
This issue was opened a long time ago, I tried to reproduce it and I wasn't able.
As we made a bunch of improvements since, I'm wondering if we haven't fixed this problem in the meantime.
Can you check on your side?

[mckaygerhard]

i was trying yo made a docker i386 over and amd64 with

docker build -t name/image:name-i386 . -f - << EOF FROM alpine ... etc etc EOF

later docker run just said no manifiest and no image found.. i am working all locally i foun this issue by google ..

i used DOCKER_DEFAULT_PLATFORM env

[jhrotko]

I can't also reproduce this anymore. I am closing this issue. If you encounter again this issue please open a new one.