docker build fails to authenticate with private repository

[alexhendel]

Description

I have noticed an issue using private repositories with authentication since upgrading to the latest docker version with Docker Engine v23.

Using basic auth (no credential helper) docker pull can pull the image from the private repository (Nexus in my case) after running docker login. But running docker build . fails with the following error:

] Building 0.1s (3/3) FINISHED                                                                                                                                                                                  
 => [internal] load build definition from Dockerfile                                                                                                                                                         0.0s
 => => transferring dockerfile: 100B                                                                                                                                                                         0.0s
 => [internal] load .dockerignore                                                                                                                                                                            0.0s
 => => transferring context: 2B                                                                                                                                                                              0.0s
 => ERROR [internal] load metadata for ***/moembed/runtime:4.0.0-arm64v8                                                                                                                       0.1s
------
 > [internal] load metadata for ***/moembed/runtime:4.0.0-arm64v8:
------
Dockerfile:1
--------------------
   1 | >>> FROM ***/moembed/runtime:4.0.0-arm64v8
   2 |     CMD echo
   3 |     
--------------------
ERROR: failed to solve: ***/moembed/runtime:4.0.0-arm64v8: pulling from host *** failed with status code [manifests 4.0.0-arm64v8]: 401 Unauthorized

Since docker pull will pull the image from the private registry after docker login the following steps will lead to a successful build:

• docker login some.registry.com
• docker pull some.registry.com/image
• docker build .

I found that the behavior is similar to docker/cli#3716 but only started for me after installing docker with Docker Engine v23.

Reproduce

1. docker login some.registry.com
2. Create Dockerfile with relevant content (with FROM clause referring to a private registry image)
3. docker build .

Expected behavior

docker build should use the given credentials after a docker login and successfully pull the referenced image from the given registry.

docker version

Client: Docker Engine - Community
 Version:           23.0.0
 API version:       1.42
 Go version:        go1.19.5
 Git commit:        e92dd87
 Built:             Wed Feb  1 17:46:28 2023
 OS/Arch:           linux/arm64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          23.0.0
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.19.5
  Git commit:       d7573ab
  Built:            Wed Feb  1 17:46:28 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.16
  GitCommit:        31aa4358a36870b21a992d3ad2bef29e1d693bec
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.15.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 3
 Server Version: 23.0.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 31aa4358a36870b21a992d3ad2bef29e1d693bec
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-58-generic
 Operating System: Ubuntu 22.04.1 LTS
 OSType: linux
 Architecture: aarch64
 CPUs: 3
 Total Memory: 5.779GiB
 Name: ubuntu-vm
 ID: e8b9743c-f49f-407c-8d8f-e0c56ad58277
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

No response

[thaJeztah]

@crazy-max @jedevc PTAL; at a glance I don't see [auth] in the output (ISTR BuildKit includes that if it's handling authentication?). Should this be moved to the buildx issue tracker?

[jedevc]

@alexhendel out of curiosity, if you run with docker buildx build instead of just docker build (after making sure the image hasn't been pulled locally) does it still break?

[alexhendel]

Actually shows the same result. buildx build will error out. But running docker pull will pull the image without errors and buildx build directly after that doesn't fail..

~/source/docker-test$ sudo docker buildx build .
[+] Building 0.1s (3/3) FINISHED                                                                                                                                                         
 => [internal] load .dockerignore                                                                                                                                                   0.0s
 => => transferring context: 2B                                                                                                                                                     0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                0.0s
 => => transferring dockerfile: 100B                                                                                                                                                0.0s
 => ERROR [internal] load metadata for ***/moembed/runtime:4.0.0-arm64v8                                                                                              0.1s
------
 > [internal] load metadata for ***/moembed/runtime:4.0.0-arm64v8:
------
Dockerfile:1
--------------------
   1 | >>> FROM ***/moembed/runtime:4.0.0-arm64v8
   2 |     CMD echo
   3 |     
--------------------
ERROR: failed to solve: ***/moembed/runtime:4.0.0-arm64v8: pulling from host *** failed with status code [manifests 4.0.0-arm64v8]: 401 Unauthorized

~/source/docker-test$ sudo docker pull ***/moembed/runtime:4.0.0-arm64v8
4.0.0-arm64v8: Pulling from moembed/runtime
Digest: sha256:2fac5f0e2e94fbd4d03658b7f8aa840a7c6089bfe0ad4ceffd055ea0d649ffe2
Status: Downloaded newer image for ***/moembed/runtime:4.0.0-arm64v8
***/moembed/runtime:4.0.0-arm64v8

~/source/docker-test$ sudo docker buildx build .
[+] Building 0.0s (5/5) FINISHED                                                                                                                                                         
 => [internal] load .dockerignore                                                                                                                                                   0.0s
 => => transferring context: 2B                                                                                                                                                     0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                0.0s
 => => transferring dockerfile: 100B                                                                                                                                                0.0s
 => [internal] load metadata for ***/moembed/runtime:4.0.0-arm64v8                                                                                                    0.0s
 => CACHED [1/1] FROM ***/moembed/runtime:4.0.0-arm64v8                                                                                                               0.0s
 => exporting to image                                                                                                                                                              0.0s
 => => exporting layers                                                                                                                                                             0.0s
 => => writing image sha256:a48d05fdcf8e508d0a8114f132f5ab2e4790e28ed425638a2ac04a0ee212b92

[thaJeztah]

@alexhendel what does your ~/.docker/config.json look like?

⚠️ ⚠️ the file may contain your credentials if you're not using a credentials helper, so be sure to sanitize those parts before posting

[alexhendel]

The config json below (I replaced the actual values with ***):

The registry value is what I would expect and the auth value appears to be hashed string.

{
        "auths": {
                "***": {
                        "auth": "***"
                }
        }
}

[thaJeztah]

Thanks! Yes, that looks fairly standard (was curious if perhaps a credential-helper was in use somewhere).

Do you happen to know what kind of registry is running on your some.registry.com ? (of course, if you can disclose, just looking for information that could help investigate)

[jedevc]

@thaJeztah sounds good to move to buildx repo 🎉

[thaJeztah]

Done!

[tonistiigi]

Are you using a remote server and is it possible the registry does additional authentication based on IP? If yes, does setting BUILDKIT_NO_CLIENT_TOKEN=true make any difference?

"***": {

Is this a clean domain like in the reference you are pulling, or URL with some protocol/path?

[alexhendel]

Are you using a remote server and is it possible the registry does additional authentication based on IP? If yes, does setting BUILDKIT_NO_CLIENT_TOKEN=true make any difference?

We are using Sonatype Nexus to host a private registry. I am not sure about the IP-based authentication.

Where would I configure BUILDKIT_NO_CLIENT_TOKEN=true?

"***": {

Is this a clean domain like in the reference you are pulling, or URL with some protocol/path?

This value is a clean domain. No path or protocol.

[alexhendel]

We did some additional testing which might give some more details. We did try podman and ran:

podman pull
podman build .

This behaves like the previous Docker version, build and pull work fine. Checking the network traffic the only difference we could find was an additional HEAD request with v23 (which was not sent previously and is also not sent by podman).

[posti-vpk]

We have experienced the exactly same issue with docker 23.0.1 against private images in GitHubs ghcr.io-registry. Adding BUILDKIT_NO_CLIENT_TOKEN=true fixed the issue for us. Also downgrading to 20.x.x solved the issue.

[tgquan67]

I can confirm that BUILDKIT_NO_CLIENT_TOKEN=1 works for us (we are using docker 23, and the private registry is artifactory with client certificate required for authentication).

[krische]

I can confirm that BUILDKIT_NO_CLIENT_TOKEN=1 works for us (we are using docker 23, and the private registry is artifactory with client certificate required for authentication).

@tgquan67 What version of Artifactory are you using? I can't seem to get it to work with Artifactory 7.49.3.

[tonistiigi]

BUILDKIT_NO_CLIENT_TOKEN=1 suggests that the difference is in some network layer and only the daemon can connect to the auth server. The code itself should be identical.

[krische]

So BUILDKIT_NO_CLIENT_TOKEN=1 is supposed to be an environment variable for the daemon process? Not the client?

[tonistiigi]

No, on the client. BUILDKIT_NO_CLIENT_TOKEN=1 docker buildx build ...

[krische]

No, on the client. BUILDKIT_NO_CLIENT_TOKEN=1 docker buildx build ...

Yup, that's what I was trying but still no luck. What's odd for me is it does seem like my layers are pushed to my Artifactory successfully, but when it tries to push the manifest then Artifactory responds with a 403. And since I'm on a nearly 6 month old version of Artifactory, I'm not 100% certain the issue with is with buildkit/buidx.

I'll have to update my Artifactory and try again.

[thaJeztah]

@tonistiigi related to these?

• Incompatible OAuth behavior with Docker's registry implementation containerd/containerd#4982
• Deal with 403 for POST in OAuth2 for private image pulls using Harbor containerd/containerd#3913
• Deal with 403 for POST method in OAuth2 authentication for private im… containerd/containerd#3912
• Tolerate 403 response for registry OAuth fallback containerd/containerd#4983

Last one has further details (it was rejected at the time); containerd/containerd#4983 (comment)

[tgquan67]

What version of Artifactory are you using? I can't seem to get it to work with Artifactory 7.49.3.

I'm on artifactory 7.41 and it worked. Btw, in my case I only test if it can pull the base image from artifactory, I did not try to push it with buildx.
Maybe disable buildkit with DOCKER_BUILDKIT=0 and see if it works first, and then check how you authenticate with artifactory.

[tonistiigi]

No, these seem to be about HTTP request methods,

[krische]

I updated my Artifactory to their newest version, but still no luck. However, I did try docker login with the admin account of my Artifactory and that worked. So I don't think my issue is with buildx but something with my Artifactory.

[krische]

Yup, it was just a permission issue on my Artifactory that caused the 403 when buildkit was trying to HEAD to see if the manifest was already there before the PUT to upload it. I fixed the permission issue and it worked. It even works for me without setting BUILDKIT_NO_CLIENT_TOKEN=1.

[tgquan67]

Now I test it again on docker 24, it works even without BUILDKIT_NO_CLIENT_TOKEN=1 on the default builder; but if i create a new builder (for example with docker buildx create --name test --driver=docker-container), I have no way to make it work when this new builder is in use (with docker buildx use test).

[typoworx-de]

Same issue here!

[fredericoschardong]

The issue persists on docker 24.0.7 and buildx v0.11.2.

[ryan-ph]

I'm still seeing this on docker 25.0.3 and buildx v0.12.1

$ docker --version
Docker version 25.0.3, build 4debf41

$ docker buildx version
github.com/docker/buildx v0.12.1-desktop.4 6996841df2f61988c2794d84d33205368f96c317

[marco-bertelli]

same issue here

[whitfin]

@alexhendel just curious if you ever resolved this on your side? We're using Nexus and facing the same thing. Works with legacy builder, just not BuildKit.

[tgquan67]

Another update: at some point after docker 25 update, it started working in my case (artifactory 7.41) with BUILDKIT_NO_CLIENT_TOKEN=1. Also since I have client certificate requirement for my registry, I also have this file at /etc/buildkitd.toml:

debug = true
[registry."artifactory.myregistry.net"]
  ca=["/opt/myregistry/certs/ca.crt"]
  [[registry."artifactory.myregistry.net".keypair]]
    key="/opt/myregistry/certs/client.key"
    cert="/opt/myregistry/certs/client.pem"

Create a new builder with

docker buildx create --use --bootstrap --name basic-builder --driver docker-container --config /etc/buildkitd.toml

And I also export these 3 variables:

export BUILDKIT_NO_CLIENT_TOKEN=1
export DOCKER_BUILDKIT=1
export BUILDX_BUILDER=basic-builder

After this in dockerfiles I can pull from artifactory.myregistry.net using credentials I logged in with docker login artifactory.myregistry.net.

[alexhendel]

@alexhendel just curious if you ever resolved this on your side? We're using Nexus and facing the same thing. Works with legacy builder, just not BuildKit.

No not directly.

There was an nginx proxy routing our traffic. We did change the nginx config to allow HEAD requests. To us it seems that at some point these HEAD requests where implemented and did not go through with our setup. It did work for our case since.

[marco-bertelli]

In my case guys was a GitHub package permission issue, for fix that go into your package detail ---> package settings ---> and check the package permissions, not sure that this is releated to your problem but maybe can help, after that on my side no more auth errors, the strange thing was the error because was not simila to 401 or 403.

[zerowebcorp]

Same issue with latest docker.

# docker --version
Docker version 27.3.1, build ce12230
# 

[thaJeztah]

@zerowebcorp are you running the docker CLI inside a container? Credentials are stored on the client-side (and/or obtained through credential-helpers on the client-side); if you're running from inside a container, make sure you're authenticated inside the container, and that authentication works there (e.g. in a situation where you bind-mount the home-directory or ~/.docker/ directory into a container, the CLI configuration file may contain configuration to use a credential-helper that's present on the host, but not available inside the container.

[zerowebcorp]

No. Outside the container. Docker registry hosted remote behind cloudflare. It started working finally when I disabled cloudflare waf.

[100tomer]

@zerowebcorp are you running the docker CLI inside a container? Credentials are stored on the client-side (and/or obtained through credential-helpers on the client-side); if you're running from inside a container, make sure you're authenticated inside the container, and that authentication works there (e.g. in a situation where you bind-mount the home-directory or ~/.docker/ directory into a container, the CLI configuration file may contain configuration to use a credential-helper that's present on the host, but not available inside the container.

Hi, when I run this locally, it works:

docker login...
docker buildx build...

However, when I try the same commands inside a Docker container during CI, it fails due to authentication issues. I believe the authentication is fine since a normal Docker push to the same repository works without any problems.