buildx can't push to artifactory or other repos that don't have /oauth/token as their token endpoint

[scr-oath]

When pushing final image using docker buildx build to an enterprise artifactory docker registry, auth seems to fail

I suspect that the request is looking directly for /oauth/token endpoint, regardless of the WWW-Authenticate response.

$ curl -v https://${DOCKER_REGISTRY}/v2/scr/tst-multi-platform/tags/list |& grep WWW
< WWW-Authenticate: Bearer realm="https://${DOCKER_REGISTRY}/artifactory/api/docker/registry/v2/token",service="${DOCKER_REGISTRY}:4443",scope="repository:scr/tst-multi-platform:pull"

• See also Error retrieving OAuth token from repositories that do not support it  IBM/portieris#51

[tonistiigi]

We should use WWW-Authenticate properly and no hardcoded endpoint addresses https://github.com/moby/buildkit/blob/a6a114a1a476c99c2501aa34811159b849df4005/vendor/github.com/containerd/containerd/remotes/docker/auth/fetch.go#L118 . Maybe someone from Artifactory can look at it. The auth code mostly comes from containerd repository.

[scr-oath]

Hmm… well that's strange… I'll try to do some more debugging when I get a chance and wireshark if possible to see what's being sent returned… unless you have some recommendations for enabling/collecting debugging logs

[100tomer]

any news about this?

[MartinLoeper]

Wow, does that mean, we cannot use buildkit with harbor and similar registries at the moment? I am trying to get things working but buildx always complains about POST /service/token not being available on the harbor registry. Bummer :(

[100tomer]

Wow, does that mean, we cannot use buildkit with harbor and similar registries at the moment? I am trying to get things working but buildx always complains about POST /service/token not being available on the harbor registry. Bummer :(

For my private repository, Buildx fails to utilize the token obtained from the docker login command, leading the repository to perceive every request as an unauthenticated request that leads to 401 unauthorized responses.

[tonistiigi]

buildx always complains about POST /service/token not being available on the harbor registry.

/service/token is not any hardcoded endpoint in buildx/buildkit. If your registry returns 401 with WWW-Authenticate that requests a token from /service/token but at the same time does not implement it, then there is not much a client can do.

[100tomer]

buildx always complains about POST /service/token not being available on the harbor registry.

/service/token is not any hardcoded endpoint in buildx/buildkit. If your registry returns 401 with WWW-Authenticate that requests a token from /service/token but at the same time does not implement it, then there is not much a client can do.

Can you assist me in resolving this issue? In my specific case, the server responds with the “WWW-Authenticate” header, along with a URL path that the buildx should utilize. However, despite this, the token is not being transmitted to the Docker push HTTP requests.

[MartinLoeper]

@tonistiigi Thanks for the insights! I checked the current goharbor implementation and noticed that they implemented the token endpoint via GET method. However, the WWW-Authenticate Header does not carry the HTTP method AFAIK. The current buildx implementation responds to the WWW-Authenticate response header by doing a POST request. Is there some wiggle room in the spec? At least that would explain why it is not working despite goharbor claiming to have implemented the spec.

My current workaround is to set BUILDKIT_NO_CLIENT_TOKEN=true during docker buildx build [...] --push as mentioned here: #1613 (comment)

see also:

• https://github.com/goharbor/harbor/blob/main/src/core/service/token/token.go#L35
• https://github.com/goharbor/harbor/blob/2ed5c1eb978d0de0cd8431d0e6682402f381ea73/src/server/middleware/v2auth/auth.go#L112

might also be the cause of confusion for e.g.:

• Incompatible OAuth behavior with Docker's registry implementation containerd/containerd#4982 (comment)