Moby client behavior rely on Moby API, and is not compatible with podman short name handling #12320

mscherer · 2021-11-16T18:27:05Z

According to https://github.com/moby/moby/blob/master/client/image_pull.go#L22-L29 , "docker.io" suffix is stripped when a program call client.ImagePull (since that's the "familiar name")

For example, trying to client.ImagePull(_, "docker.io/a6543/test_git_plugin", _ ) will result in a API call of

POST /v1.33/images/create?fromImage=a6543%2Ftest_git_plugin&tag=latest

This is not compatible with the short-name feature of podman (https://www.redhat.com/sysadmin/container-image-short-names), since this result in podman returning this error in the API (tested with the HEAD version, see #12315 for the whole story):

Error response from daemon: failed to resolve image name: short-name resolution enforced but cannot prompt without a TTY

Should the compat API mirror moby client expectation, eg, defaulting to docker.io ?

The text was updated successfully, but these errors were encountered:

Since that's the default behavior of moby engine, the compat API need to reflect that. Fixes containers#12320 Signed-off-by: Michael Scherer <misc@redhat.com>

Podman's Docker-compatible REST API is in need of a mechanism to enforce resolving to Docker Hub only. Yet there is the desire for the rest of the stack to continue honoring the system's registries.conf. We recently added a new field to containers.conf [1] which allows for opting out from enforcing Docker Hub for Podman's compat API but we still lack a way of enforcement when resolving short names; which ultimately is *the* place to do that. This change does the necessary plumbing. The compat REST handlers will set the new field in the `types.SystemContext` and pass that down to libimage and buildah. [1] containers/common@e698b8c Context: containers/podman/issues/12320 Signed-off-by: Valentin Rothberg <rothberg@redhat.com>

The Docker-compatible REST API has historically behaved just as the rest of Podman and Buildah (and the atomic Docker in older RHEL/Fedora) where `containers-registries.conf` is centrally controlling which registries a short name may resolve to during pull or local image lookups. Please refer to a blog for more details [1]. Docker, however, is only resolving short names to docker.io which has been reported (see containers#12320) to break certain clients who rely on this behavior. In order to support this scenario, `containers.conf(5)` received a new option to control whether Podman's compat API resolves to docker.io only or behaves as before. Most endpoints allow for directly normalizing parameters that represent an image. If set in containers.conf, Podman will then normalize the references directly to docker.io. The build endpoint is an outlier since images are also referenced in Dockerfiles. The Buildah API, however, supports specifying a custom `types.SystemContext` in which we can set a field that enforces short-name resolution to docker.io in `c/image/pkg/shortnames`. Notice that this a "hybrid" approach of doing the normalization directly in the compat endpoints *and* in `pkg/shortnames` by passing a system context. Doing such a hybrid approach is neccessary since the compat and the libpod endpoints share the same `libimage.Runtime` which makes a global enforcement via the `libimage.Runtime.systemContext` impossible. Having two separate runtimes for the compat and the libpod endpoints seems risky and not generally applicable to all endpoints. [1] https://www.redhat.com/sysadmin/container-image-short-names Fixes: containers#12320 Signed-off-by: Valentin Rothberg <rothberg@redhat.com>

Podman's Docker-compatible REST API is in need of a mechanism to enforce resolving to Docker Hub only. Yet there is the desire for the rest of the stack to continue honoring the system's registries.conf. We recently added a new field to containers.conf [1] which allows for opting out from enforcing Docker Hub for Podman's compat API but we still lack a way of enforcement when resolving short names; which ultimately is *the* place to do that. This change does the necessary plumbing. The compat REST handlers will set the new field in the `types.SystemContext` and pass that down to libimage and buildah. [1] containers/common@e698b8c Context: containers/podman/issues/12320 Signed-off-by: Valentin Rothberg <rothberg@redhat.com>

petersenna · 2022-05-19T10:49:38Z

@vrothberg I am not sure I ran into a corner case, so I do not really know if the behavior I describe is a bug or not. For reasons I pull container images from a private registry at VM image creation time, and then after VMs are deployed I create containers over the Podman's Docker-compatible REST API.

The issue I am having is that starting with podman 4, trying to start a container that was previously pulled fails because the container does not exist at Docker Hub. More precisely I get a 404 for docker.io/library/NAME:TAG

Here is an example. Please note that I replaced real strings with words in capital letters, like NAME, and TAG.

# podman --version
podman version 4.0.2

# podman --log-level=debug system service -t 0 tcp:0.0.0.0:2376
...
DEBU[0017] Looking up image "NAME:TAG" in local containers storage
DEBU[0017] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0017] Trying "NAME:TAG" ...
DEBU[0017] Looking up image "NAME:TAG" in local containers storage
DEBU[0017] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0017] Trying "NAME:TAG" ...
DEBU[0017] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0017] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf"
DEBU[0017] Loading registries configuration "/etc/containers/registries.conf.d/001-rhel-shortnames.conf"
DEBU[0017] Loading registries configuration "/etc/containers/registries.conf.d/002-rhel-shortnames-overrides.conf"
DEBU[0017] Trying "localhost/NAME:TAG" ...
DEBU[0017] Trying "registry.fedoraproject.org/NAME:TAG" ...
DEBU[0017] Trying "registry.access.redhat.com/NAME:TAG" ...
DEBU[0017] Trying "registry.centos.org/NAME:TAG" ...
DEBU[0017] Trying "docker.io/library/NAME:TAG" ...
DEBU[0017] Trying "localhost/NAME:TAG" ...
DEBU[0017] Trying "docker.io/library/NAME:TAG" ...
DEBU[0017] Trying "registry.fedoraproject.org/NAME:TAG" ...
DEBU[0017] Trying "registry.access.redhat.com/NAME:TAG" ...
DEBU[0017] Trying "registry.centos.org/NAME:TAG" ...
DEBU[0017] Trying "docker.io/library/NAME:TAG" ...
DEBU[0017] Trying "docker.io/library/NAME:TAG" ...
DEBU[0017] Trying "REGISTRY/PATH/NAME:TAG" ...
DEBU[0017] Trying "REGISTRY/PATH/NAME:TAG" ...
DEBU[0017] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@5fb70826285b943f31f0c4b0dc0247d29177941c6e01a99b188bdda633244892"
DEBU[0017] Found image "NAME:TAG" as "REGISTRY/PATH/NAME:TAG" in local containers storage
DEBU[0017] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@5fb70826285b943f31f0c4b0dc0247d29177941c6e01a99b188bdda633244892"
DEBU[0017] Found image "NAME:TAG" as "REGISTRY/PATH/NAME:TAG" in local containers storage
DEBU[0017] Found image "NAME:TAG" as "REGISTRY/PATH/NAME:TAG" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@5fb70826285b943f31f0c4b0dc0247d29177941c6e01a99b188bdda633244892)
DEBU[0017] Looking up image "docker.io/library/NAME:TAG" in local containers storage
DEBU[0017] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0017] Trying "docker.io/library/NAME:TAG" ...
DEBU[0017] Trying "docker.io/library/NAME:TAG" ...
DEBU[0017] Found image "NAME:TAG" as "REGISTRY/PATH/NAME:TAG" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@5fb70826285b943f31f0c4b0dc0247d29177941c6e01a99b188bdda633244892)
DEBU[0017] Looking up image "docker.io/library/NAME:TAG" in local containers storage
DEBU[0017] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0017] Trying "docker.io/library/NAME:TAG" ...
DEBU[0017] Trying "docker.io/library/NAME:TAG" ...
DEBU[0017] Trying "docker.io/library/NAME:TAG" ...
INFO[0017] Request Failed(Not Found): No such image: docker.io/library/NAME:TAG: image not known
DEBU[0017] Trying "docker.io/library/NAME:TAG" ...
INFO[0017] Request Failed(Not Found): No such image: docker.io/library/NAME:TAG: image not known

Then if I set compat_api_enforce_docker_hub to false, podman works as I would expect and will be happy with containers found in local containers storage.

Is this a bug?

P.S. On the VMs I have running docker, Docker is perfectly fine with containers found in local containers storage.

vrothberg · 2022-05-19T11:15:01Z

Thanks for the ping, @petersenna. It definitely sounds suspicious when Docker works but Podman does not on the compat API and by definition qualifies as a bug.

Judging from the logs, Podman is able to resolve the image locally when looking for "NAME:TAG":

DEBU[0017] Looking up image "NAME:TAG" in local containers storage
[...]
DEBU[0017] Found image "NAME:TAG" as "REGISTRY/PATH/NAME:TAG [...]

For some reasons, Podman is then looking up the normalized version which fails:

DEBU[0017] Looking up image "docker.io/library/NAME:TAG" in local containers storage
[...]
INFO[0017] Request Failed(Not Found): No such image: docker.io/library/NAME:TAG: image not known

Would you mind opening a new issue for it? It would be extremely useful to see the exact API request to isolate the bug.

petersenna · 2022-05-19T11:46:31Z

Created a new issue: #14291

mscherer mentioned this issue Nov 16, 2021

Support "native" podman as backend woodpecker-ci/woodpecker#85

Open

mscherer changed the title ~~Moby client API is not compatible with podman~~ Moby client behavior rely on Moby API, and is not compatible with podman short name handling Nov 16, 2021

mscherer mentioned this issue Nov 16, 2021

Add docker.io to unqualified image name #12321

Closed

mscherer added a commit to mscherer/podman that referenced this issue Nov 16, 2021

Add docker.io to unqualified image name

5ad0f0d

Since that's the default behavior of moby engine, the compat API need to reflect that. Fixes containers#12320 Signed-off-by: Michael Scherer <misc@redhat.com>

vrothberg mentioned this issue Nov 23, 2021

shortnames: mechanism to enforce resolving to Docker Hub containers/image#1415

Merged

vrothberg mentioned this issue Nov 29, 2021

compat API: allow enforcing short-names resolution to Docker Hub #12435

Merged

openshift-merge-robot closed this as completed in #12435 Nov 30, 2021

Luap99 mentioned this issue Feb 15, 2022

Podman unable to resolve fully-qualified image names from docker.io when using docker-compose #13234

Closed

petersenna mentioned this issue May 19, 2022

Podman's Docker-compatible REST API and previously pulled images. #14291

Closed

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 20, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Moby client behavior rely on Moby API, and is not compatible with podman short name handling #12320

Moby client behavior rely on Moby API, and is not compatible with podman short name handling #12320

mscherer commented Nov 16, 2021

petersenna commented May 19, 2022 •

edited

Loading

vrothberg commented May 19, 2022

petersenna commented May 19, 2022

Moby client behavior rely on Moby API, and is not compatible with podman short name handling #12320

Moby client behavior rely on Moby API, and is not compatible with podman short name handling #12320

Comments

mscherer commented Nov 16, 2021

petersenna commented May 19, 2022 • edited Loading

vrothberg commented May 19, 2022

petersenna commented May 19, 2022

petersenna commented May 19, 2022 •

edited

Loading