Allow pulling from HTTP registries when using the docker compat API

[SoMuchForSubtlety]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
I'm trying to add podman support to the testcontainers-java library (see testcontainers/testcontainers-java#6158). Many of their tests rely on a local, non-https registry to test image pulling behaviour. Podman does not allow this, and there is no straightforward workaround.

To be 100% compatible with the docker API, podman should allow pulling from HTTP container registries when being called via the docker API.

Steps to reproduce the issue:

1. start the docker registry image
2. push an image to it
3. try to pull the image you just pushed via the docker API

Describe the results you received:

docker://localhost:39617/mcipew:0c0189e2-41c4-49cc-8b34-0b8ff3a4a34e: pinging container registry localhost:39617: Get "https://localhost:39617/v2/": http: server gave HTTP response to HTTPS client

Describe the results you expected:
No error.

Additional information you deem important (e.g. issue happens only occasionally):

Modifying the tests for testcontainers-java is sadly not an option

We will gladly accept a PR that only adds the strategy and GitHub Actions setup to test it, without touching anything else.

Output of podman version:

Client:       Podman Engine
Version:      4.3.0
API Version:  4.3.0
Go Version:   go1.18.7
Built:        Fri Oct 21 10:16:35 2022
OS/Arch:      linux/amd64

Output of podman info:

host:
  arch: amd64
  buildahVersion: 1.28.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.4-3.fc36.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.4, commit: '
  cpuUtilization:
    idlePercent: 92
    systemPercent: 1.71
    userPercent: 6.29
  cpus: 16
  distribution:
    distribution: fedora
    variant: workstation
    version: "36"
  eventLogger: journald
  hostname: honestmistake
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.0.5-200.fc36.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 7807201280
  memTotal: 32834928640
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.6-2.fc36.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.6
      commit: 18cf2efbb8feb2b2f20e316520e0fd0b6c41ef4d
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-0.2.beta.0.fc36.x86_64
    version: |-
      slirp4netns version 1.2.0-beta.0
      commit: 477db14a24ff1a3de3a705e51ca2c4c1fe3dda64
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 1h 3m 42.00s (Approximately 0.04 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  localhost:50001:
    Blocked: false
    Insecure: true
    Location: localhost:50001
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: localhost:50001
    PullFromMirror: ""
store:
  configFile: /home/jakob/.config/containers/storage.conf
  containerStore:
    number: 49
    paused: 0
    running: 1
    stopped: 48
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/jakob/.local/share/containers/storage
  graphRootAllocated: 510405902336
  graphRootUsed: 310382002176
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 319
  runRoot: /run/user/1000/containers
  volumePath: /home/jakob/.local/share/containers/storage/volumes
version:
  APIVersion: 4.3.0
  Built: 1666340195
  BuiltTime: Fri Oct 21 10:16:35 2022
  GitCommit: ""
  GoVersion: go1.18.7
  Os: linux
  OsArch: linux/amd64
  Version: 4.3.0

Package info (e.g. output of rpm -q podman or apt list podman or brew info podman):

podman-4.3.0-2.fc36.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

[mheon]

@vrothberg PTAL

[vrothberg]

@SoMuchForSubtlety is it possible to add localhost:39617 as an insecure registry before running the tests (or the tests adding it)?

A drop-in .conf file in /etc/containers/registries.conf.d would be sufficient; it can removed after/by the tests.

[SoMuchForSubtlety]

That doesn't work, because the registry starts with a random port for each test case. I tried to change the tests to use a fixed port, but there were concerns that that would break the tests in some environments.

Dropping the requirement for a subdomain wildcard when specifying a prefix without a location seems like the kost straightforward solution to me.

[vrothberg]

@SoMuchForSubtlety, the idea is to generate the file when running the tests, when we know which port the registry is running on.

Dropping the requirement for a subdomain wildcard when specifying a prefix without a location seems like the kost straightforward solution to me.

It will take time some more time until we get it done.

[SoMuchForSubtlety]

@SoMuchForSubtlety, the idea is to generate the file when running the tests, when we know which port the registry is running on.

They won't allow any changes to existing tests.

We will gladly accept a PR that only adds the strategy and GitHub Actions setup to test it, without touching anything else.

I'll try to write a small external program that checks for running registries and updates the config accordingly.

Btw, I'd be happy to contribute the required changes for the prefix logic.

[vrothberg]

Thanks, @SoMuchForSubtlety! Let's continue the design discussion in #16491. Once we have consensus, we can distribute the work. Maybe we can find a less time-consuming solution.

[vrothberg]

I took another look at this issue. The following configuration does the trick. Prefix-matching matches at the host boundary such that localhost:1234 will match the registry below.

[[registry]]           
prefix = "localhost"   
location = "localhost" 
insecure = true        

The caveat. This is not documented and code comment suggests to not rely on this behavior. However, this behavior is so old that I'd be afraid to change it.

@mtrmac WDYT?

[mtrmac]

@vrothberg in short, I don’t know.

Ugh, clearly I should have actually implemented that correctly the first time.

In one sense, it has a unit test, so right now, for a user using a frozen version of Podman, I think it works well enough to rely on.

In another sense, fixing this has just become 5 times more urgent… but you’re right that changing this now might be too much of a risk.

[vrothberg]

Do I understand it correctly that a "bug" has turned into a feature? 👼

Before making a recommendation, I want to make sure that this doesn't break in the future.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@mtrmac @vrothberg what should we do with this one?

[vrothberg]

I'd recommend the upper mentioned configuration of registries.conf:

[[registry]]           
prefix = "localhost"   
location = "localhost" 
insecure = true        

While a code comment suggests this may change in the future, I think the ship has sailed already.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

Since we have a workaround can we close this issue? Should this use case be documented in a man page?

[vrothberg]

I think we need to document the workaround in the podman man pages. Probably in the --tls-verify option?

[GrimmiMeloni]

I think we need to document the workaround in the podman man pages. Probably in the --tls-verify option?

Yes please! I just lost an hour putting this all together and landing here.

Probably off-topic here, but the docs at https://podman-desktop.io/docs/getting-started/insecure-registry are not up2date either. The example there is in v1 format, which does not work any longer.

[acritelli]

Hi! I came across this issue while trying to debug an issue with an insecure registry behind a proxy. I noticed that, even with the workaround in place, podman pull will still try sending an HTTPS request to the server:

From my registries.conf:

[[registry]]
prefix = "localhost"
location = "localhost"
insecure = true

We can see that trying to pull will still try an HTTPS request first, which will fail:

# podman --log-level=debug pull localhost:5000/myimage:latest 2>&1 | grep https
time="2023-08-10T23:22:47-04:00" level=debug msg="GET https://localhost:5000/v2/"
time="2023-08-10T23:22:47-04:00" level=debug msg="Ping https://localhost:5000/v2/ err Get \"https://localhost:5000/v2/\": http: server gave HTTP response to HTTPS client (&url.Error{Op:\"Get\", URL:\"https://localhost:5000/v2/\", Err:(*errors.errorString)(0xc000352c40)})"

Is there a way to truly get Podman to only try pulling via HTTP? I ask this for a few reasons:

1. Some reverse proxies may not do a good job of handling an HTTPS request to an HTTP port. For example, in my testing with the Varnish cache, the connection ends up hanging for ~5 seconds.
2. This behavior wastes an entire round trip to the server, only to have the request fail and have to be retried as HTTP.
3. As a user, I would expect a way to only send an HTTP-based request.

I guess it isn't technically unexpected behavior, since the registries.conf docs say that both HTP and untrusted TLS connections will be allowed:

# # If true, unencrypted HTTP as well as TLS connections with untrusted
# # certificates are allowed.
# insecure = false

Given the above, I'm not sure if this belongs on this issue, or should be a separate issue or feature request (e.g., implement a way to force HTTP).

Thanks!

[vrothberg]

Is there a way to truly get Podman to only try pulling via HTTP? I ask this for a few reasons:

There is no way at the moment. Podman will always attempt HTTPS first and if the registry is set "insecure" Podman will fallback to trying HTTP.

Given the above, I'm not sure if this belongs on this issue, or should be a separate issue or feature request (e.g., implement a way to force HTTP).

I agree. It's a somehow related but independent issue. I will break this out into a new one.