TLS CA cert in certs.d/...:443 ignored

[3nprob]

Issue Description

Attempting to pull image from registry with certificate from private CA fails with

tls: failed to verify certificate: x509: certificate signed by unknown authority

Steps to reproduce the issue

Steps to reproduce the issue

1. Expose registry:2 via caddy:2, giving caddy a self-signed wildcard cert
2. Put the cert of the root CA of the wildcard cert in PEM format at ${HOME}/.config/containers/certs.d/oci.example.com:443/ca.crt
   • Also tried putting the wildcard cert itself (which Caddy uses as intermediary CA to generate leaf cert) - same result
3. Run podman pull oci.example.com/alpine
   • Fails with x509: certificate signed by unknown authority
4. Run podman pull --cert-dir ~/.config/containers/certs.d oci.example.com/alpine

Describe the results you received

All attempts fail with:

Error: internal error: unable to copy from source docker://oci.example.com/alpine:latest: initializing source docker://oci.example.com/alpine:latest: pinging container registry oci.example.com: Get "https://oci.example.com/v2/": tls: failed to verify certificate: x509: certificate signed by unknown authority

Describe the results you expected

Should pull image successfully.

podman info output

host:
  ociRuntime:
    name: crun
    package: crun-1.21-1.fc41.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.21
      commit: 10269840aa07fb7e6b7e1acff6198692d8ff5c88
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20250606.g754c6d7-1.fc41.x86_64
    version: |
      pasta 0^20250606.g754c6d7-1.fc41.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  oci.example.com:
    Blocked: false
    Insecure: false
    Location: oci.example.com
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: oci.example.com
    PullFromMirror: ""
version:
  APIVersion: 5.5.1
  BuildOrigin: Fedora Project
  Built: 1749081600
  BuiltTime: Wed Jun  4 20:00:00 2025
  GitCommit: 850db76dd78a0641eddb9ee19ee6f60d2c59bcfa
  GoVersion: go1.23.9
  Os: linux
  OsArch: linux/amd64
  Version: 5.5.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

• Fedora Linux 41

Additional information

The CA cert file works fine to verify with curl:

$ curl -L https://oci.example.com/ --cacert /home/user/.config/containers/certs.d/oci.example.com\:443/ca.crt

as well as openssl cli:

$ echo "" |openssl s_client -showcerts -servername oci.example.com -connect oci.example.com:443 -verifyCAfile /home/user/.config/containers/certs.d/oci.example.com\:443/ca.crt

Connecting to 127.0.0.1
CONNECTED(00000003)
depth=1 C=US, ST=State, L=City, O=MyCA, OU=CA, CN=MyCA
verify return:1
depth=0 C=US, ST=State, L=City, O=MyCA, OU=CA, CN=*.example.com
verify return:1
---
Certificate chain
 0 s:C=US, ST=State, L=City, O=MyCA, OU=CA, CN=*.example.com
   i:C=US, ST=State, L=City, O=MyCA, OU=CA, CN=MyCA
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256
   v:NotBefore: Jul  3 08:00:00 2025 GMT; NotAfter: Jul  3 08:00:00 2026 GMT
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
---
Server certificate
subject=C=US, ST=State, L=City, O=MyCA, OU=CA, CN=*.example.com
issuer=C=US, ST=State, L=City, O=MyCA, OU=CA, CN=MyCA
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 954 bytes and written 389 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 256 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_128_GCM_SHA256

The CA is also recognized by podman when added to the system trust root. This does work:

$ sudo cp ~/.config/containers/certs.d/oci.example.com:443/ca.crt  /etc/pki/ca-trust/source/anchors/
$ sudo update-ca-trust extract
$ podman pull oci.example.com/alpine

However, we only want to trust the CA for podman for this registry and not system-wide.

[mheon]

@mtrmac Any ideas?

[mtrmac]

2. Put the cert of the root CA of the wildcard cert in PEM format at ${HOME}/.config/containers/certs.d/oci.example.com:443/ca.crt
3. Run podman pull oci.example.com/alpine

As a first guess, oci.example.com does not match oci.example.com:443.

Please run the Podman comments with --log-level=debug. That might be sufficient to diagnose; if not, please provide the outputs, as well as listings of the relevant directories.

[mtrmac]

(I have also filed containers/common#2465 to drop the incorrect “internal error” part.)

[3nprob]

So after an upgrade (podman 5.5.1 -> 5.5.2 among others), --cert-dir is now working as expected. I'm not sure what made the difference (the dir and registry are untouched and I am rerunning the same commands from shell history) but in any case not reproducing anymore.

Putting the cert in ~/.config/containers/certs.d/oci.example.com instead of ~/.config.d/containers/certs.d/oci.example.com:443 makes it work now. Is this as expected? In that case I guess this could be distilled to a docs issue as man 5 containers-certs.d says: The name of the directory must correspond to the host:port of the registry (e.g., my-registry.com:5000)..

[mtrmac]

In that case I guess this could be distilled to a docs issue as man 5 containers-certs.d

That’s a fair point, we should improve that.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[mtrmac]

containers/container-libs#369 will improve the documentation.