-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cannot pull from custom registry: x509: certificate signed by unknown authority #13823
Comments
Hi @shoeffner , Is this a self signed certificate ? Could you please elaborate about what do you mean by |
If this is local: Please compare the output of But this is remote, isn’t it? If so, are the certificates present on the remote Linux computer? Compare #11507 . |
I'm running into the same problem with
|
@flouthoc the CA certificate is, as far as I know, self-signed; however, the server certificate is not, it is signed with the self-signed CA certificate. I already tried @mtrmac I am running a VM and, as listed above, the certificates are present on it: $ podman machine ssh
> ls /etc/containers/certs.d/
example-local.crt Here is the output as you requested: $ podman login --log-level=debug registry.example.local
INFO[0000] podman filtering at log level debug
DEBU[0000] Called login.PersistentPreRunE(podman login --log-level=debug registry.example.local)
DEBU[0000] SSH Ident Key "/Users/shoeffner/.ssh/podman-machine-default" SHA256:fcXJgc4<snip>Hkd0P2WDdw ssh-ed25519
DEBU[0000] Found SSH_AUTH_SOCK "/private/tmp/com.apple.launchd.OynLOxn0RP/Listeners", ssh-agent signer(s) enabled
DEBU[0000] DoRequest Method: GET URI: http://d/v4.0.3/libpod/_ping
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] No credentials matching registry.fme.lan found in /Users/shoeffner/.config/containers/auth.json
DEBU[0000] No credentials matching registry.fme.lan found in /Users/shoeffner/.config/containers/auth.json
DEBU[0000] No credentials matching registry.fme.lan found in /Users/shoeffner/.docker/config.json
DEBU[0000] No credentials matching registry.fme.lan found in /Users/shoeffner/.dockercfg
DEBU[0000] No credentials for registry.example.local found
Username: shoeffner
Password:
DEBU[0015] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.example.local
DEBU[0015] GET https://registry.example.local/v2/
DEBU[0016] Ping https://registry.example.local/v2/ status 401
DEBU[0016] GET https://registry.example.local/service/token?account=shoeffner&service=harbor-registry
DEBU[0016] GET https://registry.example.local/v2/
DEBU[0016] Stored credentials for registry.example.local in credential helper containers-auth.json
Login Succeeded!
DEBU[0016] Called login.PersistentPostRunE(podman login --log-level=debug registry.example.local) and $ podman pull --log-level=debug registry.example.local/team/image:label
INFO[0000] podman filtering at log level debug
DEBU[0000] Called pull.PersistentPreRunE(podman pull --log-level=debug registry.example.local/team/image:label)
DEBU[0000] SSH Ident Key "/Users/shoeffner/.ssh/podman-machine-default" SHA256:fcXJgc4<snip>Hd26cPaQ7Hkd0P2WDdw ssh-ed25519
DEBU[0000] Found SSH_AUTH_SOCK "/private/tmp/com.apple.launchd.OynLOxn0RP/Listeners", ssh-agent signer(s) enabled
DEBU[0000] DoRequest Method: GET URI: http://d/v4.0.3/libpod/_ping
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Found credentials for registry.example.local in credential helper containers-auth.json in file /Users/shoeffner/.config/containers/auth.json
DEBU[0000] DoRequest Method: POST URI: http://d/v4.0.3/libpod/images/pull
Trying to pull registry.example.local/team/image:label...
Error: initializing source docker://registry.example.local/team/image:label: pinging container registry registry.fme.lan: Get "https://registry.example.local/v2/": x509: certificate signed by unknown authority With $ podman pull --log-level=debug --tls-verify=false registry.example.local/team/image:label
INFO[0000] podman filtering at log level debug
DEBU[0000] Called pull.PersistentPreRunE(podman pull --log-level=debug registry.example.local/team/image:label)
DEBU[0000] SSH Ident Key "/Users/shoeffner/.ssh/podman-machine-default" SHA256:fcXJgc4<snip>Hd26cPaQ7Hkd0P2WDdw ssh-ed25519
DEBU[0000] Found SSH_AUTH_SOCK "/private/tmp/com.apple.launchd.OynLOxn0RP/Listeners", ssh-agent signer(s) enabled
DEBU[0000] DoRequest Method: GET URI: http://d/v4.0.3/libpod/_ping
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Found credentials for registry.example.local in credential helper containers-auth.json in file /Users/shoeffner/.config/containers/auth.json
DEBU[0000] DoRequest Method: POST URI: http://d/v4.0.3/libpod/images/pull
Trying to pull registry.example.local/team/image:label...Getting image source signatures
Copying blob sha256:9d2c24245c2d89f59843f45aa674a4d0<snip>
Copying blob sha256:7b1a6ab2e44dbac178598dabe7cff59bd<snip>
<snip>
Copying config sha256:ecd89a2dd2e5cfa1e80d96129cfefb27<snip>
Writing manifest to image destination
Storing signatures
ecd89a2dd2e5cfa1e80d96129cfefb27<snip>
DEBU[0026] Called pull.PersistentPostRunE(podman pull --log-level=debug --tls-verify=false registry.example.local/team/image:label) |
(works)
So that’s not the right location for a certificate to be found (and indeed, the remote logs don’t contain a “Looking for TLS certificates” log entry. Compare also https://github.com/containers/image/blob/main/docs/containers-certs.d.5.md#directory-structure . (I haven’t looked into how that incorrect destination was created, so I don’t know whether that might be user error or a bug in |
Alright, I see, it has to be in Now the login actually finds the certificate: $ podman login registry.example.local
...
DEBU[0070] Looking for TLS certificates and private keys in /Users/shoeffner/.config/containers/certs.d/registry.example.local
DEBU[0070] crt: /Users/shoeffner/.config/containers/certs.d/registry.example.local/example-local.crt
DEBU[0070] GET https://registry.example.local/v2/
... And the pull works. Thank you! I wonder why the login worked and the pull didn't though... Shouldn't the login reject the non-verifiable cert? |
Also, I just realized that the |
Since this looks like it is working properly, I am going to close. |
I'm still experiencing this problem :( I stored the CA certificate according to the docs (https://github.com/containers/image/blob/main/docs/containers-certs.d.5.md#directory-structure), and PS C:\Users\michael.prankl> podman login registry.mycompany.com --log-level=debug
time="2022-12-02T08:58:19+01:00" level=info msg="C:\\Program Files\\RedHat\\Podman\\podman.exe filtering at log level debug"
time="2022-12-02T08:58:19+01:00" level=debug msg="Called login.PersistentPreRunE(C:\\Program Files\\RedHat\\Podman\\podman.exe login registry.mycompany.com --log-level=debug)"
time="2022-12-02T08:58:19+01:00" level=debug msg="SSH Ident Key \"C:\\\\Users\\\\michael.prankl\\\\.ssh\\\\podman-machine-default\" SHA256:hnwVwQwuZlStVqh3ikEF/LcyrsUdFbt+BwAzeBuqDuQ ssh-ed25519"
time="2022-12-02T08:58:19+01:00" level=debug msg="DoRequest Method: GET URI: http://d/v4.3.1/libpod/_ping"
time="2022-12-02T08:58:19+01:00" level=debug msg="Loading registries configuration \"C:\\\\Users\\\\michael.prankl\\\\.config\\\\containers\\\\registries.conf\""
time="2022-12-02T08:58:19+01:00" level=debug msg="No credentials matching registry.mycompany.com found in C:\\Users\\michael.prankl\\.config\\containers\\auth.json"
time="2022-12-02T08:58:19+01:00" level=debug msg="No credentials matching registry.mycompany.com found in C:\\Users\\michael.prankl\\.config\\containers\\auth.json"
time="2022-12-02T08:58:19+01:00" level=debug msg="No credentials matching registry.mycompany.com found in C:\\Users\\michael.prankl\\.docker\\config.json"
time="2022-12-02T08:58:19+01:00" level=debug msg="No credentials matching registry.mycompany.com found in C:\\Users\\michael.prankl\\.dockercfg"
time="2022-12-02T08:58:19+01:00" level=debug msg="No credentials for registry.mycompany.com found"
Username: michael.prankl
Password:
time="2022-12-02T08:58:26+01:00" level=debug msg="Looking for TLS certificates and private keys in C:\\Users\\michael.prankl\\.config\\containers\\certs.d\\registry.mycompany.com"
time="2022-12-02T08:58:26+01:00" level=debug msg=" crt: C:\\Users\\michael.prankl\\.config\\containers\\certs.d\\registry.mycompany.com\\mycompany.crt.crt"
time="2022-12-02T08:58:26+01:00" level=debug msg="GET https://registry.mycompany.com/v2/"
time="2022-12-02T08:58:26+01:00" level=debug msg="Ping https://registry.mycompany.com/v2/ status 200"
time="2022-12-02T08:58:26+01:00" level=debug msg="GET https://registry.mycompany.com/v2/"
time="2022-12-02T08:58:27+01:00" level=debug msg="Stored credentials for registry.mycompany.com in credential helper containers-auth.json"
Login Succeeded!
time="2022-12-02T08:58:27+01:00" level=debug msg="Called login.PersistentPostRunE(C:\\Program Files\\RedHat\\Podman\\podman.exe login registry.mycompany.com --log-level=debug)" However, PS C:\Users\michael.prankl> podman pull registry.mycompany.com/metabase/metabase:latest --log-level=trace
time="2022-12-02T08:58:46+01:00" level=info msg="C:\\Program Files\\RedHat\\Podman\\podman.exe filtering at log level trace"
time="2022-12-02T08:58:46+01:00" level=debug msg="Called pull.PersistentPreRunE(C:\\Program Files\\RedHat\\Podman\\podman.exe pull registry.mycompany.com/metabase/metabase:latest --log-level=trace)"
time="2022-12-02T08:58:46+01:00" level=debug msg="SSH Ident Key \"C:\\\\Users\\\\michael.prankl\\\\.ssh\\\\podman-machine-default\" SHA256:hnwVwQwuZlStVqh3ikEF/LcyrsUdFbt+BwAzeBuqDuQ ssh-ed25519"
time="2022-12-02T08:58:46+01:00" level=debug msg="DoRequest Method: GET URI: http://d/v4.3.1/libpod/_ping"
time="2022-12-02T08:58:46+01:00" level=debug msg="Loading registries configuration \"C:\\\\Users\\\\michael.prankl\\\\.config\\\\containers\\\\registries.conf\""
time="2022-12-02T08:58:46+01:00" level=debug msg="Found credentials for registry.mycompany.com in credential helper containers-auth.json in file C:\\Users\\michael.prankl\\.config\\containers\\auth.json"
time="2022-12-02T08:58:46+01:00" level=debug msg="No credentials matching capregistry.muenchen.de found in C:\\Users\\michael.prankl\\.config\\containers\\auth.json"
time="2022-12-02T08:58:46+01:00" level=debug msg="No credentials matching capregistry.muenchen.de found in C:\\Users\\michael.prankl\\.config\\containers\\auth.json"
time="2022-12-02T08:58:46+01:00" level=debug msg="Found an empty credential entry \"capregistry.muenchen.de\" in \"C:\\\\Users\\\\michael.prankl\\\\.docker\\\\config.json\" (an unhandled credential helper marker?), moving on"
time="2022-12-02T08:58:46+01:00" level=debug msg="No credentials matching capregistry.muenchen.de found in C:\\Users\\michael.prankl\\.dockercfg"
time="2022-12-02T08:58:46+01:00" level=debug msg="No credentials for capregistry.muenchen.de found"
time="2022-12-02T08:58:46+01:00" level=debug msg="DoRequest Method: POST URI: http://d/v4.3.1/libpod/images/pull"
Trying to pull registry.mycompany.com/metabase/metabase:latest...
Error: initializing source docker://registry.mycompany.com/metabase/metabase:latest: pinging container registry registry.mycompany.com: Get "https://registry.mycompany.com/v2/": x509: certificate signed by unknown authority podman version:
my
|
@eidottermihi did you configure the certs in the Linux VM? |
@vrothberg not sure what you mean by Linux VM, I am using Podman Desktop 0.10 + podman 4.3.1 on Windows. I stored the cert in
|
Podman Desktop is creating a Linux VM where the containers (and Podman) are actually running. Hence, the certs in your Windows Home directory won't be read. @benoitf, is there a best-practice in Podman Desktop to set the certs? |
I understand, but that is rather weird than:
|
no, for now, Podman Desktop reads the Operating System certificates for its own operations but is not propagating these to the podman machine. We were planning some documentation first. @eidottermihi you would need to add these certificates into the 'podman machine' that you can reach using the I think it's because |
also if could be parameter of the |
I followed the instructions in https://github.com/containers/podman/blob/main/docs/tutorials/podman-install-certificate-authority.md to install the CA certs inside the podman machine, now it is working as expected :) |
That is the key. The custom CA must be inside the machine in a directory by the same name as the regsitry, i.e docker.io |
/kind bug
Description
I can
podman login
into our internal harbor registry (say, registry.example.local), but I cannot pull images.The registry has a certificate signed by our custom CA.
Steps to reproduce the issue:
Alternatively, first ssh into the machine and then issue
podman pull
.Describe the results you received:
Describe the results you expected:
Something like
but without
--tls-verify=false
.Additional information you deem important (e.g. issue happens only occasionally):
I'm running macOS 12.3.1 on an M1.
The certificates should be correctly installed on
machine init
as per #12709, I verified this withOutput of
podman version
:Output of
podman info --debug
:Package info (e.g. output of
rpm -q podman
orapt list podman
):$ brew info podman
Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)
Per README (as suggested in this template), the latest version is 4.0.0, I tested 4.0.3.
The items in the troubleshooting guide do not apply.
Additional environment details (AWS, VirtualBox, physical, etc.):
macOS Monterey 12.3.1, M1
The text was updated successfully, but these errors were encountered: