[BUG] Podman preset is broken for linux if known_hosts file contain an existing pub key for 192.168.130.11 #3514

praveenkumar · 2023-02-14T13:20:26Z

We have to check how to avoid podman remote command to add key to the .ssh/known_hosts file or may be remove it before starting the VM.

$ crc config set preset podman
$ crc setup && crc start
[...]
podman runtime is now running.

Use the 'podman' command line interface:
  $ eval $(crc podman-env)
  $ podman-remote COMMAND

$ which podman-remote
/home/prkumar/.crc/bin/oc/podman-remote

$ podman-remote system connection ls
Name        URI                                                            Identity                                  Default
crc         ssh://core@192.168.130.11:22/run/user/1000/podman/podman.sock  /home/prkumar/.crc/machines/crc/id_ecdsa  true
crc-root    ssh://core@192.168.130.11:22/run/podman/podman.sock            /home/prkumar/.crc/machines/crc/id_ecdsa  false

$ podman-remote --log-level=debug ps  
INFO[0000] podman-remote filtering at log level debug   
DEBU[0000] Called ps.PersistentPreRunE(podman-remote --log-level=debug ps) 
DEBU[0000] SSH Ident Key "/home/prkumar/.crc/machines/crc/id_ecdsa" SHA256:kFXwhQAuHL9azolLI2Y5nrW+N3eRcmsn+/VaiBcooDc ecdsa-sha2-nistp521 
WARN[0000] ssh host key mismatch for host 192.168.130.11:22, got key SHA256:jsWTDSKOVr55u+q5PSmEZjoqLfSvNXLFmUq6hjkiXxg of type ecdsa-sha2-nistp256 
Error: failed to connect: ssh: handshake failed: knownhosts: key mismatch

$ ssh-keygen -R 192.168.130.11

$ podman-remote --log-level=debug ps  
INFO[0000] podman-remote filtering at log level debug   
DEBU[0000] Called ps.PersistentPreRunE(podman-remote --log-level=debug ps) 
DEBU[0000] SSH Ident Key "/home/prkumar/.crc/machines/crc/id_ecdsa" SHA256:kFXwhQAuHL9azolLI2Y5nrW+N3eRcmsn+/VaiBcooDc ecdsa-sha2-nistp521 
INFO[0000] key SHA256:jsWTDSKOVr55u+q5PSmEZjoqLfSvNXLFmUq6hjkiXxg added to /home/prkumar/.ssh/known_hosts 
DEBU[0000] DoRequest Method: GET URI: http://d/v4.3.1/libpod/_ping 
DEBU[0000] DoRequest Method: GET URI: http://d/v4.3.1/libpod/containers/json 
CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS       NAMES
DEBU[0000] Called ps.PersistentPostRunE(podman-remote --log-level=debug ps)

The text was updated successfully, but these errors were encountered:

adrianriobo · 2023-02-14T15:49:41Z

We saw this issue during QE testing for crc 2.14.0 on both Linux and Macos, but I was not able to reproduce the first time the key was added the known_hosts file.

As this feature seems to be introduced on podman 4.3.x and crc matrix versions for podman is:

crc 2.12.0 -> podman 4.2.0 (this did not introduce nothing on known_hosts
crc 2.13.1 -> podman 4.3.1 (this introduce key for the first time)
crc 2.14.0 -> podman 4.3.1 (bundle is the same as so it is the key, so no missmatch)

How did you end up with a different key in there?

Just in case I also tried with podman from openshift preset but in that case even crc 2.14.0 has podman 4.2.0so not key added either

adrianriobo · 2023-02-15T11:56:21Z

Today I used the same host I was using yesterday with podman preset (setup - > start -> delete -> setup -> start did not show any mismatch error).

Today I just start it again the podman preset (cleanup -> setup -> start) and got the error:

WARN[0000] ssh host key mismatch for host 127.0.0.1:2222, got key SHA256:jsWTDSKOVr55u+q5PSmEZjoqLfSvNXLFmUq6hjkiXxg of type ecdsa-sha2-nistp256 
Error: failed to connect: ssh: handshake failed: knownhosts: key mismatch

So it seems the key is not the one generated by crc on every start

INFO Generating new SSH key pair...

Neither the key from the bundle itself but some other key which also seems somehow different if you run it on different dates. 🤷‍♂️

^^ @themr0c this may need to be added as known issue for our podman preset with 4.3.1

praveenkumar · 2023-02-15T14:11:32Z

As per my reading around how ssh connection establish is the key which is added to known_hosts is the fingerprint of server's public key. The one we generate is for the client side to get access to the server. Now if server doesn't have a ssh keypair then server's public key is generated automatically by the server when it is started up and sent to the client during the SSH handshake. Now only missing piece is does that auto generated public key during ssh handshake is time bound or not.

cfergeau · 2023-02-15T15:29:07Z

I can reproduce with:

start an OpenShift bundle
ssh -p 2222 -i ~/.crc/machines/crc/id_ecdsa 127.0.0.1
start a podman 4.3.1 bundle
podman ps

adrianriobo · 2023-02-15T15:44:19Z

Are you sure? I tried yesterday and it did not break, the content when ssh with id_ecdsa is ssh-rsa.

May you already started podman preset in the previous 6 weeks and that added the key?

[127.0.0.1]:2222 ecdsa-sha2-nistp256 (podman command) vs [127.0.0.1]:2222 ssh-rsa (ssh)

cfergeau · 2023-02-15T16:42:47Z

I made sure I had no wrong key in knows_hosts before doing my test, so I'm relatively sure. Won't have time to retest before tomorrow.

praveenkumar · 2023-02-16T08:37:08Z

@adrianriobo have you tried ssh -p 2222 -i ~/.crc/machines/crc/id_ecdsa 127.0.0.1 step from the openshift bundle ? Also did you make sure that when you ssh to the instance you don't have StrictHostKeyChecking=no ?

adrianriobo · 2023-02-16T09:54:08Z

yeah that it is, ssh for machine with openshift preset then switch to podman preset and it creates the mismatch

cfergeau · 2023-02-16T11:15:02Z

This can also be reproduced with just podman-remote ps as long as you test with a new enough version of podman-remote:

start a 4.3.1 podman bundle
run eval $(crc podman-env); podman-remote ps
copy ~/.crc/bin/oc/podman-remote to /tmp
start a 4.2.0 podman bundle
/tmp/podman-remote ps fails: Error: failed to connect: ssh: handshake failed: knownhosts: key mismatch

cfergeau · 2023-02-16T11:17:09Z

Would be useful to us if podman had the option Praveen suggested on slack:

looking into https://github.com/containers/common/blob/main/pkg/ssh/connection_golang.go#L221 can we add another flag to podman system connection add -h to have strict-host-key-checking and set it false by default so https://github.com/containers/common/blob/main/pkg/ssh/connection_golang.go#L287 should be executed?

anjannath · 2023-03-28T08:48:22Z

As a work-around, untill it is handled/resolved from podman code, we can add a preflight check or a cluster post-start or on crc delete remove the CRC host's entries from the known_hosts file.

if we just remove the 127.0.01:2222 entries from it at least users on windows and darwin will not face this issue.

to also work-around the issue in case of system-mode networking (linux) where the VM ip address might change it could be complicated

while using the podman preset in CRC the instance identity is added to the known_hosts file and when it changes podman-remote complains about it and shows the following error: ``` Error: failed to connect: ssh: handshake failed: knownhosts: key mismatch ``` this will occur in new podman releases from 4.4.1 whenever the instance identity changes which might occur if you delete and re-create the podman instance, with this patch it'll remove the [127.0.0.1]:2222 entries from known_hosts which should work-around this issue

while using the podman preset in CRC the instance identity is added to the known_hosts file and when it changes podman-remote complains about it and shows the following error: ``` Error: failed to connect: ssh: handshake failed: knownhosts: key mismatch ``` this will occur in crc when users upgrade podman from 4.3.1 to 4.4.1 with this patch it'll remove the [127.0.0.1]:2222 entries from known_hosts which should work-around this issue

anjannath · 2023-04-05T10:45:27Z

With #3564 the CRC specific entries are removed from known_hosts file when user runs crc delete. But this doesn't fix the issue for users upgrading from 2.15.0 to 2.16.0 as they'll install the newer release and then run crc setup. And since it doesn't remove the entries during crc setup or crc cleanup, user's are likely to face this issue when they upgrade..

For users's upgrading from 2.15.0 to 2.16.0 to work-around this issue, they should follow the steps:

Install 2.16.0 OpenShift Local using installer or tarball
run crc delete => this'll remove the CRC specific entries
run crc cleanup and crc setup

praveenkumar added kind/bug Something isn't working status/need triage labels Feb 14, 2023

anjannath removed the status/need triage label Apr 5, 2023

gbraad added this to Quality Engineering Apr 10, 2024

gbraad moved this to Backlog in Quality Engineering Apr 10, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] Podman preset is broken for linux if known_hosts file contain an existing pub key for 192.168.130.11 #3514

[BUG] Podman preset is broken for linux if known_hosts file contain an existing pub key for 192.168.130.11 #3514

praveenkumar commented Feb 14, 2023

adrianriobo commented Feb 14, 2023

adrianriobo commented Feb 15, 2023

praveenkumar commented Feb 15, 2023

cfergeau commented Feb 15, 2023

adrianriobo commented Feb 15, 2023 •

edited

Loading

cfergeau commented Feb 15, 2023

praveenkumar commented Feb 16, 2023

adrianriobo commented Feb 16, 2023

cfergeau commented Feb 16, 2023 •

edited

Loading

cfergeau commented Feb 16, 2023

anjannath commented Mar 28, 2023 •

edited

Loading

anjannath commented Apr 5, 2023 •

edited

Loading

[BUG] Podman preset is broken for linux if known_hosts file contain an existing pub key for 192.168.130.11 #3514

[BUG] Podman preset is broken for linux if known_hosts file contain an existing pub key for 192.168.130.11 #3514

Comments

praveenkumar commented Feb 14, 2023

adrianriobo commented Feb 14, 2023

adrianriobo commented Feb 15, 2023

praveenkumar commented Feb 15, 2023

cfergeau commented Feb 15, 2023

adrianriobo commented Feb 15, 2023 • edited Loading

cfergeau commented Feb 15, 2023

praveenkumar commented Feb 16, 2023

adrianriobo commented Feb 16, 2023

cfergeau commented Feb 16, 2023 • edited Loading

cfergeau commented Feb 16, 2023

anjannath commented Mar 28, 2023 • edited Loading

anjannath commented Apr 5, 2023 • edited Loading

adrianriobo commented Feb 15, 2023 •

edited

Loading

cfergeau commented Feb 16, 2023 •

edited

Loading

anjannath commented Mar 28, 2023 •

edited

Loading

anjannath commented Apr 5, 2023 •

edited

Loading