Document the requirements for the ssh keys

[christian-heusel]

Apparently RSA Keys do not work with breakglass:

# ecdsa
2022/03/03 23:32:04 breakglass.go:117: user "chris" successfully authorized from remote addr 1.2.3.4:57006
2022/03/03 23:32:04 ssh.go:316: Starting cmd ["/tmp/serial-busybox/ash" "-c" "sh"]
2022/03/03 23:32:06 ssh.go:142: requests exhausted
# ed25519
2022/03/03 23:32:17 breakglass.go:117: user "chris" successfully authorized from remote addr 1.2.3.4:57008
2022/03/03 23:32:17 ssh.go:316: Starting cmd ["/tmp/serial-busybox/ash" "-c" "sh"]
2022/03/03 23:32:20 ssh.go:142: requests exhausted
# rsa
2022/03/03 23:32:31 breakglass.go:174: handshake: [ssh: no auth passed yet]

ssh output (click to expand)

$ ssh -v -i testkey_rsa johnny                                                                  255 ↵
OpenSSH_8.9p1 Debian-3, OpenSSL 1.1.1m  14 Dec 2021
debug1: Reading configuration data /home/chris/.ssh/config
debug1: Reading configuration data /home/chris/.ssh/config_mathi
debug1: Reading configuration data /home/chris/.ssh/config_mathphys
debug1: Reading configuration data /home/chris/.ssh/config_mathphys_teleport_proxy
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to johnny [1.2.3.4] port 22.
debug1: connect to address 1.2.3.4 port 22: No route to host
debug1: Connecting to johnny [1.2.3.4] port 22.
debug1: connect to address 1.2.3.4 port 22: No route to host
debug1: Connecting to johnny [1.2.3.4] port 22.
debug1: Connection established.
debug1: identity file testkey_rsa type 0
debug1: identity file testkey_rsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Debian-3
debug1: Remote protocol version 2.0, remote software version Go
debug1: compat_banner: no match: Go
debug1: Authenticating to johnny:22 as 'chris'
debug1: load_hostkeys: fopen /home/chris/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-rsa SHA256:6r4pozsLKfLbvVtiJcbFBiunBpMeOR9ecnFrO4fm8KY
debug1: load_hostkeys: fopen /home/chris/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'johnny' is known and matches the RSA host key.
debug1: Found key in /home/chris/.ssh/known_hosts:341
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: get_agent_identities: bound agent to hostkey
debug1: get_agent_identities: agent returned 2 keys
debug1: Will attempt key: /home/chris/.ssh/id_rsa RSA SHA256:eZ4lCaCVZVXbOcBNqyLI4XLDjyUEoExRu3GF+hgSlMQ agent
debug1: Will attempt key: /home/chris/.ssh/christian@heusel.eu RSA SHA256:AAtYt2TtXf4QHT+c2uTLiSACq6PNdEj/6X6/CUfSXb8 agent
debug1: Will attempt key: testkey_rsa RSA SHA256:+9EnpHmzuv0vNhzsU/hwPdv03YhLMX/4tIJN/KH0Dto explicit
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/chris/.ssh/id_rsa RSA SHA256:eZ4lCaCVZVXbOcBNqyLI4XLDjyUEoExRu3GF+hgSlMQ agent
debug1: send_pubkey_test: no mutual signature algorithm
debug1: Offering public key: /home/chris/.ssh/christian@heusel.eu RSA SHA256:AAtYt2TtXf4QHT+c2uTLiSACq6PNdEj/6X6/CUfSXb8 agent
debug1: send_pubkey_test: no mutual signature algorithm
debug1: Offering public key: testkey_rsa RSA SHA256:+9EnpHmzuv0vNhzsU/hwPdv03YhLMX/4tIJN/KH0Dto explicit
debug1: send_pubkey_test: no mutual signature algorithm
debug1: No more authentication methods to try.
chris@johnny: Permission denied (publickey).

I am not sure whether this is expected and just needs documentation or if its is a bug ...

[stapelberg]

I’m not so sure either. rsa keys used to work, but at some point that stopped working. I’m not sure if it was a change in OpenSSH (i.e. on the client side) or in the x/crypto package (i.e. on the server side).

Personally I just switched to ed25519 and called it a day. Sounds like some more detective work is necessary, but we could just recommend ed25519 for now

[christian-heusel]

Yeah for me this is (now that I know about it) not big of a deal aswell, I just think the error is not that easy to spot. Especially if you do not run ssh in verbose mode you just get "Permission denied (publickey)". Newcomers (like me yesterday) might rather question if the overall gokrazy deployment they created is right than check for the ssh output.

IMO a little notice in the readme should suffice!

[stapelberg]

For sure. We can add a note for now until someone wants to look deeper into whether rsa can be made working again and how. Do you want to send a PR with the note? :)

[christian-heusel]

So apparently it was a change in OpenSSH and x/crypto/ssh, this seems to be an issue in the golang
I think I found the tracking issues:

• Overview Issue: x/crypto/ssh: rsa-sha2-256/rsa-sha2-512 tracking issue golang/go#49952
• publicKeyCallback: x/crypto/ssh: publicKeyCallback cannot handshake using ssh-rsa keys signed using the ssh-rsa-sha2-256 algorithm golang/go#39885

https://github.com/rmohr/crypto is apparently a fork that implements authentication via RSA keys, for the time being I wouldnt switch but rather just wait until its fixed in mainline! 😊

[stapelberg]

Thanks for investigating! Yeah, waiting like x/crypto fixes it sounds good to me :)

[bradfitz]

@stapelberg, I think this can be closed now. The x/crypto changes seem to work.

I just tested and was able to use an RSA key to log in to my Gokrazy VM on EC2:

% ssh -v -i ~/.ssh/id_rsa_unifi root@35.92.217.124
OpenSSH_9.6p1, LibreSSL 3.3.6
debug1: Reading configuration data /Users/bradfitz/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to 35.92.217.124 [35.92.217.124] port 22.
debug1: Connection established.
debug1: identity file /Users/bradfitz/.ssh/id_rsa_unifi type 0
debug1: identity file /Users/bradfitz/.ssh/id_rsa_unifi-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.6
debug1: Remote protocol version 2.0, remote software version Go
debug1: compat_banner: no match: Go
debug1: Authenticating to 35.92.217.124:22 as 'root'
debug1: load_hostkeys: fopen /Users/bradfitz/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-rsa SHA256:EkcngHyRLWLbreyN8IKdB+MjFxiJMSnD9IAINUd9KIQ
debug1: load_hostkeys: fopen /Users/bradfitz/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host '35.92.217.124' is known and matches the RSA host key.
debug1: Found key in /Users/bradfitz/.ssh/known_hosts:563
debug1: ssh_packet_send2_wrapped: resetting send seqnr 3
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: ssh_packet_read_poll2: resetting read seqnr 3
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_ext_info_client_parse: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-256,rsa-sha2-512,ssh-rsa,ssh-dss>
debug1: kex_ext_info_check_ver: ping@openssh.com=<0>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: get_agent_identities: bound agent to hostkey
debug1: get_agent_identities: ssh_fetch_identitylist: agent contains no identities
debug1: Will attempt key: /Users/bradfitz/.ssh/id_rsa_unifi RSA SHA256:ge7LW6YaeJQthdw3C8Omo61B2mveNVbz0SbPQcpgESA explicit
debug1: Offering public key: /Users/bradfitz/.ssh/id_rsa_unifi RSA SHA256:ge7LW6YaeJQthdw3C8Omo61B2mveNVbz0SbPQcpgESA explicit
debug1: Server accepts key: /Users/bradfitz/.ssh/id_rsa_unifi RSA SHA256:ge7LW6YaeJQthdw3C8Omo61B2mveNVbz0SbPQcpgESA explicit
Authenticated to 35.92.217.124 ([35.92.217.124]:22) using "publickey".
debug1: channel 0: new session [client-session] (inactive timeout: 0)
debug1: Entering interactive session.
debug1: pledge: filesystem
debug1: Sending environment.
debug1: channel 0: setting env LANG = "en_US.UTF-8"
/tmp/breakglass1602701109 # 

Notable lines from above:

debug1: kex: host key algorithm: rsa-sha2-512
...
debug1: Will attempt key: /Users/bradfitz/.ssh/id_rsa_unifi RSA SHA256:ge7LW6YaeJQthdw3C8Omo61B2mveNVbz0SbPQcpgESA explicit
debug1: Offering public key: /Users/bradfitz/.ssh/id_rsa_unifi RSA SHA256:ge7LW6YaeJQthdw3C8Omo61B2mveNVbz0SbPQcpgESA explicit
debug1: Server accepts key: /Users/bradfitz/.ssh/id_rsa_unifi RSA SHA256:ge7LW6YaeJQthdw3C8Omo61B2mveNVbz0SbPQcpgESA explicit
Authenticated to 35.92.217.124 ([35.92.217.124]:22) using "publickey".