F37 can't make ssh connection

[karlkleinpaste]

Upgraded a system to Fedora 37.
5.0-10.r32344.
Can't connect to a remote with ssh. Several problems to detail here.

First is that xpra (well... paramiko?) is not handling .ssh/config options right. I have been doing this for eons:
xpra attach ssh://dt/73 --border=yellow,1
What's important is that, on the local system, I'm karl, but on the remote, I'm kkleinpa. Yet it fails to handle this routine .ssh/config detail, attempting to login as karl.

Host dt
        HostName chqkkleinpa1
        User kkleinpa

The connection is correctly made to chqkkleinpa1 (see error message at end, "failed for chqkkleinpa1") but the username is wrong.

2022-11-25 07:59:13,529 Xpra GTK3 X11 client version 5.0-r32344 (gb97ec0bf1) beta 64-bit
2022-11-25 07:59:13,532  running on Linux
2022-11-25 07:59:13,532  window manager is 'compiz'
2022-11-25 07:59:13,751 GStreamer version 1.20.4
2022-11-25 07:59:13,871 created unix domain socket '/run/user/5271/xpra/clients/godiva.kleinpaste.org-129803'
2022-11-25 07:59:13,912 OpenGL_accelerate module loaded
2022-11-25 07:59:13,915 Using accelerated ArrayDatatype
2022-11-25 07:59:14,131 OpenGL enabled on 'NVIDIA GeForce RTX 2070 with Max-Q Design/PCIe/SSE2'
2022-11-25 07:59:14,518 Connected (version 2.0, client OpenSSH_8.0)
2022-11-25 07:59:15,216 Authentication (publickey) failed.
2022-11-25 07:59:15,216 SSH agent key 'MD5:87:b4:ad:51:e8:75:69:76:fe:4f:7e:25:4e:03:70:02' rejected for user 'karl'
2022-11-25 07:59:15,328 Authentication (publickey) failed.
2022-11-25 07:59:15,328 SSH agent key 'MD5:31:39:d7:4a:7c:2c:19:03:3f:dd:14:58:3c:2f:63:62' rejected for user 'karl'
2022-11-25 07:59:15,445 Authentication (publickey) failed.
2022-11-25 07:59:15,445 SSH agent key 'MD5:de:5e:ea:f6:d3:bf:dc:dc:88:c0:a2:3b:3b:9e:cd:dd' rejected for user 'karl'
2022-11-25 07:59:15,561 Authentication (publickey) failed.
2022-11-25 07:59:15,562 SSH agent key 'MD5:c6:f9:70:9c:9d:5c:27:77:1d:d8:1f:fe:32:2d:fb:0f' rejected for user 'karl'
2022-11-25 07:59:15,679 Authentication (publickey) failed.
2022-11-25 07:59:15,679 SSH agent key 'MD5:1d:f5:59:d0:b6:4d:7f:e1:04:72:b8:a3:05:2b:6f:65' rejected for user 'karl'
2022-11-25 07:59:15,808 Disconnect (code 2): Too many authentication failures
2022-11-25 07:59:15,808 SSH agent key 'MD5:84:51:b4:c3:cf:cf:0e:64:e6:58:0b:b6:51:ec:76:f0' rejected for user 'karl'
2022-11-25 07:59:15,808 agent authentication failed, tried 6 keys
2022-11-25 07:59:15,808 loaded Ed25519 private key from '/home/karl/.ssh/id_ed25519'
2022-11-25 07:59:15,808 SSH authentication using key '/home/karl/.ssh/id_ed25519' failed:
2022-11-25 07:59:15,808  No existing session
2022-11-25 07:59:15,808 retrying SSH authentication with modes none, password
2022-11-25 07:59:15,925 Connected (version 2.0, client OpenSSH_8.0)
2022-11-25 07:59:16,854 SSH password authentication failed:
2022-11-25 07:59:16,854  Bad authentication type
2022-11-25 07:59:16,854   allowed types: ['publickey', 'gssapi-keyex', 'gssapi-with-mic', 'password']
Warning: failed to connect:
 SSH Authentication failed for 'chqkkleinpa1'
2022-11-25 07:59:19,207 removing unix domain socket '/run/user/5271/xpra/clients/godiva.kleinpaste.org-129803'

This does not occur when I add --ssh=/usr/bin/ssh.

Second, I still get failure when connecting to that host. What's running at the remote is Centos8 with mate-terminal spawned under 5.0-10.r32045 that has been running since Nov 17.

xpra attach --ssh=/usr/bin/ssh ssh://dt/73 --border=yellow,1
2022-11-25 08:08:15,065 Xpra GTK3 X11 client version 5.0-r32344 (gb97ec0bf1) beta 64-bit
2022-11-25 08:08:15,068  running on Linux
2022-11-25 08:08:15,068  window manager is 'compiz'
2022-11-25 08:08:15,285 GStreamer version 1.20.4
2022-11-25 08:08:15,368 created unix domain socket '/run/user/5271/xpra/clients/godiva.kleinpaste.org-135576'
2022-11-25 08:08:15,413 OpenGL_accelerate module loaded
2022-11-25 08:08:15,417 Using accelerated ArrayDatatype
2022-11-25 08:08:15,625 OpenGL enabled on 'NVIDIA GeForce RTX 2070 with Max-Q Design/PCIe/SSE2'
2022-11-25 08:08:15,632  keyboard settings: rules=evdev, model=pc105, layout=us
2022-11-25 08:08:15,634  desktop size is 5760x1080:
2022-11-25 08:08:15,634   :0.0 (1023x191 mm - DPI: 143x144) workarea: 5760x1029 at    0x26  
2022-11-25 08:08:15,634     PRO DP-3         1920x1080 at    0x0    (344x193 mm - DPI: 142x142) workarea: 1920x1029 at    0x26
2022-11-25 08:08:15,634     AUO DP-2         1920x1080 at 1920x0    (344x193 mm - DPI: 142x142) workarea: 1920x1080 at 1920x0
2022-11-25 08:08:15,634     PRO DP-1         1920x1080 at 3840x0    (344x193 mm - DPI: 142x142) workarea: 1920x1080 at 3840x0
Received disconnect from 10.5.124.236 port 22:2: Too many authentication failures
Disconnected from 10.5.124.236 port 22
2022-11-25 08:08:16,470 Error: failed to receive anything, not an xpra server?
2022-11-25 08:08:16,470   could also be the wrong protocol, username, password or port
2022-11-25 08:08:16,470   or the session was not found
2022-11-25 08:08:16,470 Connection failed
2022-11-25 08:08:16,481 removing unix domain socket '/run/user/5271/xpra/clients/godiva.kleinpaste.org-135576'

I promise you, what's running there is in fact xpra, and it was functioning fine prior to F37 upgrade with 5.0-10.r32344. Adding --debug=ssh adds this to the log output, after "OpenGL enabled" above:

2022-11-25 08:21:37,783 debug enabled for xpra.net.ssh.exec_client / ('network', 'ssh')
2022-11-25 08:21:37,783 executing ssh command: "/usr/bin/ssh" "-l" "karl" "-T" "dt" "sh -c 'if command -v "xpra" > /dev/null 2>&1; then xpra _proxy;elif [ -x $XDG_RUNTIME_DIR/xpra/run-xpra ]; then $XDG_RUNTIME_DIR/xpra/run-xpra _proxy;elif [ -x /usr/local/bin/xpra ]; then /usr/local/bin/xpra _proxy;elif [ -x ~/.xpra/run-xpra ]; then ~/.xpra/run-xpra _proxy;elif [ -x Xpra_cmd.exe ]; then Xpra_cmd.exe _proxy;else echo "no run-xpra command found"; exit 1; fi'"

Routine ssh dt works just fine when invoked from the shell, so I can't guess right this instant how it's getting "too many auth failures."

When I don't specify --ssh, add kkleinpa explicitly, then I get this oddity:

xpra attach ssh://kkleinpa@dt/73 --border=yellow,1
2022-11-25 08:33:34,390 Xpra GTK3 X11 client version 5.0-r32344 (gb97ec0bf1) beta 64-bit
2022-11-25 08:33:34,393  running on Linux
2022-11-25 08:33:34,393  window manager is 'compiz'
2022-11-25 08:33:34,625 GStreamer version 1.20.4
2022-11-25 08:33:34,742 created unix domain socket '/run/user/5271/xpra/clients/godiva.kleinpaste.org-151977'
2022-11-25 08:33:34,783 OpenGL_accelerate module loaded
2022-11-25 08:33:34,787 Using accelerated ArrayDatatype
2022-11-25 08:33:34,988 OpenGL enabled on 'NVIDIA GeForce RTX 2070 with Max-Q Design/PCIe/SSE2'
2022-11-25 08:33:35,119 Connected (version 2.0, client OpenSSH_8.0)
2022-11-25 08:33:35,577 Authentication (publickey) successful!
2022-11-25 08:33:36,024 ssh server OS is 'linux-gnu'
2022-11-25 08:33:36,601 paramiko SSH agent forwarding enabled
2022-11-25 08:33:36,757  keyboard settings: rules=evdev, model=pc105, layout=us
2022-11-25 08:33:36,760  desktop size is 5760x1080:
2022-11-25 08:33:36,760   :0.0 (1023x191 mm - DPI: 143x144) workarea: 5760x1029 at    0x26  
2022-11-25 08:33:36,760     PRO DP-3         1920x1080 at    0x0    (344x193 mm - DPI: 142x142) workarea: 1920x1029 at    0x26
2022-11-25 08:33:36,760     AUO DP-2         1920x1080 at 1920x0    (344x193 mm - DPI: 142x142) workarea: 1920x1080 at 1920x0
2022-11-25 08:33:36,760     PRO DP-1         1920x1080 at 3840x0    (344x193 mm - DPI: 142x142) workarea: 1920x1080 at 3840x0
2022-11-25 08:33:36,961  SSH: 'xpra initialization error:'
2022-11-25 08:33:36,961  SSH: ' there are multiple servers running, please specify'
2022-11-25 08:33:36,961 SSH EOF on stderr of run-xpra
2022-11-25 08:33:36,962 Error: failed to receive anything, not an xpra server?
2022-11-25 08:33:36,962   could also be the wrong protocol, username, password or port
2022-11-25 08:33:36,962   or the session was not found
2022-11-25 08:33:36,962 Connection failed
2022-11-25 08:33:36,972 removing unix domain socket '/run/user/5271/xpra/clients/godiva.kleinpaste.org-151977'

Well... yeah, there are multiple servers; I specified 73 for a reason.

Needless to say, this is a showstopper.

[karlkleinpaste]

I tried local, self-referential testing, not depending on a remote with an older xpra. This did not go well.

xpra start ssh://pg/73 --attach=no --start-child=vterm
SSH failed to connect to pg:22

"pg" is a local VPN IP address of the machine. Problem here is that, on personal systems, I run sshd on a nonstandard port. So again, it is failing to use .ssh/config properly.

Host pg
        Port 7181

(It's actually a list of hosts using that port, including name and IP wildcards; abbreviated here for simplicity.)
So I added the port explicitly, and life just doesn't get any happier.

xpra start ssh://pg:7181/73 --attach=no --start-child=vterm
2022-11-25 08:50:55,683 Connected (version 2.0, client OpenSSH_8.8)
2022-11-25 08:50:55,991 Warning: unknown SSH host 'pg'
2022-11-25 08:50:59,597 host key confirmed
2022-11-25 08:50:59,629 Authentication (publickey) successful!
2022-11-25 08:50:59,746 ssh server OS is 'linux-gnu'
2022-11-25 08:50:59,869 paramiko SSH agent forwarding enabled
2022-11-25 08:51:00,191  SSH: 'Entering daemon mode; any further errors will be reported to:'
2022-11-25 08:51:00,191  SSH: "  '/run/user/5271/xpra/73/server.log'"
2022-11-25 08:51:00,355  SSH: 'vfb failed to start, exiting'
2022-11-25 08:51:29,943 connection timed out
2022-11-25 08:51:29,943 SSH EOF on stderr of run-xpra
2022-11-25 08:51:29,946 timeout: server did not disconnect us

The logfile is informative.

2022-11-25 08:51:00,248 no uinput module (not usually needed)
xauth: (argv):1:  bad display name "73" in "add" command
2022-11-25 08:51:00,254 Error adding xauth entry for 73
2022-11-25 08:51:00,254  using command "xauth -f /home/karl/.Xauthority add 73 MIT-MAGIC-COOKIE-1 65fc54486ac7461ab039e1d385f5563c":
2022-11-25 08:51:00,254  non-zero exit code: 1
Unrecognized option: 73
use: X [:<display>] [option]
-a #                   default pointer acceleration (factor)
...

And so on, with the rest of failed X startup diagnostics.

[totaam]

Upgraded a system to Fedora 37.

Apart from these ssh bugs, it's a good thing you're on the beta channel for F37.

First is that xpra (well... paramiko?) is not handling .ssh/config options right

Considering how much code was removed or refactored in #3599, I am actually surprised it took this long for something to break! (sorry it had to fall on you)
3280205 fixes this.

Problem here is that, on personal systems, I run sshd on a nonstandard port.

Same cause, fixed by the same commit.

Well... yeah, there are multiple servers; I specified 73 for a reason.

Similar cause, different fix: e43a738

xauth: (argv):1: bad display name "73" in "add" command

Fixed in 2a62420
I was trying to move away from the : prefix for display names, but we need this for backwards compatibility.

Unless I have missed something, this should solve all your problems.
(try 5.0-r32356 or later)

[karlkleinpaste]

(try 5.0-r32356 or later)

I'll give it another shot as soon as next *.rpm builds are available.

[karlkleinpaste]

5.0-10.r32356 and all seems well now. Thx.