paramiko client on windows cannot connect to ssh-agent, agent requests hang

[leo-b]

On my windows 10 client, the builtin paramiko ssh client enables ssh agent forwarding but cannot connect to the running ssh agent.

"C:\Program Files\Xpra\Xpra_cmd.exe" start ssh://leo@desthost/100 --start-child=qterminal --exit-with-children=yes --audio=no --webcam=no
[...]
2024-09-10 14:42:30,092 connect_to({'display_name': 'ssh://leo@strike.wu.ac.at/100', 'cmdline': ['C:\\Program Files\\Xpra\\Xpra_cmd.exe', 'start', 'ssh://leo@strike.wu.ac.at/100', '--start-child=qterminal', '--exit-with-children=yes', '--audio=no', '--webcam=no', '-d', 'ssh'], 'type': 'ssh', 'proxy_command': ['_proxy_start'], 'exit_ssh': True, 'remote_xpra': ['xpra', '$XDG_RUNTIME_DIR/xpra/run-xpra', '/usr/local/bin/xpra', '~/.xpra/run-xpra', 'Xpra_cmd.exe'], 'username': 'leo', 'host': 'strike.wu.ac.at', 'local': False, 'port': 22, 'display': '100', 'display_as_args': [':100', '--env=SSH_AGENT_UUID=9394cbc69f51b9c9721191355cc11f3cb9053289cee33742dd88ce04d7d254fd', '--debug=ssh', '--audio=no', '--webcam=no', '--exit-with-children=yes', '--start-child=qterminal'], 'is_paramiko': True, 'agent': True, 'ssh-agent-uuid': '9394cbc69f51b9c9721191355cc11f3cb9053289cee33742dd88ce04d7d254fd', 'full_ssh': ['paramiko']})
[...]
2024-09-10 14:42:36,922 paramiko agent_option='True'
2024-09-10 14:42:36,922 paramiko SSH agent forwarding enabled

Trying to use agent forwarding blocks because paramiko cannot contact the agent:

Remote system (linux):

$ echo $SSH_AUTH_SOCK 
/run/user/500/xpra/100/ssh/agent
$ ssh-add -l
# <hangs>

At the same time his request raises the following exception at the local windows client:

Exception in thread Thread-5 (run):
Traceback (most recent call last):
  File "C:/msys64/mingw64/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
  File "C:/msys64/mingw64/lib/python3.11/site-packages/paramiko/agent.py", line 145, in run
paramiko.ssh_exception.AuthenticationException: Unable to connect to SSH agent

However the agent is running: (tested in the same cmd window)

C:\Windows\System32>ssh-add -l
3072 SHA256:sRBddv98pK6MaXBXpAKKBVS2b8y5eUfUFmSKG7zbx7A leo@leo-x1 (RSA)

C:\Windows\System32>python
Python 3.11.9 (tags/v3.11.9:de54cf5, Apr  2 2024, 10:12:12) [MSC v.1938 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import paramiko.agent
>>> paramiko.agent.get_agent_connection()
<paramiko.win_openssh.OpenSSHAgentConnection object at 0x0000020730200DD0>

Using --ssh="plink -A" works but putty uses pageant and I'd prefer the openssh agent.

[totaam]

Python 3.11.9 (tags/v3.11.9:de54cf5, Apr 2 2024, 10:12:12) [MSC v.1938 64 bit (AMD64)] on win32

This doesn't look like the python that we ship.
I get: Python 3.11.9 (main, Apr 12 2024, 09:55:31) [GCC 13.2.0 64 bit (AMD64)] on win32
So the result may well be different.
You can run the same interpreter as xpra with Python_exec_cmd "from paramiko.agent import get_agent_connection;get_agent_connection()".

Looking at the paramiko source, this is the context:

class AgentProxyThread(threading.Thread):
    """
    Class in charge of communication between two channels.
    """

    def __init__(self, agent):
        threading.Thread.__init__(self, target=self.run)
        self._agent = agent
        self._exit = False

    def run(self):
        try:
            (r, addr) = self.get_connection()
            # Found that r should be either
            # a socket from the socket library or None
            self.__inr = r
            # The address should be an IP address as a string? or None
            self.__addr = addr
            self._agent.connect()
            if not isinstance(self._agent, int) and (
                self._agent._conn is None
                or not hasattr(self._agent._conn, "fileno")
            ):
                raise AuthenticationException("Unable to connect to SSH agent")
            self._communicate()
        except:
            # XXX Not sure what to do here ... raise or pass ?
            raise

I'm not sure if having an paramiko.win_openssh.OpenSSHAgentConnection is enough. Does it have a fileno?

[leo-b]

You can run the same interpreter as xpra with Python_exec_cmd "from paramiko.agent import get_agent_connection;get_agent_connection()".

Python_exec_cmd "from paramiko.agent import get_agent_connection; print(get_agent_connection())"
<paramiko.win_openssh.OpenSSHAgentConnection object at 0x000001ffb1bcd2d0>

I'm not sure if having an paramiko.win_openssh.OpenSSHAgentConnection is enough. Does it have a fileno?

No. Unfortunately self._agent._conn doesn't seem to directly correspond to that OpenSSHAgentConnection object. Looks like a debugger is needed. Unfortunately I only have windows binary release which I guess will not be sufficient.. :-(

[leo-b]

It looks like it simply isn't implemented on windows:
paramiko/paramiko#2179

Could you maybe disable agent forwarding on windows to avoid hangs when the agent is accessed on the remote system?
(I can use --ssh=paramiko:agent=no of course, but I guess others will also stumble over that.)

[totaam]

I believe the other agents do work - so the solution might be to block the import of win_openssh.OpenSSHAgentConnection.

[totaam]

Please try the 6.2-r36663 or later beta builds which include 9e98dbd - does that work for you?

Ideally paramiko would fix this bug - but seeing that even trivial fixes with unit tests aren't being merged, I seriously doubt it.

[leo-b]

Prevention of the module import of win_openssh.OpenSSHAgentConnection seems to work:

Xpra_cmd.exe start ssh://leo@strike.wu.ac.at/100 --start-child=qterminal --exit-with-children=yes --audio=no --webcam=no -d ssh
[...]
2024-09-11 19:01:14,620 Xpra GTK3 client version 6.2.0-r36663 (g9177da5ed9) beta (light build)
[...]
2024-09-11 19:01:28,213 paramiko agent_option='True'
2024-09-11 19:01:28,215 preventing 'win_openssh.OpenSSHAgentConnection' from loading
2024-09-11 19:01:28,216 paramiko SSH agent forwarding enabled

But the forwarding channel is still activated and using the agent on the remote side still blocks and causes the same exception on the client:

2024-09-11 19:01:32,711 Secsh channel 5 (auth-agent@openssh.com) opened.
Exception in thread Thread-5 (run):
Traceback (most recent call last):
  File "C:/msys64/mingw64/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
  File "C:/msys64/mingw64/lib/python3.11/site-packages/paramiko/agent.py", line 145, in run
paramiko.ssh_exception.AuthenticationException: Unable to connect to SSH agent

[totaam]

It turns out that paramiko already does its own checks before loading the agent:
https://github.com/paramiko/paramiko/blob/10115b7367a7a92223801ea22c50de6a30c8f2a3/paramiko/agent.py#L219-L244
Here is the check for pageant:
https://github.com/paramiko/paramiko/blob/10115b7367a7a92223801ea22c50de6a30c8f2a3/paramiko/win_pageant.py#L47-L60
And the one for openssh:
https://github.com/paramiko/paramiko/blob/10115b7367a7a92223801ea22c50de6a30c8f2a3/paramiko/win_openssh.py#L26-L32

So I think I will revert my blocking change. The openssh agent should work if the pipe is found.
(I guess the paramiko check could be improved by trying to open the pipe rather than just checking if it's there)

What we may need to do instead is to check if a local agent is actually available before enabling agent forwarding on the server side.
The problem with doing that is that other clients connecting to this session may want to have the agent forwarding enabled.
So for now, I suggest that you run with:

xpra_cmd.exe --env=XPRA_SSH_AGENT_DISPATCH=0 ...

This should be forwarded to the remote start command and will disable agent forwarding for the server session.
(and perhaps we should promote this env var to a command line switch)

[leo-b]

So I think I will revert my blocking change.

Makes sense.

The openssh agent should work if the pipe is found.

Unfortunately that is still not true on my system even though the openssh agent is running and working:

>ssh-add
Identity added: C:\Users\leo/.ssh/id_rsa (leo@leo-x1)

>ssh-add -l
3072 SHA256:sRBddv98pK6MaXBXpAKKBVS2b8y5eUfUFmSKG7zbx7A leo@leo-x1 (RSA)

>.\Python_exec_cmd.exe "from paramiko.win_openssh import can_talk_to_agent, OpenSSHAgentConnection; print(can_talk_to_agent()); aconn = OpenSSHAgentConnection(); print(aconn)"
True
<paramiko.win_openssh.OpenSSHAgentConnection object at 0x000001faf0b7a590>

Accessing the agent on the remote host still blocks with the above paramiko.ssh_exception.AuthenticationException: Unable to connect to SSH agent exception.

Does openssh agent forwarding acually work in your windows test environment?

I'd be suprised because I don't think that this will work on Windows:
https://github.com/paramiko/paramiko/blob/10115b7367a7a92223801ea22c50de6a30c8f2a3/paramiko/agent.py#L196

.\Python_exec_cmd.exe "import socket; socket.AF_UNIX"
Traceback (most recent call last):
  File "C:/msys64/mingw64/lib/python3.11/site-packages/cx_Freeze/initscripts/__startup__.py", line 141, in run
  File "C:/msys64/mingw64/lib/python3.11/site-packages/cx_Freeze/initscripts/console.py", line 25, in run
  File "xpra/platform/win32/scripts/exec.py", line 20, in <module>
  File "<string>", line 1, in <module>
AttributeError: module 'socket' has no attribute 'AF_UNIX'

So for now, I suggest that you run with:
xpra_cmd.exe --env=XPRA_SSH_AGENT_DISPATCH=0 ...

This doesn't work for me. It still enables a forwarding channel and connects the remote agent to it:

>xpra_cmd.exe --env=XPRA_SSH_AGENT_DISPATCH=0 start ssh://leo@strike.wu.ac.at/100 --start-child=qterminal --exit-with-children=yes --audio=no --webcam=no -d ssh
[...]
2024-09-12 15:47:18,082 connect_to({'display_name': 'ssh://leo@strike.wu.ac.at/100', 'cmdline': ['xpra_cmd.exe', '--env=XPRA_SSH_AGENT_DISPATCH=0', 'start', 'ssh://leo@strike.wu.ac.at/100', '--start-child=qterminal', '--exit-with-children=yes', '--audio=no', '--webcam=no', '-d', 'ssh'], 'type': 'ssh', 'proxy_command': ['_proxy_start'], 'exit_ssh': True, 'remote_xpra': ['xpra', '$XDG_RUNTIME_DIR/xpra/run-xpra', '/usr/local/bin/xpra', '~/.xpra/run-xpra', 'Xpra_cmd.exe'], 'username': 'leo', 'host': 'strike.wu.ac.at', 'local': False, 'port': 22, 'display': '100', 'display_as_args': [':100', '--env=SSH_AGENT_UUID=63bf288e68403ce0c9103963e2c6c6380ff4f3c18d55ccc8227565d7cc2ed8ab', '--debug=ssh', '--audio=no', '--webcam=no', '--exit-with-children=yes', '--start-child=qterminal', '--env=XPRA_SSH_AGENT_DISPATCH=0'], 'is_paramiko': True, 'agent': True, 'ssh-agent-uuid': '63bf288e68403ce0c9103963e2c6c6380ff4f3c18d55ccc8227565d7cc2ed8ab', 'full_ssh': ['paramiko']})
[...]
2024-09-12 15:47:18,222 do_connect_to(<paramiko.Transport at 0x37c35bd0 (cipher aes128-ctr, 128 bits) (connected; awaiting auth)>, 'strike.wu.ac.at', 'leo', '', {'hostname': 'strike.wu.ac.at'}, ['C:\\Users\\leo/.ssh\\id_ed25519', 'C:\\Users\\leo/.ssh\\id_ecdsa', 'C:\\Users\\leo/.ssh\\id_rsa', 'C:\\Users\\leo/.ssh\\id_dsa'], {'display_name': 'ssh://leo@strike.wu.ac.at/100', 'cmdline': ['xpra_cmd.exe', '--env=XPRA_SSH_AGENT_DISPATCH=0', 'start', 'ssh://leo@strike.wu.ac.at/100', '--start-child=qterminal', '--exit-with-children=yes', '--audio=no', '--webcam=no', '-d', 'ssh'], 'type': 'ssh', 'proxy_command': ['_proxy_start'], 'exit_ssh': True, 'remote_xpra': ['xpra', '$XDG_RUNTIME_DIR/xpra/run-xpra', '/usr/local/bin/xpra', '~/.xpra/run-xpra', 'Xpra_cmd.exe'], 'username': 'leo', 'host': 'strike.wu.ac.at', 'local': False, 'port': 22, 'display': '100', 'display_as_args': [':100', '--env=SSH_AGENT_UUID=63bf288e68403ce0c9103963e2c6c6380ff4f3c18d55ccc8227565d7cc2ed8ab', '--debug=ssh', '--audio=no', '--webcam=no', '--exit-with-children=yes', '--start-child=qterminal', '--env=XPRA_SSH_AGENT_DISPATCH=0'], 'is_paramiko': True, 'agent': True, 'ssh-agent-uuid': '63bf288e68403ce0c9103963e2c6c6380ff4f3c18d55ccc8227565d7cc2ed8ab', 'full_ssh': ['paramiko']})
[...]
2024-09-12 15:47:24,050 Authentication (password) successful!
2024-09-12 15:47:24,054 run_remote_xpra(<paramiko.Transport at 0x37c35bd0 (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>, ['_proxy_start'], ['xpra', '$XDG_RUNTIME_DIR/xpra/run-xpra', '/usr/local/bin/xpra', '~/.xpra/run-xpra', 'Xpra_cmd.exe'], '', [':100', '--env=SSH_AGENT_UUID=63bf288e68403ce0c9103963e2c6c6380ff4f3c18d55ccc8227565d7cc2ed8ab', '--debug=ssh', '--audio=no', '--webcam=no', '--exit-with-children=yes', '--start-child=qterminal', '--env=XPRA_SSH_AGENT_DISPATCH=0'], {'display_name': 'ssh://leo@strike.wu.ac.at/100', 'cmdline': ['xpra_cmd.exe', '--env=XPRA_SSH_AGENT_DISPATCH=0', 'start', 'ssh://leo@strike.wu.ac.at/100', '--start-child=qterminal', '--exit-with-children=yes', '--audio=no', '--webcam=no', '-d', 'ssh'], 'type': 'ssh', 'proxy_command': ['_proxy_start'], 'exit_ssh': True, 'remote_xpra': ['xpra', '$XDG_RUNTIME_DIR/xpra/run-xpra', '/usr/local/bin/xpra', '~/.xpra/run-xpra', 'Xpra_cmd.exe'], 'username': 'leo', 'host': 'strike.wu.ac.at', 'local': False, 'port': 22, 'display': '100', 'display_as_args': [':100', '--env=SSH_AGENT_UUID=63bf288e68403ce0c9103963e2c6c6380ff4f3c18d55ccc8227565d7cc2ed8ab', '--debug=ssh', '--audio=no', '--webcam=no', '--exit-with-children=yes', '--start-child=qterminal', '--env=XPRA_SSH_AGENT_DISPATCH=0'], 'is_paramiko': True, 'agent': True, 'ssh-agent-uuid': '63bf288e68403ce0c9103963e2c6c6380ff4f3c18d55ccc8227565d7cc2ed8ab', 'full_ssh': ['paramiko']})
[...]
2024-09-12 15:47:24,615 adding xpra_cmd='/usr/bin/xpra'
2024-09-12 15:47:24,617 cmd(['_proxy_start'], [':100', '--env=SSH_AGENT_UUID=63bf288e68403ce0c9103963e2c6c6380ff4f3c18d55ccc8227565d7cc2ed8ab', '--debug=ssh', '--audio=no', '--webcam=no', '--exit-with-children=yes', '--start-child=qterminal', '--env=XPRA_SSH_AGENT_DISPATCH=0'])="/usr/bin/xpra" "_proxy_start" ":100" "--env=SSH_AGENT_UUID=63bf288e68403ce0c9103963e2c6c6380ff4f3c18d55ccc8227565d7cc2ed8ab" "--debug=ssh" "--audio=no" "--webcam=no" "--exit-with-children=yes" "--start-child=qterminal" "--env=XPRA_SSH_AGENT_DISPATCH=0"
2024-09-12 15:47:24,617 trying to open SSH session, window-size=134217727, timeout=60
2024-09-12 15:47:24,618 [chan 4] Max packet in: 4096 bytes
2024-09-12 15:47:24,679 [chan 4] Max packet out: 32768 bytes
2024-09-12 15:47:24,684 Secsh channel 4 opened.
2024-09-12 15:47:24,686 paramiko agent_option='True'
2024-09-12 15:47:24,687 paramiko SSH agent forwarding enabled
2024-09-12 15:47:24,689 channel exec_command('"/usr/bin/xpra" "_proxy_start" ":100" "--env=SSH_AGENT_UUID=63bf288e68403ce0c9103963e2c6c6380ff4f3c18d55ccc8227565d7cc2ed8ab" "--debug=ssh" "--audio=no" "--webcam=no" "--exit-with-children=yes" "--start-child=qterminal" "--env=XPRA_SSH_AGENT_DISPATCH=0"')
2024-09-12 15:47:24,759 [chan run-xpra] Sesch channel 4 request ok
2024-09-12 15:47:24,761 exec_command sent, returning channel for service
2024-09-12 15:47:24,763 paramiko_client.connect_to({'display_name': 'ssh://leo@strike.wu.ac.at/100', 'cmdline': ['xpra_cmd.exe', '--env=XPRA_SSH_AGENT_DISPATCH=0', 'start', 'ssh://leo@strike.wu.ac.at/100', '--start-child=qterminal', '--exit-with-children=yes', '--audio=no', '--webcam=no', '-d', 'ssh'], 'type': 'ssh', 'proxy_command': ['_proxy_start'], 'exit_ssh': True, 'remote_xpra': ['xpra', '$XDG_RUNTIME_DIR/xpra/run-xpra', '/usr/local/bin/xpra', '~/.xpra/run-xpra', 'Xpra_cmd.exe'], 'username': 'leo', 'host': 'strike.wu.ac.at', 'local': False, 'port': 22, 'display': '100', 'display_as_args': [':100', '--env=SSH_AGENT_UUID=63bf288e68403ce0c9103963e2c6c6380ff4f3c18d55ccc8227565d7cc2ed8ab', '--debug=ssh', '--audio=no', '--webcam=no', '--exit-with-children=yes', '--start-child=qterminal', '--env=XPRA_SSH_AGENT_DISPATCH=0'], 'is_paramiko': True, 'agent': True, 'ssh-agent-uuid': '63bf288e68403ce0c9103963e2c6c6380ff4f3c18d55ccc8227565d7cc2ed8ab', 'full_ssh': ['paramiko']})=ssh socket: 10.0.0.116:54216 <- 137.208.89.120:22
[...]
2024-09-12 15:47:27,051 running
2024-09-12 15:47:33,465 Incoming forward agent connection
2024-09-12 15:47:33,467 [chan 5] Max packet in: 32768 bytes
2024-09-12 15:47:33,467 [chan 5] Max packet out: 16384 bytes
2024-09-12 15:47:33,468 Secsh channel 5 (auth-agent@openssh.com) opened.
Exception in thread Thread-5 (run):
Traceback (most recent call last):
  File "C:/msys64/mingw64/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
  File "C:/msys64/mingw64/lib/python3.11/site-packages/paramiko/agent.py", line 145, in run
paramiko.ssh_exception.AuthenticationException: Unable to connect to SSH agent
2024-09-12 15:47:48,217 [chan 5] EOF received (5)

On the remote side:

[leo@strike ~]$ ssh-add -l    # this one hangs
^C
[leo@strike ~]$ echo $SSH_AUTH_SOCK 
/tmp/ssh-rYK0suSUaB/agent.1683843
[leo@strike ~]$ ls -l /tmp/ssh-rYK0suSUaB/agent.1683843
srwxr-xr-x 1 leo staff 0 Sep 12 15:47 /tmp/ssh-rYK0suSUaB/agent.1683843
[leo@strike ~]$ ls -l /run/user/500/xpra/100/ssh/63bf288e68403ce0c9103963e2c6c6380ff4f3c18d55ccc8227565d7cc2ed8ab 
lrwxrwxrwx 1 leo staff 33 Sep 12 15:47 /run/user/500/xpra/100/ssh/63bf288e68403ce0c9103963e2c6c6380ff4f3c18d55ccc8227565d7cc2ed8ab -> /tmp/ssh-rYK0suSUaB/agent.1683843

[leo-b]

Btw (unrelated): The windows test build outputs this exception on every window draw: (those messages appear really often!)

2024-09-12 15:27:53,822 Error drawing on window 1
2024-09-12 15:27:53,825  using encoding jpega with options=typedict({'quality': 100, 'alpha-offset': 713926, 'flush': 0, 'window-size': (1657, 1091), 'encoding': 'jpega'})
Traceback (most recent call last):
  File "E:/xpra/xpra/client/mixins/windows.py", line 1690, in _do_draw
  File "E:/xpra/xpra/client/gui/window_base.py", line 841, in draw_region
  File "E:/xpra/xpra/client/gui/window_backing_base.py", line 903, in draw_region
  File "E:/xpra/xpra/client/gui/window_backing_base.py", line 482, in paint_jpega
  File "E:/xpra/xpra/client/gui/window_backing_base.py", line 508, in do_paint_jpeg
  File "E:/xpra/xpra/client/gui/window_backing_base.py", line 490, in nv_decode
  File "E:/xpra/xpra/client/gui/window_backing_base.py", line 391, in assign_cuda_context
ModuleNotFoundError: No module named 'xpra.codecs.nvidia'
2024-09-12 15:27:53,830 Error processing draw packet
Traceback (most recent call last):
  File "E:/xpra/xpra/client/mixins/windows.py", line 1617, in _draw_thread_loop
  File "E:/xpra/xpra/client/mixins/windows.py", line 1690, in _do_draw
  File "E:/xpra/xpra/client/gui/window_base.py", line 841, in draw_region
  File "E:/xpra/xpra/client/gui/window_backing_base.py", line 903, in draw_region
  File "E:/xpra/xpra/client/gui/window_backing_base.py", line 482, in paint_jpega
  File "E:/xpra/xpra/client/gui/window_backing_base.py", line 508, in do_paint_jpeg
  File "E:/xpra/xpra/client/gui/window_backing_base.py", line 490, in nv_decode
  File "E:/xpra/xpra/client/gui/window_backing_base.py", line 391, in assign_cuda_context
ModuleNotFoundError: No module named 'xpra.codecs.nvidia'

[totaam]

in assign_cuda_context
ModuleNotFoundError: No module named 'xpra.codecs.nvidia'

Fixed in 4ead552

check if a local agent is actually available before enabling agent forwarding on the server side.

Actually, this is wrong: we can just always enable agent forwarding, but the server should make this point to the default socket path when the client doesn't support agent forwarding.
We just need to make sure that we detect when the client doesn't support it.

.\Python_exec_cmd.exe "from paramiko.win_openssh import can_talk_to_agent, OpenSSHAgentConnection; print(can_talk_to_agent()); aconn = OpenSSHAgentConnection(); print(aconn)"
True
<paramiko.win_openssh.OpenSSHAgentConnection object at 0x000001faf0b7a590>

Ouch.

Does openssh agent forwarding acually work in your windows test environment?

I don't use it myself, but I'm pretty sure that I had tested with pageant when the ssh agent feature was released (#2303)