[Remote-SSH Bug]: Cannot reconnect to ssh agent in the same terminal after reload window

[tekumara]

Is there an existing issue for this bug?

• I have searched the existing issues

Debugging Steps

• I have followed these debugging steps
• I have tried both values of the remote.SSH.useLocalServer setting
• My SSH command works from a local terminal
• I can connect using the Remote-SSH extension on VS Code when all other VS Code extensions are disabled

->

remote.SSH.useLocalServer=false does not have this issue.

Versions

• VS Code Version: 1.74.3
• Remote-SSH Version: v0.94.0
• Remote OS Version: ubuntu 20.04

Expected Behavior

Cannot reconnect to ssh agent in the same terminal after reload window.

❯ ssh-add -L
Error connecting to agent: No such file or directory

Steps To Reproduce

1. Set remote.SSH.useLocalServer=true
2. Connect to an SSH host
3. Open a terminal and check SSH_AUTH_SOCK:

    ❯ ls -L $SSH_AUTH_SOCK
   /run/user/1001/vscode-ssh-auth-sock-334051596
   

4. Reload Window
5. In the same terminal, check SSH_AUTH_SOCK:

   ❯ ls -L $SSH_AUTH_SOCK
   ls: cannot access '/run/user/1001/vscode-ssh-auth-sock-334051596': No such file or directory
   

6. Because SSH_AUTH_SOCK is pointing at a blackhole, the ssh agent will fail to find any keys:

   ❯ ssh-add -L
   Error connecting to agent: No such file or directory
   

7. If you open a new terminal you'll get a new SSH_AUTH_SOCK value which works:

   ❯ ls -L $SSH_AUTH_SOCK        
   /run/user/1001/vscode-ssh-auth-sock-961955974
   

Remote-SSH Log

Remote-SSH Log when reloading the window

[12:37:07.828] Log Level: 2
[12:37:07.831] remote-ssh@0.94.0
[12:37:07.831] darwin arm64
[12:37:07.834] SSH Resolver called for "ssh-remote+server-compute", attempt 1
[12:37:07.835] "remote.SSH.useLocalServer": true
[12:37:07.835] "remote.SSH.path": undefined
[12:37:07.835] "remote.SSH.configFile": undefined
[12:37:07.835] "remote.SSH.useFlock": true
[12:37:07.835] "remote.SSH.lockfilesInTmp": false
[12:37:07.835] "remote.SSH.localServerDownload": auto
[12:37:07.835] "remote.SSH.remoteServerListenOnSocket": false
[12:37:07.835] "remote.SSH.showLoginTerminal": false
[12:37:07.835] "remote.SSH.defaultExtensions": ["ms-python.python","ms-python.vscode-pylance"]
[12:37:07.835] "remote.SSH.loglevel": 2
[12:37:07.836] "remote.SSH.maxReconnectionAttempts": 2
[12:37:07.836] "remote.SSH.enableDynamicForwarding": true
[12:37:07.836] "remote.SSH.enableRemoteCommand": false
[12:37:07.836] "remote.SSH.serverPickPortsFromRange": {}
[12:37:07.836] "remote.SSH.serverInstallPath": {}
[12:37:07.840] SSH Resolver called for host: server-compute
[12:37:07.840] Setting up SSH remote "server-compute"
[12:37:07.841] Acquiring local install lock: /var/folders/5y/hhvf46cx393_j75q4gm7tvzw0000gq/T/vscode-remote-ssh-0efbde2a-install.lock
[12:37:07.843] Looking for existing server data file at /Users/tekumara/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-0efbde2a-97dec172d3256f8ca4bfb2143f3f76b503ca0534-0.94.0/data.json
[12:37:07.843] Using commit id "97dec172d3256f8ca4bfb2143f3f76b503ca0534" and quality "stable" for server
[12:37:07.845] Install and start server if needed
[12:37:07.847] PATH: /Users/tekumara/.cargo/bin:/Users/tekumara/Library/Caches/fnm_multishells/61649_1673852755642/bin:/Users/tekumara/.pyenv/shims:/Users/tekumara/.pyenv/shims:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/tekumara/.local/bin:/usr/local/opt/go/libexec/bin:/Users/tekumara/go/bin:/opt/homebrew/opt/fzf/bin
[12:37:07.847] Checking ssh with "ssh -V"
[12:37:07.856] > OpenSSH_9.0p1, LibreSSL 3.3.6

[12:37:07.858] askpass server listening on /var/folders/5y/hhvf46cx393_j75q4gm7tvzw0000gq/T/vscode-ssh-askpass-992ad2b5af562fbb2de23937eec5a2d17c93a4f4.sock
[12:37:07.858] Spawning local server with {"serverId":1,"ipcHandlePath":"/var/folders/5y/hhvf46cx393_j75q4gm7tvzw0000gq/T/vscode-ssh-askpass-a27aaebd0e8c48028140920c636ee60413113036.sock","sshCommand":"ssh","sshArgs":["-v","-T","-D","64266","-o","ConnectTimeout=60","server-compute"],"serverDataFolderName":".vscode-server","dataFilePath":"/Users/tekumara/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-0efbde2a-97dec172d3256f8ca4bfb2143f3f76b503ca0534-0.94.0/data.json"}
[12:37:07.858] Local server env: {"SSH_AUTH_SOCK":"/private/tmp/com.apple.launchd.QSP5RMozFZ/Listeners","SHELL":"/bin/zsh","DISPLAY":"1","ELECTRON_RUN_AS_NODE":"1","SSH_ASKPASS":"/Users/tekumara/.vscode/extensions/ms-vscode-remote.remote-ssh-0.94.0/out/local-server/askpass.sh","VSCODE_SSH_ASKPASS_NODE":"/Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper.app/Contents/MacOS/Code Helper","VSCODE_SSH_ASKPASS_EXTRA_ARGS":"--ms-enable-electron-run-as-node","VSCODE_SSH_ASKPASS_MAIN":"/Users/tekumara/.vscode/extensions/ms-vscode-remote.remote-ssh-0.94.0/out/askpass-main.js","VSCODE_SSH_ASKPASS_HANDLE":"/var/folders/5y/hhvf46cx393_j75q4gm7tvzw0000gq/T/vscode-ssh-askpass-992ad2b5af562fbb2de23937eec5a2d17c93a4f4.sock"}
[12:37:07.859] Spawned 49701
[12:37:07.926] > local-server-1> Running ssh connection command: "-v -T -D 64266 -o ConnectTimeout=60 server-compute"
[12:37:07.928] > local-server-1> Spawned ssh, pid=49710
[12:37:07.934] stderr> OpenSSH_9.0p1, LibreSSL 3.3.6
[12:37:10.632] stderr> Connecting to instance i-0514fc429b6c3e1f0
[12:37:14.939] stderr> debug1: Server host key: ssh-ed25519 SHA256:dLWdQIJwrN1tOBe1yuRTWtBohf4QnfW2QrnQyvwH++g
[12:37:16.715] stderr> Authenticated to server-compute (via proxy) using "publickey".
[12:37:17.645] > 
> 
>   System information as of Tue Jan 17 01:37:51 UTC 2023
> 
> 
> 111 updates can be applied immediately.
> 2 of these updates are standard security updates.
> To see these additional updates run: apt list --upgradable
> 
> 
> The list of available updates is more than a week old.
> To check for new updates run: sudo apt update
> New release '22.04.1 LTS' available.
> Run 'do-release-upgrade' to upgrade to it.
> 
> 
> *** System restart required ***
[12:37:17.888] > ready: e2fb0693676b
[12:37:18.139] > Linux 5.15.0-1022-aws #26~20.04.1-Ubuntu SMP Sat Oct 15 03:22:07 UTC 2022
[12:37:18.139] Platform: linux
[12:37:18.878] > /usr/bin/zsh
[12:37:18.879] Shell: zsh
[12:37:18.879] Creating bash subshell inside "zsh"
[12:37:19.132] > bash version:  5.0.17(1)-release
[12:37:19.353] > bash version:  5.0.17(1)-release
[12:37:19.353] > bash version:  5.0.17(1)-release
[12:37:19.357] > bash version:  5.0.17(1)-release
> bash version:  5.0.17(1)-release
[12:37:19.616] > e2fb0693676b: running
[12:37:19.616] > Acquiring lock on /home/compute/.vscode-server/bin/97dec172d3256f8ca4bfb2143f3f76b503ca0534/vscode-remote-lock.compute.97dec172d3256f8ca4bfb2143f3f76b503ca0534
> Found existing installation at /home/compute/.vscode-server/bin/97dec172d3256f8ca4bfb2143f3f76b503ca0534...
> Checking /home/compute/.vscode-server/.97dec172d3256f8ca4bfb2143f3f76b503ca0534.log and /home/compute/.vscode-server/.97dec172d3256f8ca4bfb2143f3f76b503ca0534.pid for a running server
> Looking for server with pid: 4017170
> Found running server...
> Installing extensions...
[12:37:19.907] > Installing extensions...
[12:37:20.201] > Extension 'ms-python.python' v2022.20.2 is already installed. Use '--force' option to update to latest version or provide '@<version>' to install a specific version, for example: 'ms-python.python@1.2.3'.
[12:37:20.203] > Extension 'ms-python.vscode-pylance' v2023.1.20 is already installed. Use '--force' option to update to latest version or provide '@<version>' to install a specific version, for example: 'ms-python.vscode-pylance@1.2.3'.
[12:37:20.213] >  
> *
> * Visual Studio Code Server
> *
[12:37:20.214] > * By using the software, you agree to
> * the Visual Studio Code Server License Terms (https://aka.ms/vscode-server-license) and
> * the Microsoft Privacy Statement (https://privacy.microsoft.com/en-US/privacystatement).
> *
>  
[12:37:20.225] > Checking server status on port 34323 with wget
[12:37:20.437] > e2fb0693676b: start
> SSH_AUTH_SOCK==/tmp/ssh-UyvOFsodIW/agent.4018972==
> DISPLAY====
> webUiAccessToken====
> listeningOn==34323==
> osReleaseId==ubuntu==
> arch==x86_64==
> tmpDir==/run/user/1001==
> platform==linux==
> unpackResult====
> didLocalDownload==0==
> downloadTime====
> installTime====
[12:37:20.437] > extInstallTime==731==
> serverStartTime====
> connectionToken==1a111aaa-11a1-1a11-11aa-1aa1a1a1aa11==
> e2fb0693676b: end
[12:37:20.437] Received install output: 
SSH_AUTH_SOCK==/tmp/ssh-UyvOFsodIW/agent.4018972==
DISPLAY====
webUiAccessToken====
listeningOn==34323==
osReleaseId==ubuntu==
arch==x86_64==
tmpDir==/run/user/1001==
platform==linux==
unpackResult====
didLocalDownload==0==
downloadTime====
installTime====extInstallTime==731==
serverStartTime====
connectionToken==1a111aaa-11a1-1a11-11aa-1aa1a1a1aa11==

[12:37:20.438] Remote server is listening on 34323
[12:37:20.438] Parsed server configuration: {"serverConfiguration":{"remoteListeningOn":{"port":34323},"osReleaseId":"ubuntu","arch":"x86_64","webUiAccessToken":"","sshAuthSock":"/tmp/ssh-UyvOFsodIW/agent.4018972","display":"","tmpDir":"/run/user/1001","platform":"linux","connectionToken":"1a111aaa-11a1-1a11-11aa-1aa1a1a1aa11"},"extInstallTime":731,"installUnpackCode":""}
[12:37:20.439] Persisting server connection details to /Users/tekumara/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-0efbde2a-97dec172d3256f8ca4bfb2143f3f76b503ca0534-0.94.0/data.json
[12:37:20.442] Starting forwarding server. localPort 64288 -> socksPort 64266 -> remotePort 34323
[12:37:20.443] Forwarding server listening on 64288
[12:37:20.443] Waiting for ssh tunnel to be ready
[12:37:20.443] Tunneled 34323 to local port 64288
[12:37:20.443] Resolved "ssh-remote+server-compute" to "127.0.0.1:64288"
[12:37:20.444] Updating $SSH_AUTH_SOCK: ln -f -s "/tmp/ssh-UyvOFsodIW/agent.4018972" "/run/user/1001/vscode-ssh-auth-sock-961955974"
[12:37:20.444] Using cwd: vscode-remote://ssh-remote%2Bserver-compute/
[12:37:20.444] Remote extension host environment: {"SSH_AUTH_SOCK":"/run/user/1001/vscode-ssh-auth-sock-961955974"}
[12:37:20.446] [Forwarding server 64288] Got connection 0
[12:37:20.452] ------




[12:37:20.470] [Forwarding server 64288] Got connection 1
[12:37:20.994] [Forwarding server 64288] Got connection 2
[12:37:24.869] [Forwarding server 64288] Got connection 3
[12:37:24.869] [Forwarding server 64288] Got connection 4
[12:37:28.331] "Update SSH_AUTH_SOCK" terminal command done

Anything else?

This issue also occurs when the ssh connection has been lost, but vscode hasn't quit, and you reconnect by reloading the window.

[eleanorjboyd]

sorry for the delay- I'm going to attempt to repro. Thanks

[eleanorjboyd]

Hello! I think this is related to an ongoing issue we have with useLocalServer where it is not actually reusing the local server correctly. We are aware of this issue and plan to get it fixed soon so hopefully this will be the fix. You can follow along here #6227 and I will keep this open while the issue is not yet addressed.

[screencomuser]

Anyone coming here wanting a quick fix, I've done the following which works on my machine (vscode on mac, running remote on linux)

I've added the following line to my remote ~/.zshrc

[[ -S $(readlink $SSH_AUTH_SOCK) ]] || ln -sf $(ls -1rtd /tmp/ssh-*/* | tail -1 | xargs) $SSH_AUTH_SOCK

What this does is, check if the symlink in $SSH_AUTH_SOCK is pointing to a socket, if not it takes the last created agent socket and links to it.

Validate with ssh-add -L

Just my 2c

Edit: I noticed that on a regular login $SSH_AUTH_SOCK points to a socket and not a symlink. Therefore the following works better. Also, this takes into account that $SSH_AUTH_SOCK just points to a file that's just not there

# vscode SSH_AUTH_SOCK fix
(test -L $SSH_AUTH_SOCK && (
    test -S $(readlink $SSH_AUTH_SOCK) && \
        ln -sf $(ls -1rtd /tmp/ssh-*/* | tail -1 | xargs) $SSH_AUTH_SOCK
)) || ln -sf $(ls -1rtd /tmp/ssh-*/* | tail -1 | xargs) $SSH_AUTH_SOCK

[apfreelancing]

Anyone coming here wanting a quick fix, I've done the following which works on my machine (vscode on mac, running remote on linux)

I've added the following line to my remote ~/.zshrc

[[ -S $(readlink $SSH_AUTH_SOCK) ]] || ln -sf $(ls -1rtd /tmp/ssh-*/* | tail -1 | xargs) $SSH_AUTH_SOCK

What this does is, check if the symlink in $SSH_AUTH_SOCK is pointing to a socket, if not it takes the last created agent socket and links to it.

Validate with ssh-add -L

Just my 2c

Edit: I noticed that on a regular login $SSH_AUTH_SOCK points to a socket and not a symlink. Therefore the following works better. Also, this takes into account that $SSH_AUTH_SOCK just points to a file that's just not there

# vscode SSH_AUTH_SOCK fix
(test -L $SSH_AUTH_SOCK && (
    test -S $(readlink $SSH_AUTH_SOCK) && \
        ln -sf $(ls -1rtd /tmp/ssh-*/* | tail -1 | xargs) $SSH_AUTH_SOCK
)) || ln -sf $(ls -1rtd /tmp/ssh-*/* | tail -1 | xargs) $SSH_AUTH_SOCK

i tried the quickfix, but this didn't work out of the box, because the symlink could not be created (i didnt look into the matter deeper, tho)

instead, i added to my .bashrc the following:
[[ -S $(readlink $SSH_AUTH_SOCK) ]] || export SSH_AUTH_SOCK=$(ls -1rtd /tmp/ssh-*/* | tail -1 | xargs)
this basically does the same, except it does not create a symlink to $SSH_AUTH_SOCK, but instead simply overwrites the environment variable with the most recent created SSH socket

[thomasmebes]

My solution

#!/bin/bash

SSH_EXISTING_SOCK=$(find /tmp -maxdepth 1 -type d -name 'ssh-*' -user $USER | xargs -I {} find {} -type s)

if [ -n "$SSH_AUTH_SOCK" ]; then
    mkdir -p $(dirname $SSH_AUTH_SOCK)
    ln -sf $SSH_EXISTING_SOCK $SSH_AUTH_SOCK
fi