[Remote-SSH Bug]: New "Use Exec Server" option binds to localhost only

[jtfrey]

Is there an existing issue for this bug?

• I have searched the existing issues

Required Troubleshooting Steps

• I have followed these troubleshooting steps
• I have tried both values of the remote.SSH.useLocalServer setting

Connect Locally

It connects successfully

->

No response

Expected Behavior

The Bash/node.js vscode-server infrastructure opens and binds its TCP listener to all interfaces. The exec server only binds its TCP listener to the loopback interface. Visibility of the listener is radically different between the two and (seems to be) non-configurable in the exec server.

This becomes a major issue on HPC clusters, for example, where users need to access GPU hardware via a job scheduler and we do not want them to be consuming limited resources on login nodes. I've already written and deployed a proxy (which is executed as the Remote-SSH "Remote Command" on the connection) which schedules the remote session via the job scheduler. It works fine with the Bash/node.js infrastructure because the proxy can connect to the listener port on the compute node. It does not work with the exec server because a port bound solely to loopback isn't reachable from another node in the cluster.

If the default interface binding for the exec server MUST be "localhost," can the command-shell subcommand at least have an option to override that and bind to * or a specific IP?

Steps To Reproduce

No response

Remote-SSH Log

Remote-SSH Log

Our proxy receives the

=listeningOn==127.0.0.1:45678==

line from the remote CLI server and opens a listening port on the login node whose port# is substituted and returned to the vscode client. When the vscode client connects, the proxy attempts to connect to 45678 on the compute node running the interactive job — but since it's bound to loopback only, that fails.

This worked with the Bash/node.js infrastructure because the listening port was bound to all interfaces and simply responded with

=listeningOn==45678==

for which the proxy's listening port on the login node was substituted in the response to the vscode client. The client connects through the tunnel, the proxy connects to the port on the compute node, and communication is established.

Anything else?

No response

[connor4312]

The Bash/node.js vscode-server infrastructure opens and binds its TCP listener to all interfaces

From looking at the code, when we start the server in the previous/traditional metho we always passed --host=127.0.0.1 which should not have bound on all interfaces. While the output is slightly different the interface being listened on should be the same in both cases. Could you validate which process is doing the 'all interfaces' listening via lsof -nP -iTCP -sTCP:LISTEN and then ps -p $PID -o command?

[jtfrey]

[root@r00n56 ~]# ps -Hfu frey
UID        PID  PPID  C STIME TTY          TIME CMD
frey     51398 51388  0 16:07 ?        00:00:00 /bin/bash -l
frey     51443 51398  0 16:07 ?        00:00:00   bash
frey     51488 51443  0 16:07 ?        00:00:00     sh /home/1001/.vscode-server/bin/863d2581ecda6849923a2118d93a088b0745d9d6/bin/code-server --start-server --host=0.0.0.0 --accept-server-license-terms --enable-remote-auto-shutdown --port=0 --telemetry-level all --connection-token-file /home/1001/.vscode-server/.863d2581ecda6849923a2118d93a088b0745d9d6.token
frey     52184 51488  2 16:07 ?        00:00:02       /home/1001/.vscode-server/bin/863d2581ecda6849923a2118d93a088b0745d9d6/node /home/1001/.vscode-server/bin/863d2581ecda6849923a2118d93a088b0745d9d6/out/server-main.js --start-server --host=0.0.0.0 --accept-server-license-terms --enable-remote-auto-shutdown --port=0 --telemetry-level all --connection-token-file /home/1001/.vscode-server/.863d2581ecda6849923a2118d93a088b0745d9d6.token
frey     52321 52184  1 16:07 ?        00:00:01         /home/1001/.vscode-server/bin/863d2581ecda6849923a2118d93a088b0745d9d6/node --dns-result-order=ipv4first /home/1001/.vscode-server/bin/863d2581ecda6849923a2118d93a088b0745d9d6/out/bootstrap-fork --type=extensionHost --transformURIs --useHostProxy=false
frey     52333 52184  0 16:07 ?        00:00:00         /home/1001/.vscode-server/bin/863d2581ecda6849923a2118d93a088b0745d9d6/node /home/1001/.vscode-server/bin/863d2581ecda6849923a2118d93a088b0745d9d6/out/bootstrap-fork --type=ptyHost --logsPath /home/1001/.vscode-server/data/logs/20240405T160709
frey     52441 52333  0 16:07 pts/5    00:00:00           /bin/bash --init-file /home/1001/.vscode-server/bin/863d2581ecda6849923a2118d93a088b0745d9d6/out/vs/workbench/contrib/terminal/browser/media/shellIntegration-bash.sh

[root@r00n56 ~]# lsof -nP -iTCP -sTCP:LISTEN | grep node
node      52184    frey   24u  IPv4 46665696      0t0  TCP *:38987 (LISTEN)

The vscode-shell-proxy that we wrote to push VSCode onto compute nodes via the Slurm job scheduler intercepts the commands that the user's local VSCode is sending and catches the --host=127.0.0.1 and is able to substitute --host=0.0.0.0, which is why node.js is listening on all ports. If the new binary remote server preserved that option — rather than only listening on loopback and not allowing override — our proxy would still work properly.

[connor4312]

Gotcha. I'm not especially keen to do this. It worked for you before because of the way the arguments were laid out, but this was pretty much happenstance, and there's no (other) reason that command-shell should be listening on interfaces other than localhost. To the contrary, it's a high level of privilege that, while guarded by an auth token, should have as little access as possible.

I would recommend setting up something on your system to handle forwarding between interfaces. If you're on Linux, I think iptables can do this for you: https://superuser.com/questions/684275/how-to-forward-packets-between-two-interfaces. Or if you happen to have Node.js on the box, a small program net.listen(1234, '0.0.0.0', l => net.connect(1235, '127.0.0.1', l2 => { l1.pipe(l2), l2.pipe(l1) }) would do the trick.

[jtfrey]

I think the point is being missed here: vscode's remote-ssh extension is not designed to work in any circumstance other than the user's wanting the remote server component to run directly on the system into which s/he has ssh'ed. That 100% is not compatible with HPC systems, where access to pertinent hardware is gated through a head/login node and a job scheduler. The reason we've had to go to the effort to proxy remote-ssh access is because our users were leaving behind hundreds of orphaned process trees on the login nodes. We'd rather not ban its use entirely, but that's where things are headed.

Yes, it is less secure to have the remote server listening on any interface other than loopback; that's true for any software ever written that opens network sockets. Apache is less secure the moment I plumb it to an external interface, but it serves no purpose to me otherwise. You're assuming no utility to something that has utility when the user/sysadmin accepts the risk and responsibility associated with listening on a routed interface.

Your suggestion re: iptables and port forwarding/proxying just complicates the system further and accomplishes the same thing that would be accomplished far more simply and efficiently if the CLI remote server accepted and honored --host=<ip-addr>. Again, the use case is HPC systems where the placement of the CLI remote server is ephemeral and random, so static modifications to iptables to implement forwarding isn't possible.

So the question is: why not implement the same --host=<ip-addr> option present in the node.js variant of the remote server, and just ensure you document Microsoft's preference for sticking to the default loopback and why routed interfaces are more dangerous? Leave it up to the end users and sysadmins trying to integrate vscode remote-ssh into diverse systems to accept the risk or not when using something other than the default.

[connor4312]

Thanks for the info. I think the preferable approach to take in this case is actually having a proper remote.SSH.advanced.bindHost setting. Which would avoid the need for a hack for you and be actual contract to avoid it breaking in the future :)

Will be on Monday's VS Code Insiders https://code.visualstudio.com/insiders/ and the prerelease version of the Remote - SSH extension. Please let me know if it works for you. You can also work around it by reverting to the 'old mode' by setting remote.SSH.useExecServer: false, although that will go away in the next release.

I don't plan to promote this at a patch to this month's release at this time.

[jtfrey]

Thanks for the info. I think the preferable approach to take in this case is actually having a proper remote.SSH.advanced.bindHost setting. Which would avoid the need for a hack for you and be actual contract to avoid it breaking in the future :)

Offering a singular global option to determine that behavior is not a good implementation. If I need just 1 of the N ssh hosts I've configured to not bind to loopback, I have to have all N do that. That's insecurity for the sake of simplicity.

The same issue of configuration granularity exists with most other remote-ssh options. The "Use Exec Server" option that started this issue is implemented as a global binary choice, for example, where per-profile configurability (e.g. override the default in an advanced configuration mode) would be useful to the end user. Effectively all of the options that are marked with "(Applies to all profiles)" would be more useful that way, though certainly less simple and straightforward to implement on the client side. Your suggestion of remote.SSH.advanced.bindHost is just more fraught because it has security implications if it "(Applies to all profiles)."

[connor4312]

Fair, updated it to a map