Investigate using local server on remoteListenOnSocket #6086
Comments
jasperhyp
changed the title
RemoteCommand assigned but not run when connecting to a two-factor authentication two-layer HPC serverRemoteCommand given but not passed when connecting to a two-factor authentication two-layer HPC server
|
Hm and when you connect to this same host through PowerShell do you see your command in RemoteCommand being passed as expected? |
Yes, when I use PowerShell with
Never mind the error message. It just showed the remote command has been passed. |
|
Well... maybe this error message is a problem even if the However, it stays true that
|
|
Oh I missed this earlier. I see in your logs you have You need to have |
Oh looks like this parameter is set to "remote.SSH.useLocalServer": true,
"remote.SSH.enableRemoteCommand": true,
"terminal.integrated.inheritEnv": false,
"remote.SSH.remoteServerListenOnSocket": true,
"window.zoomLevel": 1,
|
|
Try setting "remote.SSH.remoteServerListenOnSocket" to false. They can't work in conjunction. |
Thank you, I actually tried set |
|
Oh I see, hm that's unfortunate. Right now we can only support RemoteCommand when local server is enabled but for some machines that method doesn't work and |
Got it, thanks! I tried again today and it just doesn't work. Maybe I'll just use VS Code to view the code but not run anything. |
|
Sorry about that! Do you need RemoteCommand to run/compile your code? |
|
This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines. Happy Coding! |
Thanks for the follow-up. Have been out of work for a few days. I am mainly using Python so usually compilation is not needed. I just sometimes want to use |
Is this done locally on your machine or on a remote machine? I believe everything you do locally should be able to be done in your remote if you wish it have it running there. Maybe just not as a RemoteCommand that runs on connection to the remote. |
I meant to say "open Jupyter Notebook on the remote machine and port forward to local machine with browser". I cannot do this with Remote SSH because I need to run |
|
Ahh I see. Thanks for explaining! Sorry about that! I'll explore alternatives to localServer with remoteListenOnSocket is needed. |
tanhakabir
changed the title
RemoteCommand given but not passed when connecting to a two-factor authentication two-layer HPC server
tanhakabir
added
debt
and removed
info-needed
|
I face the same issue. I need to use slurm for ressource allocation because the login nodes are slow (they are login nodes after all) and cannot handle vscode server properly (ie most features are significanly slowed-down). Standard ssh with port forwarding directly to the compute node doesn't works (ssh -L) but I can reverse forward from the compute to the login node (e.g. I can forward tensorboard running on the compute node) my Vscode SSH extension logs |
@tanhakabir did you had a chance to look at it and pinpoint the issue? |
|
Greeting!
Can I somehow help with this Issue, maybe by providing Logfiles or similar? |
|
Bumping this up |
|
Hello! Could the people still interest in the thread outline their exact use case for this need. This would be a complicated fix so we are trying to determine if there are any solutions this problem that might be more successful and quicker. Thanks! |
|
Hi @eleanorjboyd , the original post describes it best (first post in this issue) The overall goal is to run the vscode-server on a compute node, e.g. in a computer cluster.
This is crucial because in many compute clusters it is not allowed to run something like vscode-server on the head node itself. hope it helps. |
|
Hi @eleanorjboyd , The proposed solution is to run code-server on a remote machine we need to connect. In our case, it means scheduling a job via cluster management software (e.g. slurm) to run code-server binary on a computational node, and then get the stdout from code-server with connection details (pin code). Then use VSCode "Remote - Tunnels" plugin to connect to that code-server instance. Downside of the proposed solution is: It uses Microsoft infrastructure to tunnel the traffic between Remote Tunnel plugin and code-server instance. If everything is in the same local net, this shouldn't be needed. With "Remote - SSH" plugin the issue is the following. It allows to customize RemoteCommand, and we can use a customized RemoteCommand to run 'Code' on a computational node (instead of the node we established ssh connection to) and also to set up our own transparent tunneling between the computational node and the slurm control node, so that the 'Code' running on a computational node could be contacted via socket on a control node. However, we don't have a way for the Visual Studio Code (which is being run on a local PC) to use that particular socket instead of the one it thinks it have to use. Hope I described the use case. If it's still not very clear, feel free to engage someone technical from your company (I'm sorry if it sounds rude, I just don't want this comment to be sort of useless or lost, and don't know how to express it better in words). |
|
Thanks for the explanations but I still don't fully understand the situation. It sounds like this is issue essentially about using Besides that, it sounds like what @ekungurov is saying is that using RemoteCommand for the slurm setup is not sufficient, and I think #1722 essentially covers that issue. I don't really understand it though-
With the RemoteCommand, wouldn't that mean that Remote-SSH connects and runs its setup script in the context of the compute node, and forwards our server's port/socket from the compute node to the local machine? Why do you say you have to set up your own forwarding from the login node? |
|
@roblourens on HPC users usually don't ssh directly to a compute node to launch a task. Instead, they ssh to a control node (login node) and then submit tasks to the cluster. Cluster mgmt software automatically choose a compute node (worker node) from the pool of compute nodes which currently have requested resources available. That's why in #1722 a solution from @samfux84 (edited: this one #1722 (comment)) use ssh -L for tunneling, and solution from @simonbyrne (this one #1722 (comment)) uses nc I believe for the same purpose. |
I understand that. But I guess that even if our setup script runs in the compute node context, ssh still connected to the login node and so the forwarding that ssh sets up only goes as far as the login node, that makes sense. |
@roblourens Please see my discussion with @tanhakabir at the beginning of this post. Specifically this one. I tried combinations of true/false for those settings but there seems not one combination that would allow |
Yes, correct.
samfux84 solution is almost completely semi-manual/manual, and it establish ssh tunnel between "localhost" and the compute node (where https://github.com/coder/code-server aka "VS Code in the browser" is being run). It is supposed that end user will download / copy the generated bash script, and run it on his own local Linux/MacOS/WSL workstation, so the "localhost" is a workstation in this case. simonbyrne solution, on the other hand, runs yet another copy of sshd as a task (as a user process) on a compute node, and then 'nc' the traffic to that instance of sshd. My question... I mean, I probably have a question... Is it possible to apply custom RemoteCommand to login node, but once it applied - to connect to a compute node : either directly to the compute node and probably manually (directly - without using ssh/nc tunneling and without entering XXXX-XXXX pin code on https://github.com/login/device thus utilizing some MS Azure tunneling technology), OR if that's not possible, maybe via ssh/nc tunneling but that tunneling being set up automatically (via custom RemoteCommand or other means). |
Thanks- that configuration should work, since those are the defaults for those settings (on mac/linux, and should generally work on windows). If you are still seeing that, I can help troubleshoot if you share the full log from the Remote-SSH output channel. However one possible explanation for that error message is that TCP port forwarding is disabled on your ssh server, if that's the case, you might need to talk to your system administrator, or, you might be stuck with a socket.
Great question! I really don't know 😁. I suspect that the answer is no, and that the only solution is for a complex custom workaround of the sort that has been mentioned already. |
Thanks, could you specify which settings? |
|
|
Hi @ekungurov, I scripted our solution that we are using for the users of our HPC cluster, so for the end user it is fully automated (including the setup of the SSH tunnel and opening the local browser with the URL from the code-server): https://gitlab.ethz.ch/sfux/VSCode_remote_HPC The only prerequisite for the end user is to have SSH keys for password-less access to the cluster (as all commands are run through ssh, because the script runs on the local computer of the user). The users can download our script and run it on their local computer specifying what resources they want for the batch job and then wait until it has started. For instance, if I run the command ./start_vscode.sh -u sfux -b SLURM -n 4 -W 04:00 -m 2048 on my local computer, then it will login to the cluster as user sfux, submit a batch job via Slurm, requesting 4 cores, 4 hours of runtime and 2 GB of memory per core, setup the required SSH tunnel and start the local browser with the URL provided by the code-server. We tested the script with Linux, macOS and Windows (using git-bash) and it works. We plan to replace code-server in the future with vscode-server once there is a public release. This kind of setup can also be used for other use-cases like jupyter scripts (in case there is no jupyter hub available): https://gitlab.ethz.ch/sfux/Jupyter-on-Euler-or-Leonhard-Open |
It seems not working though. I can connect to the login node but no compute node is requested, meaning that the command is not run. My I used a very simple setup in my config: Silly question: How to get logs when logging with remote-ssh? It looks like it's now not creating logs by default if the connection is successful. |
|
View > Output > Remote-SSH in the dropdown I'd rather discuss the general "how to connect to slurm" issues in #1722 since it seems that's what that issue is generally about. This one can be "using remoteListenOnSocket with useLocalServer", but I'm not sure we'll be able to do anything about that. |
|
I also noticed the error "Cannot use RemoteCommand if you have this remote in your "remote.SSH.remotePlatform" setting", so I followed the solution here to change the Config: Error log below: Log |
|
|
|
Doesn't work, same error log. |
|
As of June 21, creating a direct connection to a Slurm job is still faulty. This would be AWESOME. |
|
Followed up all comments in 2024! And agree for your point @roblourens
I think main issue we discuss here is that,
Q1. Do you think above setting is completely unavailable by design? IMHO if there is no good way for both question we'd better close this ticket and keep discussing in main issue for HPC usage demands #1722 |
I was trying to get onto the second layer of an HPC after SSH-ing with some Slurm command (srun), as was generally described in #1722. I tried the newest
RemoteCommandfrom the Pre-release version of Remote SSH (following instructions in this issue), but it seems not passing theRemoteCommandwhen connecting.Logs
Steps to Reproduce:
Add
"remote.SSH.enableRemoteCommand": true, "remote.SSH.useLocalServer": true, "remote.SSH.remoteServerListenOnSocket": truetosettings.json."remote.SSH.remoteServerListenOnSocket": trueis a must because otherwise would result inFailed to set up socket for dynamic port forward to remote port 45824: Socket closed.SSH to
some.host.namewith the following ssh config:Does this issue occur when you try this locally?: It's a connection issue.
The text was updated successfully, but these errors were encountered: