Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature request: Added option to allow users to specify the remote IP address when forwarding server #10627

Open
yanghaku opened this issue Jan 13, 2025 · 3 comments
Open

Feature request: Added option to allow users to specify the remote IP address when forwarding server #10627

yanghaku opened this issue Jan 13, 2025 · 3 comments
Assignees
@roblourens @joshspicer
Labels
feature-request Request for new features or functionality ssh Issue in vscode-remote SSH
Milestone
Backlog Candidates

Comments

@yanghaku
Copy link

I want to run VS Code Server in the user containers and access the container directly through sshd from a remote host. The sshd configuration of the remote host is as follows:

Match Group lxc-users
    AllowTcpForwarding yes
    ForceCommand systemd-run --user --scope --no-ask-password -q -p Delegate=yes /usr/bin/lxc-attach -n $USER --clear-env --keep-var=TERM -- sudo -i -u $USER $SSH_ORIGINAL_COMMAND

The network architecture is as follows:

Image

The VS Code Server and sshd are located in two different network namespaces, which prevents them from using the default loopback network interface for communication. I noticed that the Remote SSH extension offers an option called remote.SSH.bindHost, allowing users to set the VS Code Server to listen on the other interface, such as the network bridge. However, when VS Code establishes a forwarding tunnel, it connects to 127.0.0.1. I'm curious if this hardcoding is a bug or a related functional option is unavailable.

When VS Code establishes a forwarding connection to a remote SSH server, can it connect to remote.SSH.bindHost or use a new user option such as remote.SSH.connectHost?

This is the Error log from sshd in Remote OS:

Jan 13 18:26:58 ubuntu sshd[1581]: refused local port forward: originator 127.0.0.1 port 43834, target 127.0.0.1 port 36601
Jan 13 18:26:58 ubuntu sshd[1581]: refused local port forward: originator 127.0.0.1 port 43820, target 127.0.0.1 port 36601

This is the Remote SSH Log during the connection:

Remote SSH Log 
[18:26:57.478] Log Level: 2
[18:26:57.491] SSH Resolver called for "ssh-remote+X.X.X.X", attempt 1
[18:26:57.494] remote.SSH.useLocalServer = true
[18:26:57.494] remote.SSH.useExecServer = true
[18:26:57.494] remote.SSH.path = 
[18:26:57.494] remote.SSH.configFile = 
[18:26:57.494] remote.SSH.useFlock = true
[18:26:57.494] remote.SSH.lockfilesInTmp = false
[18:26:57.494] remote.SSH.localServerDownload = auto
[18:26:57.494] remote.SSH.remoteServerListenOnSocket = false
[18:26:57.494] remote.SSH.defaultExtensions = []
[18:26:57.494] remote.SSH.defaultExtensionsIfInstalledLocally = ["GitHub.copilot","GitHub.copilot-chat"]
[18:26:57.494] remote.SSH.loglevel = 2
[18:26:57.494] remote.SSH.enableDynamicForwarding = true
[18:26:57.494] remote.SSH.enableRemoteCommand = false
[18:26:57.494] remote.SSH.serverPickPortsFromRange = {}
[18:26:57.494] remote.SSH.serverInstallPath = {}
[18:26:57.494] remote.SSH.permitPtyAllocation = false
[18:26:57.494] remote.SSH.preferredLocalPortRange = undefined
[18:26:57.495] remote.SSH.useCurlAndWgetConfigurationFiles = false
[18:26:57.495] remote.SSH.experimental.chat = true
[18:26:57.495] remote.SSH.experimental.enhancedSessionLogs = true
[18:26:57.500] VS Code version: 1.96.2
[18:26:57.500] Remote-SSH version: remote-ssh@0.117.2025011020
[18:26:57.500] linux x64
[18:26:57.502] SSH Resolver called for host: X.X.X.X
[18:26:57.502] Setting up SSH remote "X.X.X.X"
[18:26:57.505] Acquiring local install lock: /tmp/vscode-remote-ssh-0150773a-install.lock
[18:26:57.507] Looking for existing server data file at /home/yanghaku/.config/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-0150773a-fabdb6a30b49f79a7aba0f2ad9df9b399473380f-0.117.2025011020-es/data.json
[18:26:57.507] No existing data file
[18:26:57.507] Using commit id "fabdb6a30b49f79a7aba0f2ad9df9b399473380f" and quality "stable" for server
[18:26:57.507] Extensions to install: 
[18:26:57.512] Install and start server if needed
[18:26:57.514] PATH: /home/yanghaku/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/opt/rocm/bin
[18:26:57.514] Checking ssh with "ssh -V"
[18:26:57.522] > OpenSSH_9.9p1, OpenSSL 3.4.0 22 Oct 2024

[18:26:57.524] askpass server listening on /run/user/1000/vscode-ssh-askpass-e547ed63df42b79d9bdb557c9af3a90b61d01d44.sock
[18:26:57.524] Spawning local server with {"serverId":1,"ipcHandlePath":"/run/user/1000/vscode-ssh-askpass-08ca1650f5f5e04195231de7cd8b9e43945f848c.sock","sshCommand":"ssh","sshArgs":["-v","-T","-D","42591","-o","ConnectTimeout=15","X.X.X.X"],"serverDataFolderName":".vscode-server","dataFilePath":"/home/yanghaku/.config/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-0150773a-fabdb6a30b49f79a7aba0f2ad9df9b399473380f-0.117.2025011020-es/data.json"}
[18:26:57.524] Local server env: {"SHELL":"/usr/bin/zsh","DISPLAY":":0","ELECTRON_RUN_AS_NODE":"1","SSH_ASKPASS":"/home/yanghaku/.vscode/extensions/ms-vscode-remote.remote-ssh-0.117.2025011020/out/local-server/askpass.sh","VSCODE_SSH_ASKPASS_NODE":"/opt/visual-studio-code/code","VSCODE_SSH_ASKPASS_EXTRA_ARGS":"","VSCODE_SSH_ASKPASS_MAIN":"/home/yanghaku/.vscode/extensions/ms-vscode-remote.remote-ssh-0.117.2025011020/out/askpass-main.js","VSCODE_SSH_ASKPASS_HANDLE":"/run/user/1000/vscode-ssh-askpass-e547ed63df42b79d9bdb557c9af3a90b61d01d44.sock"}
[18:26:57.529] Spawned 114262
[18:26:57.529] Using connect timeout of 17 seconds
[18:26:57.595] > local-server-1> Running ssh connection command: ssh -v -T -D 42591 -o ConnectTimeout=15 X.X.X.X
[18:26:57.599] > local-server-1> Spawned ssh, pid=114274
[18:26:57.602] stderr> OpenSSH_9.9p1, OpenSSL 3.4.0 22 Oct 2024
[18:26:57.666] stderr> debug1: Server host key: ssh-ed25519 SHA256:zLL7oI7mTNyw8YfXxDPFZPXcDAVBxW5usqW5MHfacXo
[18:26:57.734] stderr> Authenticated to X.X.X.X ([X.X.X.X]:22) using "publickey".
[18:26:57.877] > ready: 8fa8ea0b1672
[18:26:57.880] > Linux 6.8.0-51-generic #52-Ubuntu SMP PREEMPT_DYNAMIC Thu Dec  5 13:09:44 UTC 2024
[18:26:57.881] Platform: linux
[18:26:57.881] > /bin/bash
[18:26:57.882] Parent Shell: bash
[18:26:57.882] Parent Shell pid: 114262
[18:26:57.884] > 394
[18:26:57.884] Waiting for pid of spawned 'sh' subshell: '394'...
[18:26:57.886] > 8fa8ea0b1672: running
> Script executing under PID: 394
[18:26:57.900] > Found existing installation at /home/foo/.vscode-server...
> Starting VS Code CLI...
> printenv:
[18:26:57.903] >     SUDO_GID=0
>     MAIL=/var/mail/foo
>     USER=foo
>     XDG_SESSION_TYPE=unspecified
>     SHLVL=1
>     HOME=/home/foo
>     DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
>     SUDO_UID=0
>     LOGNAME=foo
>     _=/usr/bin/sh
>     XDG_SESSION_CLASS=background
>     TERM=unknown
>     XDG_SESSION_ID=9
>     PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
>     XDG_RUNTIME_DIR=/run/user/1000
>     LANG=C.UTF-8
>     SUDO_COMMAND=/bin/bash
>     SHELL=/bin/bash
>     SUDO_USER=root
>     PWD=/home/foo
[18:26:57.906] > Removing old logfile at /home/foo/.vscode-server/.cli.fabdb6a30b49f79a7aba0f2ad9df9b399473380f.log
[18:26:57.910] > Spawned remote CLI: 414
[18:26:57.914] > Waiting for server log...
[18:26:57.950] > Waiting for server log...
[18:26:57.990] > 8fa8ea0b1672: start
> listeningOn==10.3.128.3:36601==
> osReleaseId==ubuntu==
> arch==x86_64==
> vscodeArch==x64==
> bitness==64==
> tmpDir==/run/user/1000==
> platform==linux==
> unpackResult====
> didLocalDownload==0==
> downloadTime====
> installTime====
> serverStartTime==84==
[18:26:57.990] > execServerToken==11aa11a1-1aaa-111a-a11a-1a1111a11a1a==
> platformDownloadPath==cli-alpine-x64==
> SSH_AUTH_SOCK====
> DISPLAY====
> 8fa8ea0b1672: end
[18:26:57.991] Received install output: 
listeningOn==10.3.128.3:36601==
osReleaseId==ubuntu==
arch==x86_64==
vscodeArch==x64==
bitness==64==
tmpDir==/run/user/1000==
platform==linux==
unpackResult====
didLocalDownload==0==
downloadTime====
installTime====
serverStartTime==84==execServerToken==11aa11a1-1aaa-111a-a11a-1a1111a11a1a==
platformDownloadPath==cli-alpine-x64==
SSH_AUTH_SOCK====
DISPLAY====

[18:26:57.991] Remote server is listening on port 36601
[18:26:57.992] Parsed server configuration: {"serverConfiguration":{"remoteListeningOn":{"port":36601},"osReleaseId":"ubuntu","arch":"x86_64","sshAuthSock":"","display":"","tmpDir":"/run/user/1000","platform":"linux","execServerToken":"11aa11a1-1aaa-111a-a11a-1a1111a11a1a"},"serverStartTime":84,"installUnpackCode":""}
[18:26:57.993] Persisting server connection details to /home/yanghaku/.config/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-0150773a-fabdb6a30b49f79a7aba0f2ad9df9b399473380f-0.117.2025011020-es/data.json
[18:26:57.998] Starting forwarding server. local port 34037 -> socksPort 42591 -> remotePort 36601
[18:26:57.998] Forwarding server listening on port 34037
[18:26:57.999] Waiting for ssh tunnel to be ready
[18:26:58.001] [Forwarding server port 34037] Got connection 0
[18:26:58.003] Tunneled port 36601 to local port 34037
[18:26:58.003] Resolved "ssh-remote+X.X.X.X" to "port 34037"
[18:26:58.016] Initizing new exec server for ssh-remote+X.X.X.X
[18:26:58.016] Resolving exec server at port 34037
[18:26:58.017] [Forwarding server port 34037] Got connection 1
[18:26:58.026] stderr> channel 3: open failed: administratively prohibited: open failed
[18:26:58.026] ERROR: TCP port forwarding appears to be disabled on the remote host. Ensure that the sshd_config has `AllowTcpForwarding yes`. Contact your system administrator if needed.
[18:26:58.027] Failed to set up socket for dynamic port forward to remote port 36601: Socket closed. TCP port forwarding may be disabled, or the remote server may have crashed. See the VS Code Server log above for details.
[18:26:58.028] stderr> channel 4: open failed: administratively prohibited: open failed
[18:26:58.029] Failed to set up socket for dynamic port forward to remote port 36601: Socket closed. TCP port forwarding may be disabled, or the remote server may have crashed. See the VS Code Server log above for details.
@vs-code-engineering vs-code-engineering bot added the ssh Issue in vscode-remote SSH label Jan 13, 2025
@joshspicer
Copy link
Member

Thank you for the detailed issue. It does indeed seem like we hardcode in the loopdevice device IP for the tunnel. It seems like we'd need a second setting for the Remote OS's interface (10.3.128.1)

Is that SSH config all you are doing to force the VS Code server installation into the container? If there's any other configuration you could share for me to get a quick repro, that would be appreciated :)

@joshspicer joshspicer added the feature-request Request for new features or functionality label Jan 14, 2025
@vs-code-engineering vs-code-engineering bot added this to the Backlog Candidates milestone Jan 14, 2025
@vs-code-engineering VS Code Engineering
Copy link

This feature request is now a candidate for our backlog. The community has 60 days to upvote the issue. If it receives 10 upvotes we will move it to our backlog. If not, we will close it. To learn more about how we handle feature requests, please see our documentation.

Happy Coding!

@yanghaku
Copy link
Author

Hello @joshspicer! Thanks for your reply.
To reproduce this problem quickly, follow these steps:

  1. Use VSCode remote SSH to connect to the VS Code Server on your remote OS. If this connection is successful, it confirms that the SSH daemon (sshd) configuration is correct, including settings such as AllowTcpForwarding yes.
  2. Change the binding interface for the VS Code Server on the remote OS by adding these settings:
"remote.SSH.bindHost": {
  "<Your Remote Host>": "<The Ip of Remote Host (not 127.0.0.1)>"
}

Expected behavior 1: The tunnel is established normally using the specified interface. (Use the remote.SSH.bindHost)

Expected behavior 2: The tunnel is established correctly after we add a setting such as:

"remote.SSH.connectHost": {
  "<Your Remote Host>": "<The Ip of Remote Host (not 127.0.0.1)>"
}

Actual behavior: Still using loop interface and tunnel establishment failed, the logs are as follows:

sshd logs:

Jan 15 15:32:28 ubuntu systemd-logind[600]: New session 20 of user test.
Jan 15 15:32:28 ubuntu systemd[1]: Started session-20.scope - Session 20 of User test.
Jan 15 15:32:28 ubuntu sshd[7551]: error: connect_to 127.0.0.1 port 35971: failed.
Jan 15 15:32:28 ubuntu sshd[7551]: error: connect_to 127.0.0.1 port 35971: failed.

Remote SSH Logs:

details 
[15:32:28.454] Log Level: 2
[15:32:28.467] SSH Resolver called for "ssh-remote+X.X.X.X", attempt 1
[15:32:28.471] remote.SSH.useLocalServer = true
[15:32:28.471] remote.SSH.useExecServer = true
[15:32:28.471] remote.SSH.path = 
[15:32:28.471] remote.SSH.configFile = 
[15:32:28.471] remote.SSH.useFlock = true
[15:32:28.471] remote.SSH.lockfilesInTmp = false
[15:32:28.471] remote.SSH.localServerDownload = auto
[15:32:28.471] remote.SSH.remoteServerListenOnSocket = false
[15:32:28.471] remote.SSH.defaultExtensions = []
[15:32:28.472] remote.SSH.defaultExtensionsIfInstalledLocally = ["GitHub.copilot","GitHub.copilot-chat"]
[15:32:28.472] remote.SSH.loglevel = 2
[15:32:28.472] remote.SSH.enableDynamicForwarding = true
[15:32:28.472] remote.SSH.enableRemoteCommand = false
[15:32:28.472] remote.SSH.serverPickPortsFromRange = {}
[15:32:28.472] remote.SSH.serverInstallPath = {}
[15:32:28.472] remote.SSH.permitPtyAllocation = false
[15:32:28.472] remote.SSH.preferredLocalPortRange = undefined
[15:32:28.472] remote.SSH.useCurlAndWgetConfigurationFiles = false
[15:32:28.472] remote.SSH.experimental.chat = true
[15:32:28.472] remote.SSH.experimental.enhancedSessionLogs = true
[15:32:28.476] VS Code version: 1.96.3
[15:32:28.476] Remote-SSH version: remote-ssh@0.117.2025011415
[15:32:28.476] linux x64
[15:32:28.477] SSH Resolver called for host: X.X.X.X
[15:32:28.477] Setting up SSH remote "X.X.X.X"
[15:32:28.481] Acquiring local install lock: /tmp/vscode-remote-ssh-0150773a-install.lock
[15:32:28.482] Looking for existing server data file at /home/yanghaku/.config/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-0150773a-91fbdddc47bc9c09064bf7acf133d22631cbf083-0.117.2025011415-es/data.json
[15:32:28.483] No existing data file
[15:32:28.483] Using commit id "91fbdddc47bc9c09064bf7acf133d22631cbf083" and quality "stable" for server
[15:32:28.483] Extensions to install: 
[15:32:28.487] Install and start server if needed
[15:32:28.489] PATH: /home/yanghaku/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/opt/rocm/bin
[15:32:28.489] Checking ssh with "ssh -V"
[15:32:28.497] > OpenSSH_9.9p1, OpenSSL 3.4.0 22 Oct 2024

[15:32:28.499] askpass server listening on /run/user/1000/vscode-ssh-askpass-ee2a6ad27bc79e27a46597adc3dda60b338d2330.sock
[15:32:28.500] Spawning local server with {"serverId":1,"ipcHandlePath":"/run/user/1000/vscode-ssh-askpass-d0b9045d5fc507cc937784b37a5c15f65b95dfd0.sock","sshCommand":"ssh","sshArgs":["-v","-T","-D","42335","-o","ConnectTimeout=15","X.X.X.X","bash"],"serverDataFolderName":".vscode-server","dataFilePath":"/home/yanghaku/.config/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-0150773a-91fbdddc47bc9c09064bf7acf133d22631cbf083-0.117.2025011415-es/data.json"}
[15:32:28.500] Local server env: {"SHELL":"/usr/bin/zsh","DISPLAY":":0","ELECTRON_RUN_AS_NODE":"1","SSH_ASKPASS":"/home/yanghaku/.vscode/extensions/ms-vscode-remote.remote-ssh-0.117.2025011415/out/local-server/askpass.sh","VSCODE_SSH_ASKPASS_NODE":"/opt/visual-studio-code/code","VSCODE_SSH_ASKPASS_EXTRA_ARGS":"","VSCODE_SSH_ASKPASS_MAIN":"/home/yanghaku/.vscode/extensions/ms-vscode-remote.remote-ssh-0.117.2025011415/out/askpass-main.js","VSCODE_SSH_ASKPASS_HANDLE":"/run/user/1000/vscode-ssh-askpass-ee2a6ad27bc79e27a46597adc3dda60b338d2330.sock"}
[15:32:28.504] Spawned 217691
[15:32:28.505] Using connect timeout of 17 seconds
[15:32:28.571] > local-server-1> Running ssh connection command: ssh -v -T -D 42335 -o ConnectTimeout=15 X.X.X.X bash
[15:32:28.574] > local-server-1> Spawned ssh, pid=217703
[15:32:28.576] stderr> OpenSSH_9.9p1, OpenSSL 3.4.0 22 Oct 2024
[15:32:28.631] stderr> debug1: Server host key: ssh-ed25519 SHA256:zLL7oI7mTNyw8YfXxDPFZPXcDAVBxW5usqW5MHfacXo
[15:32:28.696] stderr> Authenticated to X.X.X.X ([X.X.X.X]:22) using "publickey".
[15:32:28.743] > ready: 207b54073a18
[15:32:28.746] > Linux 6.8.0-51-generic #52-Ubuntu SMP PREEMPT_DYNAMIC Thu Dec  5 13:09:44 UTC 2024
[15:32:28.746] Platform: linux
[15:32:28.747] > /bin/sh
[15:32:28.747] Parent Shell: sh
[15:32:28.747] Parent Shell pid: 217691
[15:32:28.749] > 7555
[15:32:28.749] Waiting for pid of spawned 'sh' subshell: '7555'...
[15:32:28.751] > 207b54073a18: running
> Script executing under PID: 7555
[15:32:28.764] > Found existing installation at /home/test/.vscode-server...
> Starting VS Code CLI...
[15:32:28.767] > Removing old logfile at /home/test/.vscode-server/.cli.91fbdddc47bc9c09064bf7acf133d22631cbf083.log
[15:32:28.771] > Spawned remote CLI: 7573
[15:32:28.774] > Waiting for server log...
[15:32:28.809] > Waiting for server log...
[15:32:28.849] > 207b54073a18: start
> listeningOn==X.X.X.X:35971==
> osReleaseId==ubuntu==
> arch==x86_64==
> vscodeArch==x64==
> bitness==64==
> tmpDir==/run/user/1003==
> platform==linux==
> unpackResult====
> didLocalDownload==0==
> downloadTime====
> installTime====
> serverStartTime==82==
> execServerToken==aa111a11-1aa1-11aa-aa1a-11a1111aaaa1==
> platformDownloadPath==cli-alpine-x64==
> SSH_AUTH_SOCK====
> DISPLAY====
> 207b54073a18: end
[15:32:28.849] Received install output: 
listeningOn==X.X.X.X:35971==
osReleaseId==ubuntu==
arch==x86_64==
vscodeArch==x64==
bitness==64==
tmpDir==/run/user/1003==
platform==linux==
unpackResult====
didLocalDownload==0==
downloadTime====
installTime====
serverStartTime==82==
execServerToken==aa111a11-1aa1-11aa-aa1a-11a1111aaaa1==
platformDownloadPath==cli-alpine-x64==
SSH_AUTH_SOCK====
DISPLAY====

[15:32:28.850] Remote server is listening on port 35971
[15:32:28.850] Parsed server configuration: {"serverConfiguration":{"remoteListeningOn":{"port":35971},"osReleaseId":"ubuntu","arch":"x86_64","sshAuthSock":"","display":"","tmpDir":"/run/user/1003","platform":"linux","execServerToken":"aa111a11-1aa1-11aa-aa1a-11a1111aaaa1"},"serverStartTime":82,"installUnpackCode":""}
[15:32:28.851] Persisting server connection details to /home/yanghaku/.config/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-0150773a-91fbdddc47bc9c09064bf7acf133d22631cbf083-0.117.2025011415-es/data.json
[15:32:28.857] Starting forwarding server. local port 38331 -> socksPort 42335 -> remotePort 35971
[15:32:28.857] Forwarding server listening on port 38331
[15:32:28.857] Waiting for ssh tunnel to be ready
[15:32:28.859] [Forwarding server port 38331] Got connection 0
[15:32:28.861] Tunneled port 35971 to local port 38331
[15:32:28.861] Resolved "ssh-remote+X.X.X.X" to "port 38331"
[15:32:28.874] Initizing new exec server for ssh-remote+X.X.X.X
[15:32:28.874] Resolving exec server at port 38331
[15:32:28.875] [Forwarding server port 38331] Got connection 1
[15:32:28.885] stderr> channel 3: open failed: connect failed: Connection refused
[15:32:28.886] Failed to set up socket for dynamic port forward to remote port 35971: Socket closed. TCP port forwarding may be disabled, or the remote server may have crashed. See the VS Code Server log above for details.
[15:32:28.887] stderr> channel 4: open failed: connect failed: Connection refused
[15:32:28.887] Failed to set up socket for dynamic port forward to remote port 35971: Socket closed. TCP port forwarding may be disabled, or the remote server may have crashed. See the VS Code Server log above for details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@roblourens roblourens

@joshspicer joshspicer

Labels
feature-request Request for new features or functionality ssh Issue in vscode-remote SSH
Projects
None yet
Milestone
Backlog Candidates
Development

No branches or pull requests
3 participants
@roblourens @joshspicer @yanghaku