SSH via local companion not working on Ubuntu

[jkaye2012]

Bug description

First time attempting to use the local companion, something seems to be off with SSH auth.

Prior to running anything, I've had default keys set up (and working) for quite some time. The companion runs normally, connecting to the workspace and populating keys and config files:

jkaye@jkaye-linux ~/Downloads $ ls /tmp/gitpod_*
/tmp/gitpod_1a60521e-cb24-4723-b7f6-3fabd5645abc_id_rsa      /tmp/gitpod_d45ebe9d-3ff3-4eaa-a21f-7060164a2bc2_id_rsa      /tmp/gitpod_ssh_config
/tmp/gitpod_1a60521e-cb24-4723-b7f6-3fabd5645abc_id_rsa.pub  /tmp/gitpod_d45ebe9d-3ff3-4eaa-a21f-7060164a2bc2_id_rsa.pub

jkaye@jkaye-linux ~/Downloads $ cat /tmp/gitpod_ssh_config 
Host tomato-cat-gv5z3x4g
HostName 127.0.0.1
User gitpod
Port 33427
IdentityFile /tmp/gitpod_1a60521e-cb24-4723-b7f6-3fabd5645abc_id_rsa

I tried starting a new pod, so you see the second set of keys there.

Then, the unfortunate ssh command:

jkaye@jkaye-linux ~/Downloads $ ssh -F /tmp/gitpod_ssh_config tomato-cat-gv5z3x4g
gitpod@127.0.0.1: Permission denied (publickey).

Steps to reproduce

Ubuntu 18.04.5, start the local companion and attempt to connect via SSH.

Expected behavior

No response

Example repository

No response

Anything else?

I tried a few different things: directly setting the identity file, etc. I've also verified that the authorized public key uploaded to the workspace matches the public key listed in the auto-generated configuration file.

[ajhalili2006]

Probably try to do ssh-add /tmp/gitpod_workspace-id-here/rsa first, as the GNOME keyring (or your desktop environment's keyring if you're not using the stock DE or you're on an terminal without xrdp stuff) may not know the private key yet.

[akosyakov]

Could you start the app with --version and share it here?

[jkaye2012]

Thanks for the quick response!

ssh-add'ing the identity did not work unfortunately. From what I can tell, this doesn't look like it would rely on the keyring (as the identity file is being provided explicitly via the config), but I did give it a shot:

jkaye@jkaye-linux ~/Downloads $ ssh-add /tmp/gitpod_6c03cf8c-4a54-4c1d-bfc4-8321dc04c00e_id_rsa
Identity added: /tmp/gitpod_6c03cf8c-4a54-4c1d-bfc4-8321dc04c00e_id_rsa (/tmp/gitpod_6c03cf8c-4a54-4c1d-bfc4-8321dc04c00e_id_rsa)
jkaye@jkaye-linux ~/Downloads $ ssh -F /tmp/gitpod_ssh_config tomato-cat-gv5z3x4g
The authenticity of host '[127.0.0.1]:43433 ([127.0.0.1]:43433)' can't be established.
RSA key fingerprint is SHA256:v6ZAgal8lu09A1LQ2XKv44lz74WIeLnSndk2D88P6h4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[127.0.0.1]:43433' (RSA) to the list of known hosts.
gitpod@127.0.0.1: Permission denied (publickey).

Here's the version output as requested:

jkaye@jkaye-linux ~/Downloads $ ./gitpod-local-companion-linux --version
gitpod-local-companion version main.1419

I installed this on Friday using this command:

curl -OL https://gitpod.io/static/bin/gitpod-local-companion-linux && chmod +x ./gitpod-local-companion-linux

[akosyakov]

/schedule

[roboquat]

@akosyakov: Issue scheduled in the IDE team (WIP: 0)

In response to this:

/schedule

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[akosyakov]

Could you share logs of the local app please? You can send them to anton@gitpod.io. The local app auto generates keys for each workspace and uploads it to the server. It could be that key does not make it to the workspace for some reason.

[jkaye2012]

Sure, sending these over now.

I mentioned this in the original issue, but I did verify that the keys appeared to be present in the workspace (and that they matched the keys that were generated locally).

[akosyakov]

I see logs like /home/gitpod must be owned by user or root, and not writable by others in your workspace. It seems to come from dropbear (ssh server). Could you check what are permissions for this folder? Do you try to manipulate permissions somehow for home direction in your repo?

[jkaye2012]

Nope, have not modified anything in the workspace or repo.

gitpod /workspace/tribe $ ls -la /home
total 24
drwxr-xr-x 1 root   root   4096 Aug 20 15:04 .
drwxr-xr-x 1 root   root   4096 Sep 15 14:06 ..
drwxrwxr-x 1 gitpod gitpod 4096 Sep 15 14:07 gitpod
drwxr-xr-x 1 root   root   4096 May 12 12:23 linuxbrew

Looks like that is 777 somehow though. When I start a new pod, those are the same permissions that are set somehow. Any idea what could cause that?

[akosyakov]

Could you try on another repo? like https://github.com/akosyakov/parcel-demo? by default user's group should not have write access. I was assuming something is going in your Dockerfile or something. But I will check with the rest of the team.

[akosyakov]

I can reproduce it by doing chmod g+w /home/gitpod.

chmod g-w /home/gitpod/ as a workaround

[jkaye2012]

Can confirm that 755 on /home/gitpod seems to be working.

[jkaye2012]

I'll check with my team to see if there's something odd that we're doing in the Dockerfile. I'm new to this project, so it's definitely possible. Will report back.

[akosyakov]

@csweichel Do we guarantee: /home/gitpod must be owned by user or root, and not writable by others It seems that dropbear breaks otherwise.

[akosyakov]

We are going to patch dropbear for our case to remove this check.

[jkaye2012]

I also patched our environment to set the permissions correctly.

Another thing I noticed this morning is that the new "Open in VS Code" command from gitpod edits the local VS Code remote ssh configuration to point to a specific ssh config file. Something I did for myself earlier this week is add the following to my ~/.ssh/config:

Include = /tmp/gitpod_ssh_config

This makes all active workspaces automatically discoverable from the remote development extension in VS Code and removes the need to explicitly specify a config file when running ssh commands. May be a nice feature for local companion to offer.

[akosyakov]

This makes all active workspaces automatically discoverable from the remote development extension in VS Code and removes the need to explicitly specify a config file when running ssh commands. May be a nice feature for local companion to offer.

Yes, we considered it, but unfortunately there is a bug in Remote SSH which corrupts user ssh config with includes: microsoft/vscode-remote-release#5449. That's said we find mangling with remote ssh settings also not nice from user perspective. And look in alternative solutions which does not require code generation, like using ssh proxy command: #5602