DevContainer: SSH und GPG agent forwarding for WSL2 #2925

chrmarti · 2020-05-04T09:04:14Z

Furthermore I tested now also with SSH Auth but that does not work either. Then the my SSH key is not found an ssh-add -l (inside the container) says Could not open a connection to your authentication agent. But inside the WSL 2 distro it works also using SSH. (ssh-agent is running on the WSL2 distro).

Originally posted by @J0F3 in #2858 (comment)

The text was updated successfully, but these errors were encountered:

chrmarti · 2020-06-16T06:43:36Z

Looked into this and part of the challenge is to get at the ssh-agent's socket path stored in the environment variable SSH_AUTH_SOCK. That variable might be only set in the shell that spawned the ssh-agent. But that depends on how you set things up, e.g., https://stackoverflow.com/questions/18880024/start-ssh-agent-on-login has a nice snippet for .bash_profile, so each login shell has SSH_AUTH_SOCK set.

If you are using the ssh-agent in WSL: How are you starting it and how do you pass SSH_AUTH_SOCK around?

J0F3 · 2020-06-18T14:27:52Z

I am by no mean experienced in using the SSH Agent. But I am using the snipped form the official remote containers documentation in my .bashrc in the WSL (2) distro. (https://code.visualstudio.com/docs/remote/containers#_using-ssh-keys)
This seems to work quite well and starts the agents every time I launch the WSL and seems to also to set the SSH_AUTH_SOCK variable.

biltongza · 2020-06-27T10:08:46Z

I am having the same problem. Tried using the .bash_profile method as well as putting the same thing in .zprofile (because I use zsh) and neither seems to get me a working ssh agent in the container. SSH_AUTH_SOCK env var is present in WSL.

[27 ms] Start: Resolving remote
[28 ms] Start: Run: wsl -d Ubuntu -e wslpath -u \\wsl$\Ubuntu\home\logan\git\pugmod
[157 ms] Start: Run: wsl -d Ubuntu -e /bin/sh -c cd '/home/logan/git/pugmod' && printenv
[261 ms] Setting up container for folder or workspace: /home/logan/git/pugmod

[261 ms] Start: Run: wsl -d Ubuntu -e /bin/sh -c cd '/home/logan/git/pugmod' && test '-f' '/home/logan/git/pugmod/.devcontainer/devcontainer.json'
[375 ms] Start: Run: wsl -d Ubuntu -e /bin/sh -c cd '/home/logan/git/pugmod' && cat '/home/logan/git/pugmod/.devcontainer/devcontainer.json'
[482 ms] Start: Run: wsl -d Ubuntu -e /bin/sh -c cd '/home/logan/git/pugmod' && git 'rev-parse' '--show-toplevel'
[585 ms] Start: Check Docker is running
[586 ms] Start: Run: wsl -d Ubuntu -e /bin/sh -c cd '/home/logan/git/pugmod' && docker 'info'
[805 ms] Start: Run: wsl -d Ubuntu -e /bin/sh -c cd '/home/logan/git/pugmod' && docker 'ps' '-q' '-a' '--filter' 'label=vsch.local.folder=\\wsl$\Ubuntu\home\logan\git\pugmod' '--filter' 'label=vsch.quality=stable'
[960 ms] Start: Run: wsl -d Ubuntu -e /bin/sh -c cd '/home/logan/git/pugmod' && docker 'inspect' '--type' 'container' '1ac1b618badf'
[1098 ms] Start: Removing existing container.
[1098 ms] Start: Run: wsl -d Ubuntu -e /bin/sh -c cd '/home/logan/git/pugmod' && docker 'rm' '-f' '1ac1b618badf391322e0fb262d3fdab5a5a414e1a85670a8c3943dc256bae703'
[1599 ms] Start: Run: wsl -d Ubuntu -e /bin/sh -c cd '/home/logan/git/pugmod' && test '-f' '/home/logan/git/pugmod/.devcontainer/Dockerfile'
[1722 ms] Start: Run: wsl -d Ubuntu -e /bin/sh -c cd '/home/logan/git/pugmod' && docker 'build' '-f' '/home/logan/git/pugmod/.devcontainer/Dockerfile' '-t' 'vsc-pugmod-0b6c2468c3ad5a817dafb3f3e374f472' '--build-arg' 'VARIANT=20.04' '/home/logan/git/pugmod/.devcontainer'
Sending build context to Docker daemon   5.12kB
Step 1/8 : ARG VARIANT="20.04"
Step 2/8 : FROM mcr.microsoft.com/vscode/devcontainers/base:0-ubuntu${VARIANT}
 ---> 2b5fb7ba7ef2
Step 3/8 : RUN apt-get update     && export DEBIAN_FRONTEND=noninteractive     &
& apt-get -y install --no-install-recommends curl
 ---> Using cache
 ---> dac7777f59bc
Step 4/8 : RUN mkdir -p /pugmod
 ---> Using cache
 ---> b00477385ea9
Step 5/8 : WORKDIR /pugmod
 ---> Using cache
 ---> 17beaae1ca49
Step 6/8 : ADD . /pugmod
 ---> Using cache
 ---> 26c26e7377c4
Step 7/8 : RUN curl -L https://sm.alliedmods.net/smdrop/1.10/sourcemod-1.10.0-gi
t6490-linux.tar.gz -o sourcemod-1.10.0-git6490-linux.tar.gz
 ---> Using cache
 ---> c0bf64ca6fb3
Step 8/8 : RUN tar xf sourcemod-1.10.0-git6490-linux.tar.gz
 ---> Using cache
 ---> 13a98658b906
Successfully built 13a98658b906
Successfully tagged vsc-pugmod-0b6c2468c3ad5a817dafb3f3e374f472:latest
[2399 ms] Start: Run: wsl -d Ubuntu -e /bin/sh -c cd '/home/logan/git/pugmod' && docker 'inspect' '--type' 'image' 'vsc-pugmod-0b6c2468c3ad5a817dafb3f3e374f472'
[2542 ms] Start: Run: wsl -d Ubuntu -e /bin/sh -c cd '/home/logan/git/pugmod' && /bin/sh '-c' 'a="$(which '"'"'docker'"'"')" && realpath "$a"'
[2652 ms] Start: Run: wsl -d Ubuntu -e /bin/sh -c cd '/home/logan/git/pugmod' && docker 'events' '--format' '{{json .}}' '--filter' 'status=start'
[2662 ms] Start: Starting container
[2662 ms] Start: Run: wsl -d Ubuntu -e /bin/sh -c cd '/home/logan/git/pugmod' && docker 'run' '-a' 'STDOUT' '-a' 'STDERR' '--mount' 'type=bind,source=/home/logan/git/pugmod,target=/workspaces/pugmod' '-l' 'vsch.quality=stable' '-l' 'vsch.remote.devPort=0' '-l' 'vsch.local.folder=\\wsl$\Ubuntu\home\logan\git\pugmod' '--entrypoint' '/bin/sh' 'vsc-pugmod-0b6c2468c3ad5a817dafb3f3e374f472' '-c' 'echo Container started ; while sleep 1; do :; done'
[3117 ms] Start: Run: wsl -d Ubuntu -e /bin/sh -c cd '/home/logan/git/pugmod' && docker 'ps' '-q' '-a' '--filter' 'label=vsch.local.folder=\\wsl$\Ubuntu\home\logan\git\pugmod' '--filter' 'label=vsch.quality=stable'
Container started
[3257 ms] Start: Run: wsl -d Ubuntu -e /bin/sh -c cd '/home/logan/git/pugmod' && docker 'inspect' '--type' 'container' '11087d2a31ca'
[3393 ms] Start: Inspecting container
[3393 ms] Start: Run: wsl -d Ubuntu -e /bin/sh -c cd '/home/logan/git/pugmod' && docker 'inspect' '--type' 'container' '11087d2a31ca7df76225373f841df50a677c1fe109337ac7b271e7240b9eb804'
[3530 ms] Start: Run: wsl -d Ubuntu -e /bin/sh -c cd '/home/logan/git/pugmod' && docker 'exec' '-i' '-u' 'root' '-e' 'VSCODE_REMOTE_CONTAINERS_SESSION=b104b96b-745d-407d-922b-a7942d25487d1593252304166' '11087d2a31ca7df76225373f841df50a677c1fe109337ac7b271e7240b9eb804' '/bin/sh'
[3541 ms] Start: Run in container: uname -m
[3694 ms] x86_64
[3694 ms] 
[3696 ms] Start: Run in container: (cat /etc/os-release || cat /usr/lib/os-release) 2>/dev/null
[3704 ms] NAME="Ubuntu"
VERSION="20.04 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
[3704 ms] 
[3704 ms] Start: Run in container: cat /etc/passwd
[3711 ms] Start: Updating configuration state
[3711 ms] Start: Run: wsl -d Ubuntu -e /bin/sh -c cd '/home/logan/git/pugmod' && cat '/home/logan/git/pugmod/.devcontainer/Dockerfile'
[3834 ms] Start: Setup shutdown monitor
[3836 ms] Forking shutdown monitor: c:\Users\logan.dam\.vscode\extensions\ms-vscode-remote.remote-containers-0.122.1\dist\shutdownMonitorProcess \\.\pipe\vscode-remote-containers-4dbc3008882fc922bac2287db6eb1568701a7a09-sock singleContainer Info c:\Users\logan.dam\AppData\Roaming\Code\logs\20200627T115935\exthost1\ms-vscode-remote.remote-containers
[3849 ms] Start: Run in container: test -d /root/.vscode-server
[3851 ms] 
[3851 ms] 
[3851 ms] Exit code 1
[3852 ms] Start: Run in container: test -d /root/.vscode-remote
[3853 ms] 
[3853 ms] 
[3853 ms] Exit code 1
[3854 ms] Start: Run in container: set -o noclobber ; mkdir -p '/root/.vscode-server/data/Machine' && { > '/root/.vscode-server/data/Machine/.writeMachineSettingsMarker' ; } 2> /dev/null
[3858 ms] 
[3858 ms] 
[3858 ms] Start: Run in container: mkdir -p '/root/.vscode-server/data/Machine' && cat >'/root/.vscode-server/data/Machine/settings.json' <<'settingsJSON'
[3861 ms] 
[3861 ms] 
[3862 ms] Start: Run in container: test -d /root/.vscode-server/bin/cd9ea6488829f560dc949a8b2fb789f3cdc05f5d
[3864 ms] 
[3864 ms] 
[3864 ms] Exit code 1
[3864 ms] Installing VS Code Server for commit cd9ea6488829f560dc949a8b2fb789f3cdc05f5d
[3864 ms] Start: Run in container: mkdir -p /root/.vscode-server/bin/cd9ea6488829f560dc949a8b2fb789f3cdc05f5d_1593252308859
[3866 ms] 
[3867 ms] 
[3868 ms] Start: Run: wsl -d Ubuntu -e /bin/sh -c cd '/home/logan/git/pugmod' && docker 'exec' '-i' '-u' 'root' '-w' '/root/.vscode-server/bin/cd9ea6488829f560dc949a8b2fb789f3cdc05f5d_1593252308859' '11087d2a31ca7df76225373f841df50a677c1fe109337ac7b271e7240b9eb804' 'tar' '--no-same-owner' '-xz' '--strip-components' '1'
[4899 ms] 
[4899 ms] 
[4899 ms] Start: Run in container: mv -n /root/.vscode-server/bin/cd9ea6488829f560dc949a8b2fb789f3cdc05f5d_1593252308859 /root/.vscode-server/bin/cd9ea6488829f560dc949a8b2fb789f3cdc05f5d
[4905 ms] 
[4905 ms] 
[4905 ms] Start: Launching Remote-Containers helper.
[4906 ms] Start: Run: wsl -d Ubuntu -e /bin/sh -c cd '/home/logan/git/pugmod' && gpgconf '--list-dir' 'agent-extra-socket'
[5010 ms] Start: Run in container: gpgconf --list-dir agent-socket
[5012 ms] /root/.gnupg/S.gpg-agent
[5012 ms] 
[5012 ms] Start: Run in container: mkdir -p -m 700 '/root/.gnupg'
[5013 ms] Start: Run: wsl -d Ubuntu -e /bin/sh -c cd '/home/logan/git/pugmod' && cat '/home/logan/.gitconfig'
[5014 ms] 
[5014 ms] 
[5014 ms] Start: Run in container: cat <<'EOF-/tmp/vscode-remote-containers-ce6416da39ea947e05840ba872c3e4721112468c.js' >/tmp/vscode-remote-containers-ce6416da39ea947e05840ba872c3e4721112468c.js
[5014 ms] Start: Run: wsl -d Ubuntu -e /bin/sh -c cd '/home/logan/git/pugmod' && gpgconf '--list-dir' 'homedir'
[5018 ms] 
[5018 ms] 
[5018 ms] Start: Run in container: cat <<'EOF-/tmp/vscode-remote-containers-server-ce6416da39ea947e05840ba872c3e4721112468c.js' >/tmp/vscode-remote-containers-server-ce6416da39ea947e05840ba872c3e4721112468c.js
[5025 ms] 
[5025 ms] 
[5025 ms] Start: Run: wsl -d Ubuntu -e /bin/sh -c cd '/home/logan/git/pugmod' && docker 'exec' '-i' '-u' 'root' '-e' 'REMOTE_CONTAINERS_SOCKETS=["/root/.gnupg/S.gpg-agent"]' '-e' 'REMOTE_CONTAINERS_IPC=/tmp/vscode-remote-containers-ipc-ce6416da39ea947e05840ba872c3e4721112468c.sock' '11087d2a31ca7df76225373f841df50a677c1fe109337ac7b271e7240b9eb804' '/root/.vscode-server/bin/cd9ea6488829f560dc949a8b2fb789f3cdc05f5d/node' '/tmp/vscode-remote-containers-server-ce6416da39ea947e05840ba872c3e4721112468c.js'
[5142 ms] Start: Run in container: # Copy /home/logan/.gitconfig to /root/.gitconfig
[5147 ms] 
[5148 ms] 
[5148 ms] Start: Run in container: command -v git >/dev/null 2>&1 && git config --global credential.helper '!f() { /root/.vscode-server/bin/cd9ea6488829f560dc949a8b2fb789f3cdc05f5d/node /tmp/vscode-remote-containers-ce6416da39ea947e05840ba872c3e4721112468c.js $*; }; f' || true
[5151 ms] 
[5151 ms] 
[5152 ms] Start: Run in container: set -o noclobber ; mkdir -p '/root/.vscode-server/data/Machine' && { > '/root/.vscode-server/data/Machine/.installExtensionsMarker' ; } 2> /dev/null
[5155 ms] 
[5156 ms] 
[5159 ms] Start: Run in container: test -d /root/.vscode-server/extensionsCache && ls /root/.vscode-server/extensionsCache || true
[5162 ms] 
[5162 ms] 
[5162 ms] Start: Run in container: gpgconf --list-dir homedir
[5166 ms] /root/.gnupg
[5166 ms] 
[5166 ms] Start: Run in container: mkdir -p /root/.vscode-server/extensionsCache/tmp-1593252310161
[5166 ms] Start: Run: wsl -d Ubuntu -e /bin/sh -c cd '/home/logan/git/pugmod' && cat '/home/logan/.gnupg/pubring.kbx'
[5170 ms] 
[5171 ms] 
[5171 ms] Start: Run: wsl -d Ubuntu -e /bin/sh -c cd '/home/logan/git/pugmod' && docker 'exec' '-i' '-u' 'root' '-w' '/root/.vscode-server/extensionsCache/tmp-1593252310161' '11087d2a31ca7df76225373f841df50a677c1fe109337ac7b271e7240b9eb804' 'tar' 'x' '--strip-components' '1'
[5311 ms] Start: Run: wsl -d Ubuntu -e /bin/sh -c cd '/home/logan/git/pugmod' && cat '/home/logan/.gnupg/trustdb.gpg'
[5441 ms] Start: Run in container: mv /root/.vscode-server/extensionsCache/tmp-1593252310161/* /root/.vscode-server/extensionsCache && rmdir /root/.vscode-server/extensionsCache/tmp-1593252310161
[5445 ms] 
[5446 ms] 
[5446 ms] Start: Run in container: cd /root/.vscode-server/bin/cd9ea6488829f560dc949a8b2fb789f3cdc05f5d; export VSCODE_AGENT_FOLDER=/root/.vscode-server; /root/.vscode-server/bin/cd9ea6488829f560dc949a8b2fb789f3cdc05f5d/server.sh --extensions-download-dir /root/.vscode-server/extensionsCache --install-extension dreae.sourcepawn-vscode --force
[5448 ms] Start: Run: wsl -d Ubuntu -e /bin/sh -c cd '/home/logan/git/pugmod' && gpg-connect-agent 'updatestartuptty' '/bye'
[9048 ms] Installing extensions...
Installing extension 'dreae.sourcepawn-vscode' v0.1.4...
Extension 'dreae.sourcepawn-vscode' v0.1.4 was successfully installed.
[9048 ms] 
[9049 ms] Start: Run in container: ls /root/.vscode-server/extensionsCache
[9050 ms] dreae.sourcepawn-vscode-0.1.4
[9050 ms] 
[9051 ms] Start: Run in container: for pid in `cd /proc && ls -d [0-9]*`; do { echo $pid ; readlink -f /proc/$pid/cwd ; xargs -0 < /proc/$pid/environ ; xargs -0 < /proc/$pid/cmdline ; } ; echo ; done 2>/dev/null
[9068 ms] Start: Run: wsl -d Ubuntu -e /bin/sh -c cd '/home/logan/git/pugmod' && docker 'exec' '-i' '-u' 'root' '-e' 'SHELL=/bin/bash' '-e' 'VSCODE_AGENT_FOLDER=/root/.vscode-server' '-w' '/root/.vscode-server/bin/cd9ea6488829f560dc949a8b2fb789f3cdc05f5d' '11087d2a31ca7df76225373f841df50a677c1fe109337ac7b271e7240b9eb804' '/root/.vscode-server/bin/cd9ea6488829f560dc949a8b2fb789f3cdc05f5d/server.sh' '--disable-user-env-probe' '--use-host-proxy' '--port' '0' '--extensions-download-dir' '/root/.vscode-server/extensionsCache'
[9389 ms] 

*
* Visual Studio Code Server
*
* Reminder: You may only use this software with Visual Studio family products,
* as described in the license https://aka.ms/vscode-remote/license
*


[9425 ms] IP Address: 172.17.0.2
[9427 ms] Extension host agent listening on 40531


[9427 ms] Start: Run in container: echo 40531 >/root/.vscode-server/bin/cd9ea6488829f560dc949a8b2fb789f3cdc05f5d/.devport
[9427 ms] [10:05:13] Extension host agent started.
[9428 ms] 
[9428 ms] 
[9430 ms] Forwarding local port 40531 to container port 40531
[9468 ms] Start: Run: wsl -d Ubuntu -e /bin/sh -c cd '/home/logan/git/pugmod' && docker 'exec' '-i' '-u' 'root' '-e' 'VSCODE_REMOTE_CONTAINERS_SESSION=b104b96b-745d-407d-922b-a7942d25487d1593252304166' '11087d2a31ca7df76225373f841df50a677c1fe109337ac7b271e7240b9eb804' '/root/.vscode-server/bin/cd9ea6488829f560dc949a8b2fb789f3cdc05f5d/node' '-e' '
[9469 ms] Start: Run: wsl -d Ubuntu -e /bin/sh -c cd '/home/logan/git/pugmod' && docker 'exec' '-i' '-u' 'root' '-e' 'VSCODE_REMOTE_CONTAINERS_SESSION=b104b96b-745d-407d-922b-a7942d25487d1593252304166' '11087d2a31ca7df76225373f841df50a677c1fe109337ac7b271e7240b9eb804' '/root/.vscode-server/bin/cd9ea6488829f560dc949a8b2fb789f3cdc05f5d/node' '-e' '
[9759 ms] [10:05:14] [::ffff:127.0.0.1][4c23b2b3][ExtensionHostConnection] New connection established.
[9763 ms] [10:05:14] [::ffff:127.0.0.1][4c23b2b3][ExtensionHostConnection] <186> Launched Extension Host Process.
[9767 ms] [10:05:14] [::ffff:127.0.0.1][6a44d532][ManagementConnection] New connection established.

In the logs there's clear mention of gpg but nothing about ssh agent.

lbssousa · 2020-07-01T17:45:05Z

Looked into this and part of the challenge is to get at the ssh-agent's socket path stored in the environment variable SSH_AUTH_SOCK. That variable might be only set in the shell that spawned the ssh-agent. But that depends on how you set things up, e.g., https://stackoverflow.com/questions/18880024/start-ssh-agent-on-login has a nice snippet for .bash_profile, so each login shell has SSH_AUTH_SOCK set.

If you are using the ssh-agent in WSL: How are you starting it and how do you pass SSH_AUTH_SOCK around?

I've installed keychain in my WSL distro and appended this line to my .bash_profile:

eval $(keychain --eval --agents ssh id_rsa)

kaito3desuyo · 2020-07-17T06:25:22Z

I have same issue.
However, it is a slightly special environment because it shares ssh-agent with Windows using wsl-ssh-agent.
https://github.com/rupor-github/wsl-ssh-agent

ATGardner · 2020-07-17T07:11:10Z

I also encountered the same problem. But after a while I realized that since I am already developing inside a container, I might a well run the container from my windows host, instead of from inside the WSL environment. The ssh agent forwarding works well from there.
I did have some issues with EOL characters being different when I clone into my windows host, but adding gitattributes helped me with that.
And cloning directly into the container (instead of mounting from the first) will make it even simpler.

kaito3desuyo · 2020-07-17T07:38:57Z

@ATGardner
Does that mean you have to open it with VS Code on Windows shell?

ATGardner · 2020-07-17T07:40:52Z

Yeah, I just cloned into my windows environment, opened the folder in vscode, and then reopened it inside the container. After seeing up the ssh agent in powershell, it just worked inside the container.

kaito3desuyo · 2020-07-17T07:47:45Z

So that's it. However, if WSL2 Docker is launched from the Windows file system, the CPU usage will increase due to another existing problem, so I would like to put the file on WSL2 and use it if possible.

I tried accessing the project on WSL2 mounted as a network drive from VS Code started in Local as a trial and started DevContainer, but the volumes setting described in docker-compose.yml did not work and the container started did not do it.

DavidBurela · 2020-07-21T07:36:12Z

I have this issue on Windows 10 with docker on WSL2, inside of DevContainers.
I can not git pull/push to a SSH git repository.
Confirmed it does not work on another co-worker's machines with WSL2, but works on another's WSL1 machine.

Followed both of these guides
https://code.visualstudio.com/docs/remote/containers#_using-ssh-keys
https://docs.github.com/en/enterprise/2.15/user/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent#adding-your-ssh-key-to-the-ssh-agent

Repro steps:
Linux:

create key with ssh-keygen
added via ssh-add
git clone git@ssh.dev.azure.com:...
started agent eval "$(ssh-agent -s)"
code .
click to reopen in container
use VS Code UI to git pull,
get error Permission denied, please try again.

Windows:

open git bash (MINGW64)
create key with ssh-keygen
added via ssh-add
git clone git@ssh.dev.azure.com:...
started agent eval "$(ssh-agent -s)"
open Powershell and run the commands to start the ssh-agent (i.e. Set-Service ssh-agent -StartupType Automatic, Start-Service ssh-agent, Get-Service ssh-agent)
code .
click to reopen in container
use VS Code UI to git pull,
get error Permission denied, please try again.

If I copy the same SSH key into the container at ~/.ssh then doing a pull in the VS Code UI will succeed.

Extension version: 0.128.0
VS Code version: Code 1.47.2 (17299e413d5590b14ab0340ea477cdd86ff13daf, 2020-07-15T18:22:06.216Z)
OS version: Windows_NT x64 10.0.19041
Remote OS version: Linux x64 4.19.84-microsoft-standard

ATGardner · 2020-07-21T07:45:26Z

I understand the problem, and I am seeing the exact same behavior. But I still don't understand why it matters.

I clone a git repo to my Windows environment (not into WSL).
I open it in VS Code from the Windows environment.
It has a devcontainer.json file, so I reopen it inside a container.

Now my VS Code is running inside a linux container, which is running on my window machine directly. SSH works as expected. The volume mount does not cross WSL<->windows boundaries. Does mounting from Windows into a "standard" Linux container also have some performance cost?
If it does - well, the best solution anyway (IMO) is to clone into a container directly, and don't have any mounted volumes at all. As long as I do it from VS Code on Windows (and not while remote-connected to WSL), the SSH-Agent works as expected.

update:
Just to clarify - it works well from my Windows machine. When you run git pull from your Windows Powershell prompt, it works? @DavidBurela

J0F3 · 2020-07-21T09:12:23Z

@ATGardner

Now my VS Code is running inside a linux container, which is running on my window machine directly.

With WSL 2 the docker daemon is runny actually always in WSL (in its own docker-desktop Distribution) and so any Linux container will actually run also inside the WSL (not on windows directly). Beside the performance impact of mounting the Windows Filesystem to the Container inside the WSL I also had a lot of permission problem when not running as root in the container. (The files from Windows belongs always to root).

The best practices from docker is also to store your code on the Linux (WSL) filesystem: https://docs.docker.com/docker-for-windows/wsl/#best-practices

If it does - well, the best solution anyway (IMO) is to clone into a container directly, and don't have any mounted volumes at all.

That makes not a lot of sense for me. For me the whole point of the dev container is to open an already cloned repo inside a containerized dev environment. If I have to (re-)clone everything again the inside the container first I would lose actually a lot of the advantages of the dev containers.

ATGardner · 2020-07-21T09:30:41Z

Thanks for explaining that any docker Linux mount to Windows will not be as performant.

Regarding the 2nd point - that's a valid one. But I just think that slowly clearing my USERPROFILE\sources\ folder from locally cloned repositories, and switching over to having the same project cloned directly as Docker containers is a pretty neat way to keep things organized. It's true that it will take some time and getting used to. But I don't see any major drawbacks to this approach (though I'm sure there are use-cases against it).

kaito3desuyo · 2020-07-22T03:49:15Z

If it does - well, the best solution anyway (IMO) is to clone into a container directly, and don't have any mounted volumes at all.

That's true, but unfortunately it's actually difficult to have no mounted volumes at all.
It's important to be able to seamlessly sync files on a container to local files.
This keeps git consistent.

issacg · 2020-07-22T15:14:53Z

If you are using the ssh-agent in WSL: How are you starting it and how do you pass SSH_AUTH_SOCK around?

I am using npiperelay

I start it from .zshrc as follows:

if ! grep docker /proc/1/cgroup -qa; then
    export SSH_AUTH_SOCK=$HOME/.ssh/agent.sock
    ss -a | grep -q $SSH_AUTH_SOCK
    if [ $? -ne 0 ]; then
            rm -f $SSH_AUTH_SOCK
            (setsid socat UNIX-LISTEN:$SSH_AUTH_SOCK,fork EXEC:"$HOME/.ssh/npiperelay.exe -ei -s //./pipe/openssh-ssh-agent",nofork & )  >/dev/null 2>&1
    fi
fi

Because I often share my $HOME/.config with containers (so I can keep my Zsh and oh-my-zsh stuff consistent), I wrap it all with the check to run it on WSL (which can run an .exe) and in a container (which can't)

tomasaschan · 2020-07-23T20:06:25Z

I also use keychain to manage SSH agents in WSL. If I do this (in my WSL2 distro), it seems that the active agent socket is always the last one listed:

 λ ls -l --time-style=full-iso /tmp/ssh-*/agent* | sort -k6,7
srw------- 1 taschan taschan 0 2020-05-12 22:38:57.610000000 +0200 /tmp/ssh-XQPxrwqLY3uH/agent.2269
srw------- 1 taschan taschan 0 2020-05-13 22:33:02.780000000 +0200 /tmp/ssh-Ijl5B5vZXfVO/agent.81
srw------- 1 taschan taschan 0 2020-05-14 21:54:16.960000000 +0200 /tmp/ssh-HSWG4KXf01xG/agent.98
srw------- 1 taschan taschan 0 2020-05-31 12:29:20.860000000 +0200 /tmp/ssh-EE2ei9WquTyN/agent.44
srw------- 1 taschan taschan 0 2020-06-02 21:03:58.660000000 +0200 /tmp/ssh-W5ly2kvhlEU7/agent.44
srw------- 1 taschan taschan 0 2020-06-10 21:50:18.909529000 +0200 /tmp/ssh-EDHb8IIwr7so/agent.58
srw------- 1 taschan taschan 0 2020-06-25 22:52:30.640000000 +0200 /tmp/ssh-gkeuHmjTyBZQ/agent.105
srw------- 1 taschan taschan 0 2020-06-26 19:21:13.440000000 +0200 /tmp/ssh-hnUkJFBMZ1hn/agent.44
srw------- 1 taschan taschan 0 2020-06-26 21:31:03.381634300 +0200 /tmp/ssh-MCgqvMjEk6NC/agent.106
srw------- 1 taschan taschan 0 2020-06-26 22:27:27.720000000 +0200 /tmp/ssh-FhIK5icgrw3w/agent.44
srw------- 1 taschan taschan 0 2020-06-26 23:17:07.090000000 +0200 /tmp/ssh-XHy3dB4A33SI/agent.44
srw------- 1 taschan taschan 0 2020-07-22 17:28:24.440000000 +0200 /tmp/ssh-lKUaVzGHImu5/agent.44
srw------- 1 taschan taschan 0 2020-07-23 00:00:21.987419900 +0200 /tmp/ssh-WjOKbAnTuYjA/agent.74
srw------- 1 taschan taschan 0 2020-07-23 14:12:47.440000000 +0200 /tmp/ssh-6mRO0MJUssi6/agent.44
srw------- 1 taschan taschan 0 2020-07-23 20:22:11.932017100 +0200 /tmp/ssh-UeppqrcrltVK/agent.78

The Devcontainer Startup Logs (expand to view in full) show some stuff about GPG, but nothing about SSH-agent. Also quite a lot of `exit code 1`s, which I don't know if they are a problem indicator or not.

[111 ms] Start: Resolving remote
[114 ms] Start: Run: wsl -d Ubuntu-20.04 -e wslpath -u \\wsl$\Ubuntu-20.04\home\taschan\code\github\dandeliondeathray\niancat-haskell
[431 ms] Start: Run: wsl -d Ubuntu-20.04 -e /bin/sh -c cd '/home/taschan/code/github/dandeliondeathray/niancat-haskell' && printenv
[623 ms] Setting up container for folder or workspace: /home/taschan/code/github/dandeliondeathray/niancat-haskell

[805 ms] Start: Run: wsl -d Ubuntu-20.04 -e /bin/sh -c cd '/home/taschan/code/github/dandeliondeathray/niancat-haskell' && test '-f' '/home/taschan/code/github/dandeliondeathray/niancat-haskell/.devcontainer/devcontainer.json'
[1108 ms] Start: Run: wsl -d Ubuntu-20.04 -e /bin/sh -c cd '/home/taschan/code/github/dandeliondeathray/niancat-haskell' && cat '/home/taschan/code/github/dandeliondeathray/niancat-haskell/.devcontainer/devcontainer.json'
[1524 ms] Start: Run: wsl -d Ubuntu-20.04 -e /bin/sh -c cd '/home/taschan/code/github/dandeliondeathray/niancat-haskell' && git 'rev-parse' '--show-toplevel'
[1977 ms] Start: Check Docker is running
[1978 ms] Start: Run: wsl -d Ubuntu-20.04 -e /bin/sh -c cd '/home/taschan/code/github/dandeliondeathray/niancat-haskell' && docker 'info'
[4887 ms] Start: Run: wsl -d Ubuntu-20.04 -e /bin/sh -c cd '/home/taschan/code/github/dandeliondeathray/niancat-haskell' && docker 'ps' '-q' '-a' '--filter' 'label=vsch.local.folder=\\wsl$\Ubuntu-20.04\home\taschan\code\github\dandeliondeathray\niancat-haskell' '--filter' 'label=vsch.quality=stable'
[5537 ms] Start: Run: wsl -d Ubuntu-20.04 -e /bin/sh -c cd '/home/taschan/code/github/dandeliondeathray/niancat-haskell' && docker 'inspect' '--type' 'container' 'a1227e1431e9'
[5907 ms] Start: Starting container
[5907 ms] Start: Run: wsl -d Ubuntu-20.04 -e /bin/sh -c cd '/home/taschan/code/github/dandeliondeathray/niancat-haskell' && docker 'start' 'a1227e1431e9bff048ea15c95635e49d8417eb0b2e1b4b77a313b436bf40b748'
[7186 ms] Start: Run: wsl -d Ubuntu-20.04 -e /bin/sh -c cd '/home/taschan/code/github/dandeliondeathray/niancat-haskell' && docker 'ps' '-q' '-a' '--filter' 'label=vsch.local.folder=\\wsl$\Ubuntu-20.04\home\taschan\code\github\dandeliondeathray\niancat-haskell' '--filter' 'label=vsch.quality=stable'
[7463 ms] Start: Run: wsl -d Ubuntu-20.04 -e /bin/sh -c cd '/home/taschan/code/github/dandeliondeathray/niancat-haskell' && docker 'inspect' '--type' 'container' 'a1227e1431e9'
[7714 ms] Start: Inspecting container
[7714 ms] Start: Run: wsl -d Ubuntu-20.04 -e /bin/sh -c cd '/home/taschan/code/github/dandeliondeathray/niancat-haskell' && docker 'inspect' '--type' 'container' 'a1227e1431e9bff048ea15c95635e49d8417eb0b2e1b4b77a313b436bf40b748'
[8001 ms] Start: Run: wsl -d Ubuntu-20.04 -e /bin/sh -c cd '/home/taschan/code/github/dandeliondeathray/niancat-haskell' && docker 'exec' '-i' '-u' 'dev' '-e' 'VSCODE_REMOTE_CONTAINERS_SESSION=0fea16de-b436-48a3-b854-c82e4e3065571595582068873' 'a1227e1431e9bff048ea15c95635e49d8417eb0b2e1b4b77a313b436bf40b748' '/bin/sh'
[8010 ms] Start: Run in container: uname -m
[8457 ms] x86_64
[8458 ms] 
[8458 ms] Start: Run in container: (cat /etc/os-release || cat /usr/lib/os-release) 2>/dev/null
[8462 ms] NAME="Ubuntu"
VERSION="20.04 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
[8462 ms] 
[8463 ms] Start: Run in container: cat /etc/passwd
[8474 ms] Start: Setup shutdown monitor
[8497 ms] Forking shutdown monitor: c:\Users\tomas\.vscode\extensions\ms-vscode-remote.remote-containers-0.128.0\dist\shutdownMonitorProcess \\.\pipe\vscode-remote-containers-d6c333c8ab8bb1543e5e42329c688fceda9c1d2e-sock singleContainer Info c:\Users\tomas\AppData\Roaming\Code\logs\20200723T174433\exthost3\ms-vscode-remote.remote-containers
[8512 ms] Start: Run in container: test -d /home/dev/.vscode-server
[8515 ms] 
[8515 ms] 
[8519 ms] Start: Run in container: set -o noclobber ; mkdir -p '/home/dev/.vscode-server/data/Machine' && { > '/home/dev/.vscode-server/data/Machine/.writeMachineSettingsMarker' ; } 2> /dev/null
[8524 ms] 
[8525 ms] 
[8525 ms] Exit code 2
[8586 ms] Start: Run in container: test -d /home/dev/.vscode-server/bin/17299e413d5590b14ab0340ea477cdd86ff13daf
[8589 ms] 
[8589 ms] 
[8590 ms] Start: Launching Remote-Containers helper.
[8590 ms] Start: Run: wsl -d Ubuntu-20.04 -e /bin/sh -c cd '/home/taschan/code/github/dandeliondeathray/niancat-haskell' && gpgconf '--list-dir' 'agent-extra-socket'
[8887 ms] Start: Run in container: gpgconf --list-dir agent-socket
[8892 ms] /home/dev/.gnupg/S.gpg-agent
[8892 ms] 
[8892 ms] Start: Run in container: gpgconf --list-dir homedir
[8896 ms] /home/dev/.gnupg
[8897 ms] 
[8897 ms] Start: Run in container: ls '/home/dev/.gnupg/private-keys-v1.d' 2>/dev/null
[8901 ms] 
[8901 ms] 
[8901 ms] Exit code 2
[8901 ms] Start: Run in container: mkdir -p -m 700 '/home/dev/.gnupg'
[8904 ms] Start: Run: wsl -d Ubuntu-20.04 -e /bin/sh -c cd '/home/taschan/code/github/dandeliondeathray/niancat-haskell' && cat '/home/taschan/.gitconfig'
[8911 ms] 
[8911 ms] 
[8911 ms] Start: Run: wsl -d Ubuntu-20.04 -e /bin/sh -c cd '/home/taschan/code/github/dandeliondeathray/niancat-haskell' && gpgconf '--list-dir' 'homedir'
[8983 ms] Start: Run in container: cat <<'EOF-/tmp/vscode-remote-containers-057b10a3bc932788c9382f2de3ef5815d2ae0929.js' >/tmp/vscode-remote-containers-057b10a3bc932788c9382f2de3ef5815d2ae0929.js
[8989 ms] 
[8989 ms] 
[8989 ms] Start: Run in container: cat <<'EOF-/tmp/vscode-remote-containers-server-057b10a3bc932788c9382f2de3ef5815d2ae0929.js' >/tmp/vscode-remote-containers-server-057b10a3bc932788c9382f2de3ef5815d2ae0929.js
[8993 ms] 
[8994 ms] 
[8994 ms] Start: Run: wsl -d Ubuntu-20.04 -e /bin/sh -c cd '/home/taschan/code/github/dandeliondeathray/niancat-haskell' && docker 'exec' '-i' '-u' 'dev' '-e' 'REMOTE_CONTAINERS_SOCKETS=["/home/dev/.gnupg/S.gpg-agent"]' '-e' 'REMOTE_CONTAINERS_IPC=/tmp/vscode-remote-containers-ipc-057b10a3bc932788c9382f2de3ef5815d2ae0929.sock' 'a1227e1431e9bff048ea15c95635e49d8417eb0b2e1b4b77a313b436bf40b748' '/home/dev/.vscode-server/bin/17299e413d5590b14ab0340ea477cdd86ff13daf/node' '/tmp/vscode-remote-containers-server-057b10a3bc932788c9382f2de3ef5815d2ae0929.js'
[9338 ms] Start: Run in container: # Test for /home/dev/.gitconfig and git
[9342 ms] /home/dev/.gitconfig exists
[9342 ms] 
[9342 ms] Exit code 1
[9343 ms] Start: Run in container: command -v git >/dev/null 2>&1 && git config --global credential.helper '!f() { /home/dev/.vscode-server/bin/17299e413d5590b14ab0340ea477cdd86ff13daf/node /tmp/vscode-remote-containers-057b10a3bc932788c9382f2de3ef5815d2ae0929.js $*; }; f' || true
[9349 ms] 
[9349 ms] 
[9349 ms] Start: Run: wsl -d Ubuntu-20.04 -e /bin/sh -c cd '/home/taschan/code/github/dandeliondeathray/niancat-haskell' && cat '/home/taschan/.ssh/known_hosts'
[9406 ms] Start: Run in container: gpgconf --list-dir homedir
[9413 ms] /home/dev/.gnupg
[9413 ms] 
[9414 ms] Start: Run: wsl -d Ubuntu-20.04 -e /bin/sh -c cd '/home/taschan/code/github/dandeliondeathray/niancat-haskell' && cat '/home/taschan/.gnupg/pubring.kbx'
[9667 ms] Start: Run in container: # Test for /home/dev/.ssh/known_hosts and ssh
[9670 ms] /home/dev/.ssh/known_hosts exists
[9670 ms] 
[9670 ms] Exit code 1
[9671 ms] Start: Run in container: set -o noclobber ; mkdir -p '/home/dev/.vscode-server/data/Machine' && { > '/home/dev/.vscode-server/data/Machine/.installExtensionsMarker' ; } 2> /dev/null
[9675 ms] 
[9675 ms] 
[9675 ms] Exit code 2
[9680 ms] Start: Run in container: for pid in `cd /proc && ls -d [0-9]*`; do { echo $pid ; readlink -f /proc/$pid/cwd ; xargs -0 < /proc/$pid/environ ; xargs -0 < /proc/$pid/cmdline ; } ; echo ; done 2>/dev/null
[9722 ms] Start: Run in container: /home/dev/.vscode-server/bin/17299e413d5590b14ab0340ea477cdd86ff13daf/server.sh --disable-user-env-probe --use-host-proxy --port 0 --extensions-download-dir /home/dev/.vscode-server/extensionsCache
[9764 ms] Start: Run in container: # Test for /home/dev/.gnupg/pubring.kbx and gpg
[9767 ms] /home/dev/.gnupg/pubring.kbx exists
[9767 ms] 
[9767 ms] Exit code 1
[9767 ms] Start: Run: wsl -d Ubuntu-20.04 -e /bin/sh -c cd '/home/taschan/code/github/dandeliondeathray/niancat-haskell' && cat '/home/taschan/.gnupg/trustdb.gpg'
[10065 ms] Start: Run: wsl -d Ubuntu-20.04 -e /bin/sh -c cd '/home/taschan/code/github/dandeliondeathray/niancat-haskell' && gpg-connect-agent 'updatestartuptty' '/bye'
[10640 ms] 

*
* Visual Studio Code Server
*
* Reminder: You may only use this software with Visual Studio family products,
* as described in the license https://aka.ms/vscode-remote/license
*


[10718 ms] IP Address: 172.17.0.2
[10725 ms] Extension host agent listening on 35019
[10725 ms] Start: Run in container: echo 35019 >/home/dev/.vscode-server/bin/17299e413d5590b14ab0340ea477cdd86ff13daf/.devport
[10727 ms] 

[09:14:41] Extension host agent started.
[10728 ms] 
[10728 ms] 
[10731 ms] Forwarding local port 35019 to container port 35019
[10733 ms] 

--------------------------
Published Ports:
3000/tcp -> 127.0.0.1:3000
--------------------------

[10828 ms] Start: Run: wsl -d Ubuntu-20.04 -e /bin/sh -c cd '/home/taschan/code/github/dandeliondeathray/niancat-haskell' && docker 'exec' '-i' '-u' 'dev' '-e' 'VSCODE_REMOTE_CONTAINERS_SESSION=0fea16de-b436-48a3-b854-c82e4e3065571595582068873' 'a1227e1431e9bff048ea15c95635e49d8417eb0b2e1b4b77a313b436bf40b748' '/home/dev/.vscode-server/bin/17299e413d5590b14ab0340ea477cdd86ff13daf/node' '-e' '
[10829 ms] Start: Run: wsl -d Ubuntu-20.04 -e /bin/sh -c cd '/home/taschan/code/github/dandeliondeathray/niancat-haskell' && docker 'exec' '-i' '-u' 'dev' '-e' 'VSCODE_REMOTE_CONTAINERS_SESSION=0fea16de-b436-48a3-b854-c82e4e3065571595582068873' 'a1227e1431e9bff048ea15c95635e49d8417eb0b2e1b4b77a313b436bf40b748' '/home/dev/.vscode-server/bin/17299e413d5590b14ab0340ea477cdd86ff13daf/node' '-e' '
[11716 ms] [09:14:42] [::ffff:127.0.0.1][a00b3795][ExtensionHostConnection] New connection established.
[11723 ms] [09:14:42] [::ffff:127.0.0.1][a00b3795][ExtensionHostConnection] <139> Launched Extension Host Process.
[11782 ms] [09:14:42] [::ffff:127.0.0.1][46a684ee][ManagementConnection] New connection established.

JP-Dhabolt · 2020-07-31T17:47:26Z

I am also running into this problem (ssh-agent not being forwarded to remote-container when launched from WSL2). I believe the root problem is related to another issue I have seen (#3086). It would seem that the way that remote-containers invokes docker for WSL2 based repos does not provide for a way to load a profile, hence no environment variables are set (SSH_AUTH_SOCK in this case).

chrmarti · 2020-08-03T09:41:15Z

Added a fix to extract the environment variables from a login shell (#3456). Looking into forwarding to a socket in WSL now.

chrmarti · 2020-08-03T13:22:44Z

The fix to connect to a WSL socket requires socat to be installed in the WSL distro. I'm adding a notification saying so in case it is not installed.

Hebilicious · 2020-08-27T19:14:11Z

@chrmarti By any chance do you know how to properly use SSH_AUTH_SOCK with WSL2 and docker (Using docker for windows wsl2 backend) ?
This approach should work out of the box : https://gist.github.com/d11wtq/8699521 with Ubuntu, but I'm not sure if there's anything special to do with WSL2.

I couldnt find anything in the official docs, but this issue has more information : docker/compose#551

The docker for mac docs mention macOS specific things though : https://docs.docker.com/docker-for-mac/osxfs/#ssh-agent-forwarding

I've tried to do it, and I'm able to echo $SSH_AUTH_SOCK inside the container, but I'm getting some ssh errors. If you could point me in the right direction that would be great !

chrmarti · 2020-08-28T07:26:54Z

@Hebilicious I haven't tried that, best to ask on StackOverflow. Alternatively you could use the forwarding I have implemented, just make sure that your WSL shell's login script set SSH_AUTH_SOCK and have socat installed in the WSL distro.

chrmarti self-assigned this May 4, 2020

chrmarti added this to the May 2020 milestone May 4, 2020

chrmarti added bug Issue identified by VS Code Team member as probable bug containers Issue in vscode-remote containers labels May 4, 2020

chrmarti modified the milestones: May 2020, June 2020 Jun 4, 2020

chrmarti mentioned this issue Jun 19, 2020

No GPG forwarding if using a remote container from within WSL2 #3191

Closed

chrmarti modified the milestones: June 2020, July 2020 Jul 2, 2020

lbssousa mentioned this issue Jul 20, 2020

WSL2 + Remote Containers -- failing to forward host ssh-agent #3392

Closed

chrmarti closed this as completed Aug 3, 2020

connor4312 added the verified Verification succeeded label Aug 7, 2020

chrmarti changed the title ~~DevContainer: SSH auth not working with WSL2~~ DevContainer: SSH und GPG agent forwarding for WSL2 Aug 10, 2020

frozenbonito mentioned this issue Sep 4, 2020

Could not clone private dotfiles repo to dev containers with WSL2 #3630

Closed

github-actions bot locked and limited conversation to collaborators Sep 17, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DevContainer: SSH und GPG agent forwarding for WSL2 #2925

DevContainer: SSH und GPG agent forwarding for WSL2 #2925

chrmarti commented May 4, 2020

chrmarti commented Jun 16, 2020

J0F3 commented Jun 18, 2020

biltongza commented Jun 27, 2020 •

edited

Loading

lbssousa commented Jul 1, 2020 •

edited

Loading

kaito3desuyo commented Jul 17, 2020 •

edited

Loading

ATGardner commented Jul 17, 2020

kaito3desuyo commented Jul 17, 2020

ATGardner commented Jul 17, 2020

kaito3desuyo commented Jul 17, 2020 •

edited

Loading

DavidBurela commented Jul 21, 2020

ATGardner commented Jul 21, 2020 •

edited

Loading

J0F3 commented Jul 21, 2020

ATGardner commented Jul 21, 2020

kaito3desuyo commented Jul 22, 2020 •

edited

Loading

issacg commented Jul 22, 2020

tomasaschan commented Jul 23, 2020 •

edited

Loading

JP-Dhabolt commented Jul 31, 2020

chrmarti commented Aug 3, 2020

chrmarti commented Aug 3, 2020 •

edited

Loading

Hebilicious commented Aug 27, 2020

chrmarti commented Aug 28, 2020

DevContainer: SSH und GPG agent forwarding for WSL2 #2925

DevContainer: SSH und GPG agent forwarding for WSL2 #2925

Comments

chrmarti commented May 4, 2020

chrmarti commented Jun 16, 2020

J0F3 commented Jun 18, 2020

biltongza commented Jun 27, 2020 • edited Loading

lbssousa commented Jul 1, 2020 • edited Loading

kaito3desuyo commented Jul 17, 2020 • edited Loading

ATGardner commented Jul 17, 2020

kaito3desuyo commented Jul 17, 2020

ATGardner commented Jul 17, 2020

kaito3desuyo commented Jul 17, 2020 • edited Loading

DavidBurela commented Jul 21, 2020

ATGardner commented Jul 21, 2020 • edited Loading

J0F3 commented Jul 21, 2020

ATGardner commented Jul 21, 2020

kaito3desuyo commented Jul 22, 2020 • edited Loading

issacg commented Jul 22, 2020

tomasaschan commented Jul 23, 2020 • edited Loading

JP-Dhabolt commented Jul 31, 2020

chrmarti commented Aug 3, 2020

chrmarti commented Aug 3, 2020 • edited Loading

Hebilicious commented Aug 27, 2020

chrmarti commented Aug 28, 2020

biltongza commented Jun 27, 2020 •

edited

Loading

lbssousa commented Jul 1, 2020 •

edited

Loading

kaito3desuyo commented Jul 17, 2020 •

edited

Loading

kaito3desuyo commented Jul 17, 2020 •

edited

Loading

ATGardner commented Jul 21, 2020 •

edited

Loading

kaito3desuyo commented Jul 22, 2020 •

edited

Loading

tomasaschan commented Jul 23, 2020 •

edited

Loading

chrmarti commented Aug 3, 2020 •

edited

Loading