postCreateCommand doesn't seem to run when using a named volume for ~/.vscode-server

[tomasaschan]

Issue Type: Bug

Even though I've added a postCreateCommand entry to my devcontainers.json, the command does not seem to run at all.

Steps to Reproduce:

1. Follow instructions for sharing git credentials with the container
2. Open the workspace for remote development
3. cd root && ls -a shows .ssh-localhost, but not .ssh 🐛

I see no error messages in the logs. I've tried triggering "Remote Development: Rebuild Container" as well as deleting both container and image before reloading the window, but get the same result anyway.

(For reference: this was cross-posted on stackoverflow.com, but hasn't attracted any attention there.)

Extension version: 0.67.3
VS Code version: Code 1.37.0 (036a6b1d3ac84e5ca96a17a44e63a87971f8fcc8, 2019-08-08T02:33:50.993Z)
OS version: Windows_NT x64 10.0.17763
Remote OS version: Linux x64 4.9.184-linuxkit

System Info

Item	Value
CPUs	Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz (8 x 2794)
GPU Status	2d_canvas: enabled flash_3d: enabled flash_stage3d: enabled flash_stage3d_baseline: enabled gpu_compositing: enabled multiple_raster_threads: enabled_on native_gpu_memory_buffers: disabled_software oop_rasterization: disabled_off protected_video_decode: unavailable_off rasterization: enabled skia_deferred_display_list: disabled_off skia_renderer: disabled_off surface_synchronization: enabled_on video_decode: enabled viz_display_compositor: disabled_off webgl: enabled webgl2: enabled
Load (avg)	undefined
Memory (System)	15.91GB (8.61GB free)
Process Argv
Screen Reader	no
VM	0%

Item	Value
Remote	Dev Container
OS	Linux x64 4.9.184-linuxkit
CPUs	Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz (2 x 2644)
Memory (System)	1.93GB (1.03GB free)
VM	0%

[chrmarti]

Could you append the output from the Dev Container terminal? Note that the postCreateCommand is only run on the very first startup on a container, after that it is skipped (also when you add the postCreateCommand later).

[tomasaschan]

@chrmarti Thanks for looking into this.

Here's the output from the Dev Container terminal, on first remote open of the workspace after cleaning out all containers and images generated by VS Code.

Full log

Setting up container for folder: d:\Projects\******\customer-portal-v2\backend
Run: docker build -f d:\Projects\******\customer-portal-v2\backend\.devcontainer\Dockerfile -t vsc-backend-b53ee5aafe20fdf6fc497e5b0e33bb76 d:\Projects\******\customer-portal-v2\backend\.devcontainer
Sending build context to Docker daemon  6.144kB
Step 1/6 : FROM python:stretch
 ---> 71c6ddda7cab
Step 2/6 : WORKDIR /workspace
 ---> Using cache
 ---> 65e0e5d5f602
Step 3/6 : RUN apt-get update -qq &&     echo "Installing package metadata tooling..." &&     apt-get install -yqq apt-transport-https ca-certificates curl gnupg-agent software-properties-common lsb-release &&     echo "Adding additional repositories to apt-get..." &&     curl -fsSL https://download.docker.com/linux/$(lsb_release -is | tr '[:upper:]' '[:lower:]')/gpg | apt-key add - 2>/dev/null &&     add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/$(lsb_release -is | tr '[:upper:]' '[:lower:]') $(lsb_release -cs) stable" &&     echo "deb https://deb.nodesource.com/node_10.x $(lsb_release -cs) main" > /etc/apt/sources.list.d/nodesource.list &&     curl -fsSL https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add - &&     apt-get update -qq &&     echo "Updating pip..." &&     pip install -U pip && pip install pipenv &&     echo "Installing nodejs and Docker..." &&     apt-get install -yqq nodejs python docker-ce-cli &&     curl -sSL "https://github.c
om/docker/compose/releases/download/1.24.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose &&     chmod +x /usr/local/bin/docker-compose &&     echo "Installing AWS SAM CLI..." &&     pip3 install awscli aws-sam-cli --upgrade &&     echo "Installing additional tools..." &&     apt-get install -yqq git zip groff locales &&     apt-get clean &&     locale-gen en_US.UTF-8
 ---> Using cache
 ---> b90a1bba50d3
Step 4/6 : RUN npm install --global parcel-bundler typescript
 ---> Using cache
 ---> c51fe753b59e
Step 5/6 : ADD .gitconfig /root/.gitconfig
 ---> Using cache
 ---> fb38ea5b3dce
Step 6/6 : CMD 'bash'
 ---> Using cache
 ---> 94c0b7f218f6
Successfully built 94c0b7f218f6
Successfully tagged vsc-backend-b53ee5aafe20fdf6fc497e5b0e33bb76:latest
SECURITY WARNING: You are building a Docker image from Windows against a non-Windows Docker host. All files and directories added to build context will have '-rwxr-xr-x' permissions. It is recommended to double check and reset permissions for sensitive files and directories.
Run: docker run -a STDOUT -a STDERR --mount type=bind,source=d:/Projects/******/customer-portal-v2/backend,target=/workspaces/backend -l vsch.quality=stable -l vsch.local.folder=d:\Projects\******\customer-portal-v2\backend -l vsch.remote.devPort=0 -v /var/run/docker.sock:/var/run/docker.sock -v ******-backend-extensions:/root/.vscode-server -v C:\Users\Tomas Aschan/.ssh:/root/.ssh-localhost:ro --entrypoint /bin/sh vsc-backend-b53ee5aafe20fdf6fc497e5b0e33bb76 -c echo Container started ;  while sleep 1; do :; done
Run: docker exec eade48d9e6da67a304148d3b139c800b3c5af203f7a26088c4943d924214e833 /bin/sh -c (cat /etc/os-release || cat /usr/lib/os-release) 2>/dev/null
Container started
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
Run: docker cp eade48d9e6da67a304148d3b139c800b3c5af203f7a26088c4943d924214e833:/etc/passwd -
Run: docker exec eade48d9e6da67a304148d3b139c800b3c5af203f7a26088c4943d924214e833 test -d /root/.vscode-server
Run: docker exec eade48d9e6da67a304148d3b139c800b3c5af203f7a26088c4943d924214e833 /bin/sh -c set -o noclobber ; mkdir -p '/root/.vscode-server/data/Machine' && { > '/root/.vscode-server/data/Machine/.writeMachineSettingsMarker' ; } 2> /dev/null
Run: docker exec eade48d9e6da67a304148d3b139c800b3c5af203f7a26088c4943d924214e833 /bin/sh -c set -o noclobber ; mkdir -p '/root/.vscode-server/data/Machine' && { > '/root/.vscode-server/data/Machine/.copyGitConfigMarker' ; } 2> /dev/null
Run: docker exec eade48d9e6da67a304148d3b139c800b3c5af203f7a26088c4943d924214e833 test -d /root/.vscode-server/bin/036a6b1d3ac84e5ca96a17a44e63a87971f8fcc8
Run: docker exec eade48d9e6da67a304148d3b139c800b3c5af203f7a26088c4943d924214e833 test -d /root/.vscode-server/extensions
Run: docker exec eade48d9e6da67a304148d3b139c800b3c5af203f7a26088c4943d924214e833 /bin/sh -c set -o noclobber ; mkdir -p '/root/.vscode-server/data/Machine' && { > '/root/.vscode-server/data/Machine/.postCreateCommandMarker' ; } 2> /dev/null
Run: docker exec -w / -u 0 eade48d9e6da67a304148d3b139c800b3c5af203f7a26088c4943d924214e833 /bin/sh -c command -v git >/dev/null 2>&1 && git config --system credential.helper '!f() { command -v code >/dev/null 2>&1 && code --gitCredential $*; }; f' || true
Run: docker exec -w /root/.vscode-server/bin/036a6b1d3ac84e5ca96a17a44e63a87971f8fcc8 -e SHELL=/bin/bash -e VSCODE_AGENT_FOLDER=/root/.vscode-server eade48d9e6da67a304148d3b139c800b3c5af203f7a26088c4943d924214e833 /root/.vscode-server/bin/036a6b1d3ac84e5ca96a17a44e63a87971f8fcc8/server.sh --disable-user-env-probe --port 0


*
* Visual Studio Code Server
*
* Reminder: You may only use this software with Visual Studio family products,
* as described in the license https://aka.ms/vscode-remote/license
*


IP Address: 172.17.0.2
webview server listening on 39069
Extension host agent listening on 46867
Run: docker exec -w /root/.vscode-server/bin/036a6b1d3ac84e5ca96a17a44e63a87971f8fcc8 eade48d9e6da67a304148d3b139c800b3c5af203f7a26088c4943d924214e833 /bin/sh -c echo 46867 >.devport
==> Received a management connection
==> Received an extension host connection.
==> Using UI language: en
Got start params: { language: 'en',
  break: undefined,
  port: undefined,
  debugId: undefined }

(You'll notice that it's using the build cache a lot when building the docker image; that's because I've found it saves me a lot of time to build and tag the image outside of VS Code when I change the Dockerfile, so I've basically done cd .devcontainers && docker build -t workspace-baseline . before opening the workspace.)

[chrmarti]

The output suggests that the container being started already has a /root/.vscode-server/data/Machine/.postCreateCommandMarker file. How did that get there?

[tomasaschan]

Hum; good catch - that's probably because I've set up a local docker volume for the .vscode-server directory, to avoid having to reinstall extensions (according to recommendation here).

After running docker run --rm -v my-volume:/cache alpine rm /cache/data/Machine/.postCreateCommandMarker and recreating the container, I can see that the command is properly run.

I realize this is a tall order, but is there a way to cache extensions between runs, but not cache postCreateCommand execution? Would it be a bad idea to volume-mount e.g. /root/.vscode-server/extensions instead of the config root dir?

[chrmarti]

Volume-mounting the extensions folder should work. Let us know if that works as expected. (@Chuxel We will need to update the hint in the documentation on this.)

[tomasaschan]

@chrmarti Volume mounting the extensions folder seems to make the postCreateCommand run properly, but now extensions are not installing.

If I read the logs correctly, that's because the /root/.vscode-server/extensions folder already exists (since it was created by the volume mount), but I don't understand why that makes it skip installing extensions listed in devcontainer.json - what if I add a new extension to an already non-empty list? Anyway, the extensions folder remains empty despite specifying a number of extensions in the JSON config.

I guess it could be a permissions issue, too, if the volume mounted directory has incorrect permissions, but I don't see any error messages to that effect, and since it worked fine when volume-mounting .vscode-server I think that's unlikely to be the issue here.

Manually installing extensions through the UI works, and populates the directory. It's just the list of extensions in devcontainer.json that's ignored.

[chrmarti]

Extensions are installed only once, after that we don't want to update based on "extensions" in the devcontainer.json because the user might have removed an extension through the UI and it would seem unexpected to see that come back.

Currently the existence of the extensions folder is used as the only indication that extensions have been installed already. We could change that.

[tomasaschan]

For this scenario, it would be useful to have a similar marker file (maybe inside the extensions directory?) - that would make it easy to volume-mount the extensions directory, without breaking the detection scheme.

[Chuxel]

@tomasaschan @chrmarti I don't think we want to recommend bind mounting extensions unless you are on the same OS the container is running. (So this would work for Linux users.) Some extensions do download secondary dependencies that are platform specific, so some may not work if they are volume mounted into the container. I know I've seen this in extensions like Live Share.

If you're looking to have the extensions survive a rebuild, you can consider using a named volume instead. This is discussed here along with the note about postCreateCommand: https://code.visualstudio.com/docs/remote/containers-advanced#_avoiding-extension-reinstalls-on-container-rebuild

We also now have a command you can run to install all local extensions in the container if this is what you are looking to do.

[Chuxel]

@tomasaschan @chrmarti Oops!! Apologies - I misread this. You are doing a named volume - not a bind mount I assume?

You can mount just the extensions folder to avoid this particular problem. So:

"runArgs": ["-v","your-volume-name-goes-here:/root/.vscode-server/extensions"]

I can tweak the docs. You can then add the "bin" folder if you want to avoid downloading the server but still reset the data folder.

"runArgs": ["-v","your-volume-name-goes-here-extensions:/root/.vscode-server/extensions", 
                   "-v","your-volume-name-goes-here-bin:/root/.vscode-server/bin"]

It would be a good thought to look at a different spot for the postCreateCommand marker so you could keep your data as well.

[tomasaschan]

@Chuxel @chrmarti Yeah, sorry for my confusing use of incorrect terminology here - I'm not using a bind mount, but a named volume, as you correctly inferred from my config sample.

Mounting a named volume for the extensions folder is a good idea in theory, but because existence of this folder is used as the condition for installing extensions, doing this makes VS Code skip the extension installation step entirely; therefore, I suggest a similar .installExtensions marker file, e.g. in the extensions folder, to allow this scenario to work fully.

A command for bulk installing extensions would be useful, but I think it would make more sense then to install all extensions listed in devcontainer.json, rather than all extensions I happen to have installed locally. (One of my main use cases for remote development with containers is to isolate tool sets between different projects, so I'm aiming for having almost no tooling extensions installed locally...)

Is the code for the startup sequence of a dev container open source? I tried looking through this repo for it, but didn't find anything, but if it's public I could take a stab at a PR to extend this functionality.

[chrmarti]

If the marker file is in the extensions folder it would be persisted in the volume and installation would only run for the first container using that volume. Is that what you are looking for?

No, this is not open source.

[tomasaschan]

If the marker file is in the extensions folder it would be persisted in the volume and installation would only run for the first container using that volume. Is that what you are looking for?

I don't know, to be honest. Does installing an extension require more actions than putting stuff in that folder? If so, it would probably be better to have the marker file somewhere else. Since you were also talking about moving the .postCreateCommand marker out of data, maybe it would be a good idea to create a new directory under .vscode-server which can contain only the marker files? Then user's can opt in or out of caching them between runs as they please.

[chrmarti]

Installing extensions only copies the extensions to that folder. I thought the question might be more whether or not you want your latest list of extensions applied each time you rebuild that container (as opposed to just once when the volume is new).

[tomasaschan]

Hm, that's a good point; if a colleague adds an extension to the config file, I want that to be installed when I open the workspace on my laptop. So I guess I think the marker file should not be in the volume :)

[Chuxel]

@chrmarti @tomasaschan Yeah I played with this a bit - there's actually an extra step here too if you are using a non-root user right now. You need to pre-create the ~/.vscode-server folder in your Dockerfile with the right owner - otherwise the volume is created as root. Unfortunately postCreateCommand is too late, since the server has to be up before it fires. There doesn't appear to be a way to specify using a different user for a new volume, but you can create a volume that points to an existing directory structure which then inherits permissions.

#421 is probably the right answer here - a 1st class feature. The directions are effectively a workaround for this gap, but getting the marker in place would help marker the workaround a bit better. I also agree that the markers should not be in the volume in these cases.

I'll update docs and try to make the current behavior a bit more obvious in the near term - it's called out, but not well enough.

[Chuxel]

Docs have been updated - I'm closing this in favor of 421, which is really the real fix here.