docker-images: explicitly create storage folders in Dockerfile…

[ggilmore]

Part of #7829

See also https://github.com/sourcegraph/infrastructure/pull/1792

Many of our docker images write to external volumes for caches, data storage, etc. The file system permissions are identical from both the container's and host's POV.

For example, let's say you mount a folder (/data) into a container and on the host, only uid 11 has r/w access to it. If the container is running as some other unprivileged user (say uid 22) then the container process can't access the folder. This is the source of all the permission issues/container crashes in #7829.

There are multiple ways to mount volumes in Docker:

• When using bind mounts, what we currently tell people to do by default, if the host folder that's being mounted has mismatched file permissions you're forced to run chmod commands as a workaround. This is hacky, brittle, and might not be possible to run in every environment.

• When using 'named' volumes, the method that's recommended in the official documentation, the volume's initial contents and permissions are propagated from the container. However, this only applies if the path in the container that the volume is being attached to existed beforehand (otherwise, the volume will be attached with root permissions).

This PR ensures that all of the storage paths in our docker images are created ahead of time in their respective Dockerfiles with permissions that are accessible to our "sourcegraph" user.

There was one complication with https://github.com/sourcegraph/sourcegraph/tree/master/docker-images/prometheus. The upstream image that we inherit from explicitly declares a VOLUME in the Dockerfile with permissions that are only writeable via their default "nobody" user. It is not possible to change the permissions on this volume with a Dockerfile that inherits from this image. (Note that using explicit VOLUME's in Dockerfiles has been a long-standing complaint of many base images: docker-library/postgres#404, moby/moby#3465, moby/moby#3465). The workaround used here is to effectively copy the upstream Dockerfile's assets into our own Dockerfile that deliberately omits that VOLUME directive. We'll need to ensure that our Dockerfile is kept in sync whenever we need to upgrade the Prometheus version, but https://github.com/prometheus/prometheus/commits/master/Dockerfile doesn't change all that often in practice.

--- Test plan

I'm pretty confident our users won't need to do any manual migration due to this change. (Our k8s configuration still runs all our images as root, and anyone still using bind mounts shouldn't have to do anything since the existing host folder's permissions still override anything that's set in the container).

However, just to be sure, we'll need to test a fresh kubernetes deployment, the kubernetes upgrade path, and a fresh docker-compose deployment( note that https://github.com/sourcegraph/deploy-sourcegraph-docker/tree/permission depends on changes from this PR). This change doesn't affect sourcegraph/server.

[uwedeportivo]

@ggilmore walked me through these changes yesterday, and his description of this PR is super nice.

i don't have concerns about the prometheus image changes.

LGTM

[emidoots]

I didn't review heavily, but I see this is effectively what me and @ggilmore have been discussing actively over Slack, so LGTM.

[beyang]

LGTM. @ggilmore please own testing these in a k8s cluster before merging (or immediately after merging if that saves time).

[ggilmore]

@uwedeportivo We'll need to release a new version of sourcegraph/prometheus. What should the tag be? It's currently at 2.15.2 but we're not bumping the version of prometheus itself. 2.15.2-1?

[uwedeportivo]

actually sourcegraph/prometheus versioning is decoupled from prometheus/prometheus versioning
it's at 10.0.6 (version.sh in docker-images/prometheus). so 10.0.7 ?

[ggilmore]

The kubernetes upgrade path seems to work fine. I’ll merge this in the morning and cherry pick this into 3.12

…

On Thu, Jan 23, 2020 at 4:44 PM uwedeportivo ***@***.***> wrote: actually sourcegraph/prometheus versioning is decoupled from prometheus/prometheus versioning it's at 10.0.6 (version.sh in docker-images/prometheus). so 10.0.7 ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/sourcegraph/sourcegraph/pull/7991?email_source=notifications&email_token=ACE2UOZ5HZP5AOAINOXF4BDQ7I2V5A5CNFSM4KK4DXX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJZL2VY#issuecomment-577944919>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACE2UOYFRC3CH7NMJIUN33DQ7I2V5ANCNFSM4KK4DXXQ> .