Make images workable with /usr/local/cargo bind-mounted to host's directory

[mzabaluev]

As I explained in my answer to a StackOverflow question, currently one has to carefully mount all subdirectories of /usr/local/cargo that Cargo will use as cache during the crate build, to avoid redownloading the crates.io registry and the dependencies with every container run. This is troublesome also because some of these depend on the choice of the VCS that the dependencies use, and basically need trial-and-error to discover them all.

Bind-mounting the entire /usr/local/cargo is not currently possible, because the official images also install the toolchain binaries under it. It would be much more convenient if the binaries would be installed elsewhere where PATH inside the container would find them, and the whole /usr/local/cargo would be treated as a perishable cache directory (what with it being world-writable).

[sfackler]

I think this is a change that would need to be made in Cargo or rustup itself - rustup is the thing that's installing the toolchain binaries in $CARGO_HOME/bin.

[andre-richter]

If you want a cache approach towards cargo downloads, please have a look at how the redox-os docker image implements it.

There, I've put all of cargo and rustup into docker-volumes, so that stuff is cached between container runs, and can be deleted in one go if needed to start clean. All without interfering with your local rustup/cargo install. The logic is in the run file though, not the Dockerfile.

https://github.com/redox-os/redox/tree/master/docker

[mzabaluev]

@andre-richter Your approach appears to use much more permissions mojo than I was willing to dig into. Volumes that are associated with users, but managed by Docker, certainly seem like a bit of overkill for most users.

A simple fix would be a step at the end of the Rust image's Dockerfile that copies the binaries to /usr/local/bin. There's hardly a need to run rustup inside the container, and for the host, Docker does the job of rustup.

[sfackler]

There's hardly a need to run rustup inside the container

At the very least, it's somewhat common to use rustup in the container to pull in extra targets.

[mzabaluev]

At the very least, it's somewhat common to use rustup in the container to pull in extra targets.

So as to not defeat the purpose of container images, that's better be done once after the Rust image pull, by a Dockerfile that builds an incremental image. In that container run, /usr/local/cargo can be left unmounted, and the installed binaries/libraries copied away afterwards, same as for the mother Rust image build.

It would be ideal if the Rust team provided incremental images for any additional rustup target and feature alongside every Rust image, though I realize it might raise the complexity of the logistics.

[sfackler]

I don't really know what you mean by "defeating the purpose of container images". Am I defeating the purpose of container images here? https://github.com/sfackler/cargo-sls-distribution/blob/master/.circleci/config.yml#L52

It would be ideal if the Rust team provided incremental images for any additional rustup target and feature alongside every Rust image, though I realize it might raise the complexity of the logistics.

There doesn't need to be a base image for every possible use case.

Again, this is a change best handled upstream.

[mzabaluev]

@sfackler

Am I defeating the purpose of container images here?

I'm inclined to say yes: every build installs a lot of off-the-shelf stuff on top of what seems to be a fairly stable base Rust image (not updated since 1.19.0). Unless you absolutely need those Debian packages up to date with every build, all that should be rolled into an image and published in a Docker repo that CircleCI can pull from. I bet that will drop the run time of that job, and the one above it, to a fraction of what it is now. If an incremental image is not practical, at least caching the installed files in the CircleCI infra could avoid a lot of wasted work.

In any case, container runs that install extra dependencies into /usr/local/cargo (presumably not bind-mounted) are not the concern here.

[sfackler]

I bet that will drop the run time of that job, and the one above it, to a fraction of what it is now.

It takes 7 seconds to install the Debian dependencies, 1 second to install the musl target, and 4 seconds to install the Gem. In contrast, the cargo build takes 1 minute and 32 seconds.

If an incremental image is not practical, at least caching the installed files in the CircleCI infra could avoid a lot of wasted work.

Why would downloading these files from an S3 bucket be faster than downloading them out of their normal upstreams?

This is getting wildly off topic. I believe the way forward here is pretty clear.

[mzabaluev]

@sfackler I should have said "setup time", not run time. As I understand, CircleCI does some locally-optimized image caching, so it's not necessarily about pulling them from S3. And incrementally mounting a filesystem image takes far less CPU work than unpacking and installing a lot of files into the container. But yes, this is a digression.