sccache seems to have no effect for Rust docker builds

[oren0e]

Hi,

According to what I understand from the readme here, in order to make sccache work with cargo build --release in Dockerfile I only need to install it using cargo install sccache and then set the environment variable RUSTC_WRAPPER, and then build the project normally. And when the build is triggered for the second time (when there is a cache present) - it should be much faster.

Well, this is not the case for me - maybe I'm doing something wrong. Here is my Dockerfile which I then trigger in Gitlab CI:

FROM rust:1.51 as builder

ENV NIGHTLY_VERSION nightly-2021-08-18
ENV LC_ALL C.UTF-8
ENV LANG C.UTF-8

# Use sccache
RUN cargo install sccache
ENV RUSTC_WRAPPER /usr/local/cargo/bin/sccache
ENV SCCACHE_CACHE_SIZE 90G
ENV SCCACHE_DIR /mnt/dispatcher-cache

RUN rustup override set $NIGHTLY_VERSION
RUN rustup component add rustfmt clippy --toolchain $NIGHTLY_VERSION-x86_64-unknown-linux-gnu
RUN cargo new --bin dispatcherr
WORKDIR ./dispatcherr
COPY ./Cargo.toml ./Cargo.toml
RUN cargo build --release
RUN rm src/*.rs

ADD . ./

RUN rm ./target/release/deps/dispatcherr*
RUN cargo build --release

FROM python:3.9-slim-buster

RUN apt-get update \
    && apt-get install -y ca-certificates tzdata \
    && rm -rf /var/lib/apt/lists/*

RUN pip install scottypy

EXPOSE 8000

RUN mkdir -p /home/dispatcherr
COPY --from=builder /dispatcherr/target/release/dispatcherr /home/dispatcherr
RUN chmod +x /home/dispatcherr

WORKDIR /home/dispatcherr

CMD ["./dispatcherr"]

When I run this stage, each time the compilation takes exactly the same (long) time. When I look at the directory where I defined to save the cache - it is empty. It seems like either I missed some steps from the readme or that sccache just does not work for some reason.
I Would appreciate some help.

[mitchhentges]

Hmm, nothing is jumping out at me that might be incorrect.
It might be worth running cargo -v build to see what it's specifically calling - perhaps it's not invoking sccache.
You could even try manually running /usr/local/cargo/bin/sccache rustc ... (with a rustc command as printed by cargo -v build) to see if that generates files in the cache.

[jschwe]

I'm curious as to why you want to use sccache when building the docker image. Usually you would want to use sccache when running inside the docker image.

Also from what I'm seeing you are first building an empty project generated from cargo new, then copying some stuff over and then rebuilding. If the project files change, sccache can't have cached them.

[oren0e]

I'm curious as to why you want to use sccache when building the docker image. Usually you would want to use sccache when running inside the docker image.

Also from what I'm seeing you are first building an empty project generated from cargo new, then copying some stuff over and then rebuilding. If the project files change, sccache can't have cached them.

I want to save all the building information so in CI/CD later, when I change something that is not the code or dependencies - I won't have to build the image over and over again from scratch. In the Dockerfile above I first build only the dependencies and then rest of the project - if the code changes it is a different thing than when dependencies change.

[jschwe]

That is reasonable, but the dependencies are already cached by cargo (i.e. cargo won't rebuild them), and the result of that step is cached anyway by docker, so I don't see what could gained here by adding sccache to the mix.

[oren0e]

@jschwe you are correct, but at the end of the "docker build" step I purge everything from the CI runner not to leave any traces from the build as a standard procedure. I thought I would be able to use sccache to cache the results into a dedicated mounted place inside the server. You can argue that both options are identical.

[jschwe]

It appears as if you don't save and restore /mnt/dispatcher-cache, which would be needed to restore the local sccache cache. But maybe it would be easier to use the Redis, Memcached, or one of the Cloud storage options to store the cache persistently?

[oren0e]

That is what I did not understand - I thought that merely defining the env var with ENV SCCACHE_DIR /mnt/dispatcher-cache is enough. Using Redis and the like does not improve performance much because of the downloading times.

[lcmgh]

@oren0e have you tried utilizing Docker's buildx cache functionality to persist /mnt/dispatcher-cache on the host?

RUN --mount=type=cache,target=$SCCACHE_DIR cargo build --release

Edit: On my side this did not work so I opened #1475

[ferologics]

did anyone figure out how to make this work? i'm trying the buildx way but it's hard to tell if it's working or not, the mounted sccache directory is empty for whatever reason

# more info on Docker mount-type cache:
# - https://docs.docker.com/build/guide/mounts/
# - https://docs.docker.com/engine/reference/builder/#run---mounttypecache
# - https://docs.docker.com/build/cache/#use-your-package-manager-wisely

# the tag is replaced by the actual tag in the create_contriner.sh build script
FROM instrumentisto/rust:nightly-2023-08-03 as builder

ARG TARGETARCH
ENV APT_CACHE_DIR=/var/cache/apt
ENV APT_LIB_DIR=/var/lib/apt
ENV SCCACHE_VERSION=0.5.4
ENV CARGO_HOME=~/.cargo
# default Linux location from https://github.com/mozilla/sccache/blob/main/docs/Local.md
ENV SCCACHE_DIR=~/.cache/sccache
ENV RUSTC_WRAPPER=/usr/local/bin/sccache

WORKDIR /usr/src/control_center

# install sccache to speed up builds - only happens on n=1 run, will be cached in n+1 runs
RUN --mount=type=cache,id=apt,target=/var/cache/apt,sharing=locked \
  --mount=type=cache,id=apt,target=/var/lib/apt,sharing=locked \
  apt-get update && apt-get install -y wget
RUN rm -rf /var/lib/apt/lists/*
RUN case ${TARGETARCH} in \
        amd64) ARCH=x86_64 ;; \
        arm64) ARCH=aarch64 ;; \
    esac \
    && wget -O sccache.tar.gz https://github.com/mozilla/sccache/releases/download/v${SCCACHE_VERSION}/sccache-v${SCCACHE_VERSION}-${ARCH}-unknown-linux-musl.tar.gz \
    && tar xzf sccache.tar.gz \
    && mv sccache-v*/sccache /usr/local/bin/sccache \
    && chmod +x /usr/local/bin/sccache
RUN rm -rf .cargo/ # remove invalid path to sccache (we use homebrewed sccache for local development)

COPY macros macros
COPY control_center control_center

# build control center
RUN --mount=type=cache,id=sccache,target=${SCCACHE_DIR} \
  --mount=type=cache,id=cargo,target=${CARGO_HOME} \
  cargo build -r --target-dir="/usr/src/cc/target/" \
    --manifest-path control_center/Cargo.toml \
  && sccache --show-stats

# We do not need the Rust toolchain to run the binary!
FROM debian:bullseye-slim
RUN --mount=type=cache,id=apt,target=/var/cache/apt,sharing=locked \
  --mount=type=cache,id=apt,target=/var/lib/apt,sharing=locked \
  apt-get update && apt-get install -y ca-certificates
RUN rm -rf /var/lib/apt/lists/*
COPY --from=builder /usr/src/cc/target/release/control_center /usr/local/bin/control_center
CMD ["control_center"]

[lcmgh]

This works for me:

RUN --mount=type=cache,mode=0777,target=/home/root/.cache/sccache \
    cargo build --release && sccache --show-stats

It also prints stats so you can verify if the cache was being used or not.

What did not work was using a env. variable as target value:

RUN --mount=type=cache,mode=0777,target=${SCCACHE_DIR} \
    cargo build --release && sccache --show-stats

[ferologics]

strange, for me this doesn't work, always getting 0 cache hits, with or without the env variable

#19 112.7    Compiling control_center v0.1.0 (/usr/src/control_center/control_center)
#19 238.5     Finished release [optimized] target(s) in 3m 58s
#19 238.5 Compile requests                    339
#19 238.5 Compile requests executed           273
#19 238.5 Cache hits                            0
#19 238.5 Cache misses                        273
#19 238.5 Cache misses (C/C++)                 50
#19 238.5 Cache misses (Rust)                 223
#19 238.5 Cache timeouts                        0
#19 238.5 Cache read errors                     0
#19 238.5 Forced recaches                       0
#19 238.5 Cache write errors                    0
#19 238.5 Compilation failures                  0
#19 238.5 Cache errors                          0
#19 238.5 Non-cacheable compilations            0
#19 238.5 Non-cacheable calls                  63
#19 238.5 Non-compilation calls                 3
#19 238.5 Unsupported compiler calls            0
#19 238.5 Average cache write               0.001 s
#19 238.5 Average compiler                  1.974 s
#19 238.5 Average cache read hit            0.000 s
#19 238.5 Failed distributed compilations       0
#19 238.5 
#19 238.5 Non-cacheable reasons:
#19 238.5 crate-type                           50
#19 238.5 unknown source language              12
#19 238.5 -                                     1
#19 238.5 
#19 238.5 Cache location                  Local disk: "~/.cache/sccache"
#19 238.5 Version (client)                0.5.4
#19 238.5 Cache size                          220 MiB
#19 238.5 Max cache size                       10 GiB
#19 DONE 238.6s

[ferologics]

perhaps the key is to use the mode=0777 attribute?
edit: adding this didn't help either. 0 cache hits.

[lcmgh]

Did you use buildx?
docker buildx build .

Have you set the wrapper variables in the Dockerfile?

ENV CARGO_HOME=/usr/local/cargo
ENV RUSTC_WRAPPER=/usr/local/cargo/bin/sccache
ENV SCCACHE_DIR=/home/root/.cache/sccache # must be same as the 'target' value for the mount cmd

[ferologics]

@lcmgh yes i am using buildx :) i suspect the mount type cache is not preserved across my CircleCI builds. Perhaps because it's not part of the import/export cache. when testing locally my builds are much faster, but the CI doesn't seem to make use of the cache 🤷 i'd like to have full control over the files in docker cache, but it looks like it's not that straightforward (cache mounts are BuildKit managed and thus cannot control their location. Cannot use bind mounts because read only)

[Xuanwo]

It is recommended to use cloud storage services such as S3 or GCS when using sccache in Docker or CI environments.

[ferologics]

@Xuanwo i will not build the rust executable in the Docker container, and will only copy the final artifact in once its ready