Skip to content

Commit

Permalink
ref(docker): leverage cache mount with bind mounts
Browse files Browse the repository at this point in the history
This update eliminates the need for external tools like cargo-chef to leverage caching layers, resulting in an average build time reduction of 4m30s (~36% improvement).

While this solution doesn't fully resolve the issues mentioned in #6169 (comment), it represents the best possible approach without resorting to custom solutions, which we'd prefer to avoid.
  • Loading branch information
gustavovalverde committed Aug 29, 2024
1 parent cdb9efd commit 35c41a6
Show file tree
Hide file tree
Showing 3 changed files with 248 additions and 86 deletions.
22 changes: 22 additions & 0 deletions README.Docker.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
### Building and running your application

When you're ready, start your application by running:
`docker compose up --build`.

Your application will be available at http://localhost:8233.

### Deploying your application to the cloud

First, build your image, e.g.: `docker build -t myapp .`.
If your cloud uses a different CPU architecture than your development
machine (e.g., you are on a Mac M1 and your cloud provider is amd64),
you'll want to build the image for that platform, e.g.:
`docker build --platform=linux/amd64 -t myapp .`.

Then, push it to your registry, e.g. `docker push myregistry.com/myapp`.

Consult Docker's [getting started](https://docs.docker.com/go/get-started-sharing/)
docs for more detail on building and pushing.

### References
* [Docker's Rust guide](https://docs.docker.com/language/rust/)
159 changes: 73 additions & 86 deletions docker/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
# syntax=docker/dockerfile:1
# check=skip=UndefinedVar

# If you want to include a file in the Docker image, add it to .dockerignore.
#
# We are using five stages:
# - chef: installs cargo-chef
# - planner: computes the recipe file
# - deps: caches our dependencies and sets the needed variables
# We are using 4 stages:
# - deps: install build dependencies and sets the needed variables
# - tests: builds tests
# - release: builds release binary
# - runtime: is our runtime environment
Expand All @@ -20,29 +21,17 @@ ARG TEST_FEATURES="lightwalletd-grpc-tests zebra-checkpoints"
ARG EXPERIMENTAL_FEATURES=""

ARG APP_HOME="/opt/zebrad"
# This stage implements cargo-chef for docker layer caching
FROM rust:bookworm as chef
RUN cargo install cargo-chef --locked

ARG APP_HOME
ENV APP_HOME=${APP_HOME}
WORKDIR ${APP_HOME}

# Analyze the current project to determine the minimum subset of files
# (Cargo.lock and Cargo.toml manifests) required to build it and cache dependencies
#
# The recipe.json is the equivalent of the Python requirements.txt file
FROM chef AS planner
COPY . .
RUN cargo chef prepare --recipe-path recipe.json

ARG RUST_VERSION=1.79.0
# In this stage we download all system requirements to build the project
#
# It also captures all the build arguments to be used as environment variables.
# We set defaults for the arguments, in case the build does not include this information.
FROM chef AS deps
FROM rust:${RUST_VERSION}-bookworm AS deps
SHELL ["/bin/bash", "-xo", "pipefail", "-c"]
COPY --from=planner ${APP_HOME}/recipe.json recipe.json

ARG APP_HOME
ENV APP_HOME=${APP_HOME}
WORKDIR ${APP_HOME}

# Install zebra build deps and Dockerfile deps
RUN apt-get -qq update && \
Expand All @@ -52,27 +41,8 @@ RUN apt-get -qq update && \
clang \
ca-certificates \
protobuf-compiler \
rsync \
rocksdb-tools \
; \
rm -rf /var/lib/apt/lists/* /tmp/*

# Install google OS Config agent to be able to get information from the VMs being deployed
# into GCP for integration testing purposes, and as Mainnet nodes
# TODO: this shouldn't be a hardcoded requirement for everyone
RUN if [ "$(uname -m)" != "aarch64" ]; then \
apt-get -qq update && \
apt-get -qq install -y --no-install-recommends \
curl \
lsb-release \
&& \
echo "deb http://packages.cloud.google.com/apt google-compute-engine-$(lsb_release -cs)-stable main" > /etc/apt/sources.list.d/google-compute-engine.list && \
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - && \
apt-get -qq update && \
apt-get -qq install -y --no-install-recommends google-osconfig-agent; \
fi \
&& \
rm -rf /var/lib/apt/lists/* /tmp/*
&& rm -rf /var/lib/apt/lists/* /tmp/*

# Build arguments and variables set for tracelog levels and debug information
#
Expand All @@ -90,7 +60,7 @@ ARG COLORBT_SHOW_HIDDEN
ENV COLORBT_SHOW_HIDDEN=${COLORBT_SHOW_HIDDEN:-1}

ARG SHORT_SHA
# If this is not set, it must be the empty string, so Zebra can try an alternative git commit source:
# If this is not set, it must be an empty string, so Zebra can try an alternative git commit source:
# https://github.com/ZcashFoundation/zebra/blob/9ebd56092bcdfc1a09062e15a0574c94af37f389/zebrad/src/application.rs#L179-L182
ENV SHORT_SHA=${SHORT_SHA:-}

Expand All @@ -102,12 +72,6 @@ ENV CARGO_HOME="${APP_HOME}/.cargo/"
# An entrypoint.sh is only available in this step for easier test handling with variables.
FROM deps AS tests

COPY --from=electriccoinco/lightwalletd:latest /usr/local/bin/lightwalletd /usr/local/bin/

# cargo uses timestamps for its cache, so they need to be in this order:
# unmodified source files < previous build cache < modified source files
COPY . .

# Skip IPv6 tests by default, as some CI environment don't have IPv6 available
ARG ZEBRA_SKIP_IPV6_TESTS
ENV ZEBRA_SKIP_IPV6_TESTS=${ZEBRA_SKIP_IPV6_TESTS:-1}
Expand All @@ -120,26 +84,41 @@ ARG EXPERIMENTAL_FEATURES
# TODO: add empty $EXPERIMENTAL_FEATURES when we can avoid adding an extra space to the end of the string
ARG ENTRYPOINT_FEATURES="${FEATURES} ${TEST_FEATURES}"

# Re-hydrate the minimum project skeleton identified by `cargo chef prepare` in the planner stage,
# over the top of the original source files,
# and build it to cache all possible sentry and test dependencies.
#
# This is the caching Docker layer for Rust tests!
# It creates fake empty test binaries so dependencies are built, but Zebra is not fully built.
#
# TODO: add --locked when cargo-chef supports it
RUN cargo chef cook --tests --release --features "${ENTRYPOINT_FEATURES}" --workspace --recipe-path recipe.json
# Undo the source file changes made by cargo-chef.
# rsync invalidates the cargo cache for the changed files only, by updating their timestamps.
# This makes sure the fake empty binaries created by cargo-chef are rebuilt.
COPY --from=planner ${APP_HOME} zebra-original
RUN rsync --recursive --checksum --itemize-changes --verbose zebra-original/ .
RUN rm -r zebra-original

# Build Zebra test binaries, but don't run them
RUN cargo test --locked --release --features "${ENTRYPOINT_FEATURES}" --workspace --no-run
RUN cp ${APP_HOME}/target/release/zebrad /usr/local/bin
RUN cp ${APP_HOME}/target/release/zebra-checkpoints /usr/local/bin

# Leverage a cache mount to /usr/local/cargo/registry/
# for downloaded dependencies, a cache mount to /usr/local/cargo/git/db
# for git repository dependencies, and a cache mount to ${APP_HOME}/target/ for
# compiled dependencies which will speed up subsequent builds.
# Leverage a bind mount to each crate directory to avoid having to copy the
# source code into the container. Once built, copy the executable to an
# output directory before the cache mounted ${APP_HOME}/target/ is unmounted.
RUN --mount=type=bind,source=zebrad,target=zebrad \
--mount=type=bind,source=zebra-chain,target=zebra-chain \
--mount=type=bind,source=zebra-network,target=zebra-network \
--mount=type=bind,source=zebra-state,target=zebra-state \
--mount=type=bind,source=zebra-script,target=zebra-script \
--mount=type=bind,source=zebra-consensus,target=zebra-consensus \
--mount=type=bind,source=zebra-rpc,target=zebra-rpc \
--mount=type=bind,source=zebra-node-services,target=zebra-node-services \
--mount=type=bind,source=zebra-test,target=zebra-test \
--mount=type=bind,source=zebra-utils,target=zebra-utils \
--mount=type=bind,source=zebra-scan,target=zebra-scan \
--mount=type=bind,source=zebra-grpc,target=zebra-grpc \
--mount=type=bind,source=tower-batch-control,target=tower-batch-control \
--mount=type=bind,source=tower-fallback,target=tower-fallback \
--mount=type=bind,source=Cargo.toml,target=Cargo.toml \
--mount=type=bind,source=Cargo.lock,target=Cargo.lock \
--mount=type=cache,target=${APP_HOME}/target/ \
--mount=type=cache,target=/usr/local/cargo/git/db \
--mount=type=cache,target=/usr/local/cargo/registry/ \
cargo test --locked --release --features "${ENTRYPOINT_FEATURES}" --workspace --no-run && \
cp ${APP_HOME}/target/release/zebrad /usr/local/bin && \
cp ${APP_HOME}/target/release/zebra-checkpoints /usr/local/bin

# Copy the lightwalletd binary and source files to be able to run tests
COPY --from=electriccoinco/lightwalletd:latest /usr/local/bin/lightwalletd /usr/local/bin/
COPY ./ ./

COPY ./docker/entrypoint.sh /etc/zebrad/entrypoint.sh

Expand All @@ -154,28 +133,34 @@ ENTRYPOINT [ "/etc/zebrad/entrypoint.sh" ]

# In this stage we build a release (generate the zebrad binary)
#
# This step also adds `cargo chef` as this stage is completely independent from the
# This step also adds `cache mounts` as this stage is completely independent from the
# `test` stage. This step is a dependency for the `runtime` stage, which uses the resulting
# zebrad binary from this step.
FROM deps AS release

COPY . .

ARG FEATURES

# This is the caching layer for Rust zebrad builds.
# It creates a fake empty zebrad binary, see above for details.
#
# TODO: add --locked when cargo-chef supports it
RUN cargo chef cook --release --features "${FEATURES}" --package zebrad --bin zebrad --recipe-path recipe.json

# Undo the source file changes made by cargo-chef, so the fake empty zebrad binary is rebuilt.
COPY --from=planner ${APP_HOME} zebra-original
RUN rsync --recursive --checksum --itemize-changes --verbose zebra-original/ .
RUN rm -r zebra-original

# Build zebrad
RUN cargo build --locked --release --features "${FEATURES}" --package zebrad --bin zebrad
RUN --mount=type=bind,source=zebrad,target=zebrad \
--mount=type=bind,source=zebra-chain,target=zebra-chain \
--mount=type=bind,source=zebra-network,target=zebra-network \
--mount=type=bind,source=zebra-state,target=zebra-state \
--mount=type=bind,source=zebra-script,target=zebra-script \
--mount=type=bind,source=zebra-consensus,target=zebra-consensus \
--mount=type=bind,source=zebra-rpc,target=zebra-rpc \
--mount=type=bind,source=zebra-node-services,target=zebra-node-services \
--mount=type=bind,source=zebra-test,target=zebra-test \
--mount=type=bind,source=zebra-utils,target=zebra-utils \
--mount=type=bind,source=zebra-scan,target=zebra-scan \
--mount=type=bind,source=zebra-grpc,target=zebra-grpc \
--mount=type=bind,source=tower-batch-control,target=tower-batch-control \
--mount=type=bind,source=tower-fallback,target=tower-fallback \
--mount=type=bind,source=Cargo.toml,target=Cargo.toml \
--mount=type=bind,source=Cargo.lock,target=Cargo.lock \
--mount=type=cache,target=${APP_HOME}/target/ \
--mount=type=cache,target=/usr/local/cargo/git/db \
--mount=type=cache,target=/usr/local/cargo/registry/ \
cargo build --locked --release --features "${FEATURES}" --package zebrad --bin zebrad && \
cp ${APP_HOME}/target/release/zebrad /usr/local/bin

COPY ./docker/entrypoint.sh ./

Expand All @@ -189,15 +174,16 @@ FROM debian:bookworm-slim AS runtime
ARG APP_HOME
ENV APP_HOME=${APP_HOME}
WORKDIR ${APP_HOME}
COPY --link --from=release /usr/local/bin /usr/local/bin
COPY --link --from=release /entrypoint.sh /

RUN apt-get update && \
apt-get install -y --no-install-recommends \
ca-certificates \
curl \
rocksdb-tools \
gosu \
&& \
rm -rf /var/lib/apt/lists/* /tmp/*
&& rm -rf /var/lib/apt/lists/* /tmp/*

# Create a non-privileged user that the app will run under.
# Running as root inside the container is running as root in the Docker host
Expand All @@ -224,6 +210,7 @@ ARG FEATURES
ENV FEATURES=${FEATURES}

# Path and name of the config file
# This are set here to always this variable set, even if the user does not set it
ENV ZEBRA_CONF_DIR=${ZEBRA_CONF_DIR:-/etc/zebrad}
ENV ZEBRA_CONF_FILE=${ZEBRA_CONF_FILE:-zebrad.toml}

Expand Down
Loading

0 comments on commit 35c41a6

Please sign in to comment.