Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove CI Caching to preserve diskspace #2948

Merged
merged 2 commits into from
Jul 20, 2022
Merged

Remove CI Caching to preserve diskspace #2948

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6b1ae8b
Test running without caching target directory
alamb Jul 20, 2022
91c20f3
Only check during test
alamb Jul 20, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 6 additions & 49 deletions .github/workflows/rust.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ on:
pull_request:

jobs:
# build the library, a compilation step used by multiple steps below
# Check crate compiles
linux-build-lib:
name: Build Libraries on AMD64 Rust ${{ matrix.rust }}
runs-on: ubuntu-latest
Expand All @@ -46,36 +46,29 @@ jobs:
# and thus do not depend on the OS, arch nor rust version.
path: /github/home/.cargo
key: cargo-cache-
- name: Cache Rust dependencies
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have never been convinced this caching is particularly effective so I am going to test what happens when I remove it...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if something is wrong with the config 🤔

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found this https://github.com/marketplace/actions/rust-cache which does perform some heuristics and cleaning.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe what's wrong here is that the .cargo die is not cached?

https://doc.rust-lang.org/cargo/guide/cargo-home.html#caching-the-cargo-home-in-ci

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

This cache is automatically keyed by:

the github [job_id](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_id),
the rustc release / host / hash,
the value of some compiler-specific environment variables (eg. RUSTFLAGS, etc), and
a hash of all Cargo.lock / Cargo.toml files found anywhere in the repository (if present).
a hash of all rust-toolchain / rust-toolchain.toml files in the root of the repository (if present).

Seems particularly relevant to why the basic cache wasn't working for us

uses: actions/cache@v3
with:
# these represent compiled steps of both dependencies and arrow
# and thus are specific for a particular OS, arch and rust version.
path: /github/home/target
key: ${{ runner.os }}-${{ matrix.arch }}-target-cache-${{ matrix.rust }}-
- name: Setup Rust toolchain
uses: ./.github/actions/setup-builder
with:
rust-version: ${{ matrix.rust }}
- name: Build workspace in debug mode
- name: Check workspace in debug mode
run: |
cargo build
cargo check
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:👍

env:
CARGO_HOME: "/github/home/.cargo"
CARGO_TARGET_DIR: "/github/home/target/debug"
- name: Build workspace in release mode
- name: Check workspace in release mode
run: |
cargo check --release
env:
CARGO_HOME: "/github/home/.cargo"
CARGO_TARGET_DIR: "/github/home/target/release"
- name: Check DataFusion builds without default features
- name: Check workspace without default features
run: |
cargo check --no-default-features -p datafusion
env:
CARGO_HOME: "/github/home/.cargo"
CARGO_TARGET_DIR: "/github/home/target"
- name: Check Workspace builds with all features
- name: Check workspace with all features
run: |
cargo check --workspace --benches --features avro,jit,scheduler,json
env:
Expand Down Expand Up @@ -107,12 +100,6 @@ jobs:
path: /github/home/.cargo
# this key equals the ones on `linux-build-lib` for re-use
key: cargo-cache-
- name: Cache Rust dependencies
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For what it is worth the last time this CI check passed on master was https://github.com/apache/arrow-datafusion/runs/7399254770?check_suite_focus=true where it took 24m and 26 seconds (after the linux-build-lib job finished)

uses: actions/cache@v3
with:
path: /github/home/target
# this key equals the ones on `linux-build-lib` for re-use
key: ${{ runner.os }}-${{ matrix.arch }}-target-cache-${{ matrix.rust }}
- name: Setup Rust toolchain
uses: ./.github/actions/setup-builder
with:
Expand Down Expand Up @@ -240,12 +227,6 @@ jobs:
path: /github/home/.cargo
# this key equals the ones on `linux-build-lib` for re-use
key: cargo-cache-
- name: Cache Rust dependencies
uses: actions/cache@v3
with:
path: /github/home/target
# this key equals the ones on `linux-build-lib` for re-use
key: ${{ runner.os }}-${{ matrix.arch }}-target-cache-${{ matrix.rust }}
- uses: actions/setup-python@v3
with:
python-version: "3.8"
Expand Down Expand Up @@ -305,12 +286,6 @@ jobs:
path: /home/runner/.cargo
# this key is not equal because the user is different than on a container (runner vs github)
key: cargo-coverage-cache3-
- name: Cache Rust dependencies
uses: actions/cache@v3
with:
path: /home/runner/target
# this key is not equal because coverage uses different compilation flags.
key: ${{ runner.os }}-${{ matrix.arch }}-target-coverage-cache3-${{ matrix.rust }}-
- name: Run coverage
run: |
export CARGO_HOME="/home/runner/.cargo"
Expand Down Expand Up @@ -351,12 +326,6 @@ jobs:
path: /github/home/.cargo
# this key equals the ones on `linux-build-lib` for re-use
key: cargo-cache-
- name: Cache Rust dependencies
uses: actions/cache@v3
with:
path: /github/home/target
# this key equals the ones on `linux-build-lib` for re-use
key: ${{ runner.os }}-${{ matrix.arch }}-target-cache-${{ matrix.rust }}
- name: Setup Rust toolchain
uses: ./.github/actions/setup-builder
with:
Expand Down Expand Up @@ -429,12 +398,6 @@ jobs:
path: /github/home/.cargo
# this key equals the ones on `linux-build-lib` for re-use
key: cargo-cache-
- name: Cache Rust dependencies
uses: actions/cache@v3
with:
path: /github/home/target
# this key equals the ones on `linux-build-lib` for re-use
key: ${{ runner.os }}-${{ matrix.arch }}-target-cache-${{ matrix.rust }}
- name: Setup Rust toolchain
uses: ./.github/actions/setup-builder
with:
Expand Down Expand Up @@ -474,12 +437,6 @@ jobs:
path: /github/home/.cargo
# this key equals the ones on `linux-build-lib` for re-use
key: cargo-cache-
- name: Cache Rust dependencies
uses: actions/cache@v3
with:
path: /github/home/target
# this key equals the ones on `linux-build-lib` for re-use
key: ${{ runner.os }}-${{ matrix.arch }}-target-cache-${{ matrix.rust }}
- name: Setup Rust toolchain
uses: ./.github/actions/setup-builder
with:
Expand Down