chore: use sccache

[yihau]

related to #26171

• test sccache works fine in CI/CD pipeline.
• setup sccache to all buildkite agents

sccache config has setup on buildkite-agent server. This PR is only for

• remove old cache mechanism
• trigger sccache in docker container

[t-nelson]

do we have any agents running this for testing? I did some brief local testing with sccache and found that not much actually wound up in the cache (~800M in the sccache dir, 27GB in ./target). Admittedly, I haven't taken the time to investigate past an initial basic setup

[yihau]

just tested it. target folder is removed by each steps

build steps \ cache method	cargo-target-cache	scache
build 1343037	real 4m20.960s user 112m17.938s sys 5m52.708s	real 5m51.370s user 22m52.985s sys 2m25.030s Cache hits 0 Cache misses 922 Cache misses (C/C++) 366 Cache misses (Rust) 556
build 1343037	real 0m0.692s user 0m0.575s sys 0m0.116s	real 1m36.539s user 22m34.121s sys 2m23.897s Cache hits 920 Cache hits (C/C++) 366 Cache hits (Rust) 554 Cache misses 2 Cache misses (Rust) 2
build 9723a33	real 2m20.319s user 46m14.406s sys 1m26.461s	real 4m2.598s user 21m30.982s sys 2m22.199s Cache hits 864 Cache hits (C/C++) 366 Cache hits (Rust) 498 Cache misses 58 Cache misses (Rust) 58
build 1343037	real 2m16.891s user 44m36.418s sys 1m21.895s	real 1m37.054s user 22m25.801s sys 2m23.270s Cache hits 920 Cache hits (C/C++) 366 Cache hits (Rust) 554 Cache misses 2 Cache misses (Rust) 2

the test script

#!/bin/bash

function silent_checkout() {
       git checkout $1 >/dev/null 2>&1
}

function remove_target_and_build_with_target_cache() {
       rm -rf target/

       echo "=> pre command"
       source .buildkite/hooks/pre-command >/dev/null 2>&1
       echo "=> build"
       time cargo +1.60.0 build --release --bins -q
       echo "=> post command"
       source .buildkite/hooks/post-command >/dev/null 2>&1
}

function remove_target_and_build_with_sccache() {
       rm -rf target/

       echo "=> build"
       time RUSTC_WRAPPER=/var/lib/buildkite-agent/.cargo/bin/sccache cargo +1.60.0 build --release --bins -q
}


echo "delete all cache"
rm -rf /var/lib/buildkite-agent/cargo-target-cache/
rm -rf /var/lib/buildkite-agent/.cache/sccache/
rm -rf target/


echo "===== cargo-target-cache ====="

echo "==> checkout 134303714"
silent_checkout 134303714
remove_target_and_build_with_target_cache

echo "==> build again"
remove_target_and_build_with_target_cache

echo "==> checkout "9723a33d2
silent_checkout 9723a33d2
remove_target_and_build_with_target_cache

echo "==> checkout 134303714"
silent_checkout 134303714
remove_target_and_build_with_target_cache


echo "delete all cache"
rm -rf /var/lib/buildkite-agent/cargo-target-cache/
rm -rf /var/lib/buildkite-agent/.cache/sccache/
rm -rf target/


echo "===== sccache ====="

echo "==> checkout 134303714"
silent_checkout 134303714
remove_target_and_build_with_sccache
sccache --stop-server

echo "==> build again"
remove_target_and_build_with_sccache
sccache --stop-server

echo "==> checkout "9723a33d2
silent_checkout 9723a33d2
remove_target_and_build_with_sccache
sccache --stop-server

echo "==> checkout 134303714"
silent_checkout 134303714
remove_target_and_build_with_sccache
sccache --stop-server

[yihau]

Seems our target cache mechanism saves more time than sccache. I think if we just would like to get rid of cargo-target-cache, the result seems acceptable? If we really care about the build time, I think we can keep cargo-target-cache but store the cache on cloud storage.

[yihau]

do we give it a shot? or I maybe I can try to search another way to improve our cargo-target-cache

[alessandrod]

Might be worth trying with the sccache s3 backend? We use that for https://github.com/solana-labs/rust and it seems to work great

[alessandrod]

Mind that depending on the signature version of the bucket you're using, you might need mozilla/sccache#1086

[yihau]

@alessandrod thank you for pointing it out!
for now I use a private bucket without any policy and use a key from a restricted user.

I think maybe I should force it to use v4 signature when the branch merged 🤔