Сaching through docker volume #10477

TriplEight · 2019-03-14T13:32:12Z

Runner config:

> sudo cat /etc/gitlab-runner/config.toml
concurrent = 6
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "QA_experiments"
  url = "https://gitlab.parity.io"
  token = "d2f32b1a13dda2fa9e629f8eb92d1d"
  executor = "docker"
  output_limit = 16384
  [runners.docker]
    privileged = false
    disable_entrypoint_overwrite = false
    disable_cache = false
    volumes = ["/home/gitlab-runner/sccache:/cargo:rw"]
    shm_size = 0
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

I've sorted 153 sccache problems during test-linux in CI, which do not allow using caches for these crates

CannotCache(crate-type, No crate-type passed) - 78
CannotCache(crate-type, proc-macro) - 8

CannotCache(incremental) - 8
CannotCache(crate-type, bin) - 57
CannotCache(-) -2

Most of CannotCache(crate-type, bin) look like: ["--crate-name", "build_script_build", "/cargo/test-linux/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.4.0/build.rs".
And I'm not sure if there's something to do with it.

CannotCache(incremental) can not be handled by sccache.

And No crate-type passed, proc-macro is probably something we can fix later.

parity-cla-bot · 2019-03-14T13:32:16Z

It looks like @TriplEight signed our Contributor License Agreement. 👍

Many thanks,

Parity Technologies CLA Bot

`qa` -> `linux-docker`

…ckerfile won't be updated

…/parity-ethereum into caching_through_docker_volume

* _old codebase_ before docker update * before docker update, testing runnr * docker update, testing the caching * distributed job cargo homes * distributed job cargo homes 2 * distributed job cargo homes 3 * dockerfile with gitlab checkout, audit uses template * dockerfile gets repo in volume * change builds_dir * trying docker cache for repo * repo cached automatically * after script is not concatenated * check sccache non-cacheable reasons nature * watch cache * log sccache * log sccache 2 * debug log sccache * fix debug log sccache * fix debug log sccache 2 * debug log cache 3 * debug log cache 3 * trace log all sccache * test wo cargo cache * test w removed cargo cache * report non-cacheable reasons, cargo cache is back and empty * report non-cacheable reasons, cargo cache is back and empty 2 * report non-cacheable reasons, cargo cache is back and empty 3 * wrap into after_script * restore CI tags `qa` -> `linux-docker` * return to main runners, this will fail until config on runners And Dockerfile won't be updated * typo fix CI lint * return to docker tag

* version: bump beta * Сaching through docker volume (#10477) * _old codebase_ before docker update * before docker update, testing runnr * docker update, testing the caching * distributed job cargo homes * distributed job cargo homes 2 * distributed job cargo homes 3 * dockerfile with gitlab checkout, audit uses template * dockerfile gets repo in volume * change builds_dir * trying docker cache for repo * repo cached automatically * after script is not concatenated * check sccache non-cacheable reasons nature * watch cache * log sccache * log sccache 2 * debug log sccache * fix debug log sccache * fix debug log sccache 2 * debug log cache 3 * debug log cache 3 * trace log all sccache * test wo cargo cache * test w removed cargo cache * report non-cacheable reasons, cargo cache is back and empty * report non-cacheable reasons, cargo cache is back and empty 2 * report non-cacheable reasons, cargo cache is back and empty 3 * wrap into after_script * restore CI tags `qa` -> `linux-docker` * return to main runners, this will fail until config on runners And Dockerfile won't be updated * typo fix CI lint * return to docker tag * fix win&mac build (#10486) add CARGO_HOME: "${CI_PROJECT_DIR}/.cargo" * fix(extract `timestamp_checked_add` as lib) (#10383) * fix(extract `timestamp_checked_add` as lib) * fix(whisper types): remove unused `EmptyTopics` * fix(time-lib): feature-flag to use time-lib or std This commit adds conditional compilation checks that falls back to `our time-lib` when `time_checked_add` is not available in the standard library Note, `time_checked_add` covers both `checked_add` and `checked_sub` * fix(grumble): use cfg_attr to define rustc feature

* version: bump stable * Сaching through docker volume (#10477) * _old codebase_ before docker update * before docker update, testing runnr * docker update, testing the caching * distributed job cargo homes * distributed job cargo homes 2 * distributed job cargo homes 3 * dockerfile with gitlab checkout, audit uses template * dockerfile gets repo in volume * change builds_dir * trying docker cache for repo * repo cached automatically * after script is not concatenated * check sccache non-cacheable reasons nature * watch cache * log sccache * log sccache 2 * debug log sccache * fix debug log sccache * fix debug log sccache 2 * debug log cache 3 * debug log cache 3 * trace log all sccache * test wo cargo cache * test w removed cargo cache * report non-cacheable reasons, cargo cache is back and empty * report non-cacheable reasons, cargo cache is back and empty 2 * report non-cacheable reasons, cargo cache is back and empty 3 * wrap into after_script * restore CI tags `qa` -> `linux-docker` * return to main runners, this will fail until config on runners And Dockerfile won't be updated * typo fix CI lint * return to docker tag * fix win&mac build (#10486) add CARGO_HOME: "${CI_PROJECT_DIR}/.cargo" * fix(extract `timestamp_checked_add` as lib) (#10383) * fix(extract `timestamp_checked_add` as lib) * fix(whisper types): remove unused `EmptyTopics` * fix(time-lib): feature-flag to use time-lib or std This commit adds conditional compilation checks that falls back to `our time-lib` when `time_checked_add` is not available in the standard library Note, `time_checked_add` covers both `checked_add` and `checked_sub` * fix(grumble): use cfg_attr to define rustc feature

* master: (48 commits) ethcore: remove eth social and easthub chain configs (#10531) Initial support sccache for windows build (#10520) fix(light): make `OnDemand` generic instead of using the concrete type (#10514) private-tx: replace error_chain (#10510) Add trace information to eth_estimateGas (#10519) ethcore: add clique engine (#9981) verbose flag for cpp tests (#10524) Add a more realistic Batch test (#10511) docs: add changelogs for 2.3.{6,7,8} and 2.4.{1,2,3} (#10494) fix(rpc): fix a bunch of clippy lints (#10493) fix Sha3/keccak256 hash calculation for binaries (#10509) Add additional request tests (#10503) whisper/cli: add p2p port and ip parameters (#10057) fix(time-utils): add missing license (#10497) fix(extract `timestamp_checked_add` as lib) (#10383) fix(rpc): lint `unused_extern_crates` + fix warns (#10489) fix win&mac build (#10486) Сaching through docker volume (#10477) OpenBlock::new take IntoIterator instead of mutable ref to Iterator (#10480) simplify block module and usage (#10479) ...

_old codebase_ before docker update

2fc8973

TriplEight requested a review from General-Beck March 14, 2019 13:32

TriplEight added 5 commits March 14, 2019 15:22

before docker update, testing runnr

aa56a7a

docker update, testing the caching

340001b

distributed job cargo homes

5f878da

distributed job cargo homes 2

4b93502

distributed job cargo homes 3

ee5d195

soc1c added this to the 2.5 milestone Mar 14, 2019

soc1c added A3-inprogress ⏳ Pull request is in progress. No review needed at this stage. B1-patch-beta 🕷🕷 M1-ci 🙉 Continuous integration. B0-patch-stable 🕷 Pull request should also be back-ported to the stable branch. labels Mar 14, 2019

TriplEight added 17 commits March 15, 2019 12:14

dockerfile with gitlab checkout, audit uses template

2e74d00

dockerfile gets repo in volume

061c596

change builds_dir

eca48a7

trying docker cache for repo

828354c

repo cached automatically

dbee0bb

after script is not concatenated

673ee3e

check sccache non-cacheable reasons nature

3e33843

watch cache

204d835

log sccache

79b79c8

log sccache 2

8b920ba

debug log sccache

ef8800a

fix debug log sccache

2c0ac55

fix debug log sccache 2

4883b02

debug log cache 3

2ce36ea

debug log cache 3

ad2bbc8

trace log all sccache

3c7aadd

test wo cargo cache

b009a6e

TriplEight added 5 commits March 18, 2019 20:53

test w removed cargo cache

908106a

report non-cacheable reasons, cargo cache is back and empty

f982a4d

report non-cacheable reasons, cargo cache is back and empty 2

ff892fe

report non-cacheable reasons, cargo cache is back and empty 3

62edfd5

wrap into after_script

80f86b4

General-Beck approved these changes Mar 18, 2019

View reviewed changes

General-Beck and others added 6 commits March 19, 2019 02:34

restore CI tags

dcbefb0

`qa` -> `linux-docker`

return to main runners, this will fail until config on runners And Do…

712abd6

…ckerfile won't be updated

typo fix CI lint

038670c

return to main runners, this will fail until config on runners And Do…

bf0dd29

…ckerfile won't be updated

Merge branch 'caching_through_docker_volume' of github.com:paritytech…

bf10499

…/parity-ethereum into caching_through_docker_volume

return to docker tag

1555975

TriplEight changed the title ~~WIP: caching through docker volume~~ Сaching through docker volume Mar 18, 2019

devops-parity approved these changes Mar 18, 2019

View reviewed changes

General-Beck merged commit a8ee3c9 into master Mar 19, 2019

General-Beck deleted the caching_through_docker_volume branch March 19, 2019 00:15

This was referenced Mar 19, 2019

2.3.7 stable backports #10487

Merged

2.4.2 beta backports #10488

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Сaching through docker volume #10477

Сaching through docker volume #10477

TriplEight commented Mar 14, 2019 •

edited

Loading

parity-cla-bot commented Mar 14, 2019

Сaching through docker volume #10477

Сaching through docker volume #10477

Conversation

TriplEight commented Mar 14, 2019 • edited Loading

parity-cla-bot commented Mar 14, 2019

TriplEight commented Mar 14, 2019 •

edited

Loading