zebrad sync tests fail with a filesystem error on Windows CI

[teor2345]

Version

zebrad 1.0.0-alpha.2

Platform

Windows CI:
https://github.com/ZcashFoundation/zebra/pull/1650/checks?check_run_id=1782427148

Description

Compiling the zebrad acceptance tests for the large sync tests on Windows CI almost always fails with one of the following errors:

fatal error LNK1201: error writing to program database; check for insufficient disk space, invalid path, or insufficient privilege

  = note:    Creating library D:\a\zebra\zebra\target\debug\deps\acceptance-2152ea11e4e85cec.lib and object D:\a\zebra\zebra\target\debug\deps\acceptance-2152ea11e4e85cec.exp
          LINK : fatal error LNK1201: error writing to program database 'D:\a\zebra\zebra\target\debug\deps\acceptance-2152ea11e4e85cec.pdb'; check for insufficient disk space, invalid path, or insufficient privilege

From the command:

Caused by:
  process didn't exit successfully: `rustc --crate-name acceptance --edition=2018 zebrad\tests\acceptance.rs --error-format=json --json=diagnostic-rendered-ansi --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 --test -C metadata=2152ea11e4e85cec -C extra-filename=-2152ea11e4e85cec --out-dir D:\a\zebra\zebra\target\debug\deps -C incremental=D:\a\zebra\zebra\target\debug\incremental -L dependency=D:\a\zebra\zebra\target\debug\deps --extern abscissa_core=D:\a\zebra\zebra\target\debug\deps\libabscissa_core-482460cdfe9b98d1.rlib --extern atty=D:\a\zebra\zebra\target\debug\deps\libatty-4566f53f310690a7.rlib --extern chrono=D:\a\zebra\zebra\target\debug\deps\libchrono-92f3b6a4b1d86091.rlib --extern color_eyre=D:\a\zebra\zebra\target\debug\deps\libcolor_eyre-d1a659b13c1b8609.rlib --extern dirs=D:\a\zebra\zebra\target\debug\deps\libdirs-7a84cb7c9d803fd5.rlib --extern futures=D:\a\zebra\zebra\target\debug\deps\libfutures-160eb05ef01c659b.rlib --extern gumdrop=D:\a\zebra\zebra\target\debug\deps\libgumdrop-21ac5e79d96b47c2.rlib --extern hyper=D:\a\zebra\zebra\target\debug\deps\libhyper-70c126bfb7eaa2df.rlib --extern inferno=D:\a\zebra\zebra\target\debug\deps\libinferno-3c9fed08adfffb2c.rlib --extern metrics=D:\a\zebra\zebra\target\debug\deps\libmetrics-2c0cffdf8d0bbe23.rlib --extern metrics_exporter_prometheus=D:\a\zebra\zebra\target\debug\deps\libmetrics_exporter_prometheus-1623bbda273fcaae.rlib --extern once_cell=D:\a\zebra\zebra\target\debug\deps\libonce_cell-306550a52ed0a0f5.rlib --extern pin_project=D:\a\zebra\zebra\target\debug\deps\libpin_project-3edaa0506a077e99.rlib --extern rand=D:\a\zebra\zebra\target\debug\deps\librand-81424f0376c44a85.rlib --extern sentry=D:\a\zebra\zebra\target\debug\deps\libsentry-3258c4c3ff40bee9.rlib --extern sentry_tracing=D:\a\zebra\zebra\target\debug\deps\libsentry_tracing-27841cbf18ab93f1.rlib --extern serde=D:\a\zebra\zebra\target\debug\deps\libserde-c4f121013fea4a71.rlib --extern tempdir=D:\a\zebra\zebra\target\debug\deps\libtempdir-f84e0b18234bd3df.rlib --extern thiserror=D:\a\zebra\zebra\target\debug\deps\libthiserror-46a688c7f048c63f.rlib --extern tokio=D:\a\zebra\zebra\target\debug\deps\libtokio-ff5f3aba12694f82.rlib --extern toml=D:\a\zebra\zebra\target\debug\deps\libtoml-50ef3c9a6aac7785.rlib --extern tower=D:\a\zebra\zebra\target\debug\deps\libtower-fcb580aa77f90738.rlib --extern tracing=D:\a\zebra\zebra\target\debug\deps\libtracing-587e032a960289b6.rlib --extern tracing_error=D:\a\zebra\zebra\target\debug\deps\libtracing_error-97191fd928461405.rlib --extern tracing_flame=D:\a\zebra\zebra\target\debug\deps\libtracing_flame-17717c2baea77ff3.rlib --extern tracing_futures=D:\a\zebra\zebra\target\debug\deps\libtracing_futures-c2cd826fe6ffde22.rlib --extern tracing_subscriber=D:\a\zebra\zebra\target\debug\deps\libtracing_subscriber-3a575f893db2157b.rlib --extern zebra_chain=D:\a\zebra\zebra\target\debug\deps\libzebra_chain-29304324f780b532.rlib --extern zebra_consensus=D:\a\zebra\zebra\target\debug\deps\libzebra_consensus-b07131123cc6bbdd.rlib --extern zebra_network=D:\a\zebra\zebra\target\debug\deps\libzebra_network-4bf326f968b083e9.rlib --extern zebra_state=D:\a\zebra\zebra\target\debug\deps\libzebra_state-82c2b1ca1e5cd588.rlib --extern zebra_test=D:\a\zebra\zebra\target\debug\deps\libzebra_test-27d65dd33b4d9b64.rlib --extern zebrad=D:\a\zebra\zebra\target\debug\deps\libzebrad-2310f2d802b79828.rlib -L native=D:\a\zebra\zebra\target\debug\build\ring-2baef81cc6ca1581\out -L native=D:\a\zebra\zebra\target\debug\build\secp256k1-sys-ca0d3c08f098db56\out -L native=D:\a\zebra\zebra\target\debug\build\zcash_script-0e36f3ab5dac7e48\out -L native=D:\a\zebra\zebra\target\debug\build\librocksdb-sys-2fb0faaa02ea91af\out -L native=D:\a\zebra\zebra\target\debug\build\librocksdb-sys-2fb0faaa02ea91af\out -L native=D:\a\zebra\zebra\target\debug\build\librocksdb-sys-2fb0faaa02ea91af\out -L native=D:\a\zebra\zebra\target\debug\build\librocksdb-sys-2fb0faaa02ea91af\out -L native=D:\a\zebra\zebra\target\debug\build\librocksdb-sys-2fb0faaa02ea91af\out -L native=D:\a\zebra\zebra\target\debug\build\librocksdb-sys-2fb0faaa02ea91af\out` (exit code: 1)

fatal error LNK1318: Unexpected PDB error; FILE_SYSTEM (3)

  = note:    Creating library D:\a\zebra\zebra\target\debug\deps\acceptance-7abd02c8246b2b6d.lib and object D:\a\zebra\zebra\target\debug\deps\acceptance-7abd02c8246b2b6d.exp
          LINK : fatal error LNK1318: Unexpected PDB error; FILE_SYSTEM (3) 'D:\a\zebra\zebra\target\debug\deps\acceptance-7abd02c8246b2b6d.pdb'

From the command:

Caused by:
  process didn't exit successfully: `rustc --crate-name acceptance --edition=2018 zebrad\tests\acceptance.rs --error-format=json --json=diagnostic-rendered-ansi --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 --test -C metadata=7abd02c8246b2b6d -C extra-filename=-7abd02c8246b2b6d --out-dir D:\a\zebra\zebra\target\debug\deps -C incremental=D:\a\zebra\zebra\target\debug\incremental -L dependency=D:\a\zebra\zebra\target\debug\deps --extern abscissa_core=D:\a\zebra\zebra\target\debug\deps\libabscissa_core-482460cdfe9b98d1.rlib --extern atty=D:\a\zebra\zebra\target\debug\deps\libatty-4566f53f310690a7.rlib --extern chrono=D:\a\zebra\zebra\target\debug\deps\libchrono-92f3b6a4b1d86091.rlib --extern color_eyre=D:\a\zebra\zebra\target\debug\deps\libcolor_eyre-d1a659b13c1b8609.rlib --extern dirs=D:\a\zebra\zebra\target\debug\deps\libdirs-7a84cb7c9d803fd5.rlib --extern futures=D:\a\zebra\zebra\target\debug\deps\libfutures-160eb05ef01c659b.rlib --extern gumdrop=D:\a\zebra\zebra\target\debug\deps\libgumdrop-21ac5e79d96b47c2.rlib --extern hyper=D:\a\zebra\zebra\target\debug\deps\libhyper-70c126bfb7eaa2df.rlib --extern inferno=D:\a\zebra\zebra\target\debug\deps\libinferno-3c9fed08adfffb2c.rlib --extern metrics=D:\a\zebra\zebra\target\debug\deps\libmetrics-2c0cffdf8d0bbe23.rlib --extern metrics_exporter_prometheus=D:\a\zebra\zebra\target\debug\deps\libmetrics_exporter_prometheus-1623bbda273fcaae.rlib --extern once_cell=D:\a\zebra\zebra\target\debug\deps\libonce_cell-306550a52ed0a0f5.rlib --extern pin_project=D:\a\zebra\zebra\target\debug\deps\libpin_project-3edaa0506a077e99.rlib --extern rand=D:\a\zebra\zebra\target\debug\deps\librand-81424f0376c44a85.rlib --extern sentry=D:\a\zebra\zebra\target\debug\deps\libsentry-3258c4c3ff40bee9.rlib --extern sentry_tracing=D:\a\zebra\zebra\target\debug\deps\libsentry_tracing-27841cbf18ab93f1.rlib --extern serde=D:\a\zebra\zebra\target\debug\deps\libserde-c4f121013fea4a71.rlib --extern tempdir=D:\a\zebra\zebra\target\debug\deps\libtempdir-f84e0b18234bd3df.rlib --extern thiserror=D:\a\zebra\zebra\target\debug\deps\libthiserror-46a688c7f048c63f.rlib --extern tokio=D:\a\zebra\zebra\target\debug\deps\libtokio-ff5f3aba12694f82.rlib --extern toml=D:\a\zebra\zebra\target\debug\deps\libtoml-50ef3c9a6aac7785.rlib --extern tower=D:\a\zebra\zebra\target\debug\deps\libtower-fcb580aa77f90738.rlib --extern tracing=D:\a\zebra\zebra\target\debug\deps\libtracing-587e032a960289b6.rlib --extern tracing_error=D:\a\zebra\zebra\target\debug\deps\libtracing_error-97191fd928461405.rlib --extern tracing_flame=D:\a\zebra\zebra\target\debug\deps\libtracing_flame-17717c2baea77ff3.rlib --extern tracing_futures=D:\a\zebra\zebra\target\debug\deps\libtracing_futures-c2cd826fe6ffde22.rlib --extern tracing_subscriber=D:\a\zebra\zebra\target\debug\deps\libtracing_subscriber-3a575f893db2157b.rlib --extern zebra_chain=D:\a\zebra\zebra\target\debug\deps\libzebra_chain-29304324f780b532.rlib --extern zebra_consensus=D:\a\zebra\zebra\target\debug\deps\libzebra_consensus-d00eb77f5fedb240.rlib --extern zebra_network=D:\a\zebra\zebra\target\debug\deps\libzebra_network-8e0819f8fd2b16a1.rlib --extern zebra_state=D:\a\zebra\zebra\target\debug\deps\libzebra_state-b2076e0c04aaa42f.rlib --extern zebra_test=D:\a\zebra\zebra\target\debug\deps\libzebra_test-27d65dd33b4d9b64.rlib --extern zebrad=D:\a\zebra\zebra\target\debug\deps\libzebrad-7d670f5ec14cf287.rlib -L native=D:\a\zebra\zebra\target\debug\build\ring-2baef81cc6ca1581\out -L native=D:\a\zebra\zebra\target\debug\build\secp256k1-sys-ca0d3c08f098db56\out -L native=D:\a\zebra\zebra\target\debug\build\zcash_script-0e36f3ab5dac7e48\out -L native=D:\a\zebra\zebra\target\debug\build\librocksdb-sys-2fb0faaa02ea91af\out -L native=D:\a\zebra\zebra\target\debug\build\librocksdb-sys-2fb0faaa02ea91af\out -L native=D:\a\zebra\zebra\target\debug\build\librocksdb-sys-2fb0faaa02ea91af\out -L native=D:\a\zebra\zebra\target\debug\build\librocksdb-sys-2fb0faaa02ea91af\out -L native=D:\a\zebra\zebra\target\debug\build\librocksdb-sys-2fb0faaa02ea91af\out -L native=D:\a\zebra\zebra\target\debug\build\librocksdb-sys-2fb0faaa02ea91af\out` (exit code: 1)
Error: The process 'C:\Rust\.cargo\bin\cargo.exe' failed with exit code 101

Commands

https://github.com/ZcashFoundation/zebra/blob/main/.github/workflows/ci.yml#L34

      # Explicitly run any tests that are usually #[ignored], modulo ZEBRA_SKIP_NETWORK_TESTS
      - name: Run zebrad large sync tests
        env:
          RUST_BACKTRACE: full
        uses: actions-rs/cargo@v1
        with:
          command: test
          args: --verbose --manifest-path zebrad/Cargo.toml sync_large_checkpoints_ -- --ignored

Analysis

It seems like the previous compilation or sync tests are filling up the disk, or otherwise making compilation impossible.
Can we get CI to re-use the results of the previous step?

We tried enabling very verbose cargo builds --verbose --verbose, but didn't get any useful output. (#1652)
It looks like we need to set the cargo tracing filter to info instead, so we see the output of these traces in cargo:

                sess.struct_err(&format!("linking with `{}` failed: {}",
                                         pname.display(),
                                         prog.status))
                    .note(&format!("{:?}", &cmd))
                    .note(&escape_string(&output))
                    .emit();
                sess.abort_if_errors();
            }
            info!("linker stderr:\n{}", escape_string(&prog.stderr));
            info!("linker stdout:\n{}", escape_string(&prog.stdout));

https://github.com/rust-lang/rust-gha/blob/f50d6ea348c2dd7c2f76e35ecde6560d87bb98ec/src/librustc_codegen_ssa/back/link.rs#L671

Priority

We merged #1658 to temporarily disable these tests, so this ticket can be done in sprint 2 or 3. It's still a high priority, because when we disable tests, we risk merging other bugs.

[teor2345]

We merged #1658 to temporarily disable these tests, so this ticket can be done in sprint 2 or 3. It's still a high priority, because when we disable tests, we risk merging other bugs.

[mpguerra]

Re-opened to link to #1726 in zenhub