llvm-cov report - no filenames specified (failed only on windows and works on linux)

[LilyFoote]

I've been trying to add coverage for the Rust part of a combined Python and Rust project. I've got it working for our linux CI builds, but trying the same thing for Windows is leading to this error:

Run cargo llvm-cov report --manifest-path=rust\Cargo.toml --codecov --output-path codecov.json -vvv
  
     Running `C:\Users\runneradmin\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\x86_64-pc-windows-msvc\bin\llvm-profdata.exe merge -sparse -f D:\a\kolo\kolo\python\rust\target\rust-profraw-list -o D:\a\kolo\kolo\python\rust\target\rust.profdata`
     Running `C:\Users\runneradmin\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\x86_64-pc-windows-msvc\bin\llvm-cov.exe export -format=text -use-color=0 -instr-profile=D:\a\kolo\kolo\python\rust\target\rust.profdata -ignore-filename-regex \\rustc\\[0-9a-f]+\\|^D:\\a\\kolo\\kolo\\python\\rust(\\.*)?\\(tests|examples|benches)\\|^D:\\a\\kolo\\kolo\\python\\rust\\target($|\\)|^C:\\Users\\runneradmin\\\.cargo\\(registry|git)\\|^C:\\Users\\runneradmin\\\.rustup\\toolchains($|\\)`
error: failed to generate report: process didn't exit successfully: `C:\Users\runneradmin\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\x86_64-pc-windows-msvc\bin\llvm-cov.exe export -format=text -use-color=0 -instr-profile=D:\a\kolo\kolo\python\rust\target\rust.profdata -ignore-filename-regex \\rustc\\[0-9a-f]+\\|^D:\\a\\kolo\\kolo\\python\\rust(\\.*)?\\(tests|examples|benches)\\|^D:\\a\\kolo\\kolo\\python\\rust\\target($|\\)|^C:\\Users\\runneradmin\\\.cargo\\(registry|git)\\|^C:\\Users\\runneradmin\\\.rustup\\toolchains($|\\)` (exit code: 1)
--- stderr
No filenames specified!

I have confirmed that the .profraw files are generated and are in the directory I expect them in:

Run dir rust\target
  

    Directory: D:\a\kolo\kolo\python\rust\target

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d----           8/11/2023 10:56 AM                debug
-a---           8/11/2023 10:55 AM           1048 .rustc_info.json
-a---           8/11/2023  9:00 AM            177 CACHEDIR.TAG
-a---           8/11/2023 10:56 AM        2377336 rust-1720-9197467843878644390_0.profraw
-a---           8/11/2023 10:57 AM        2377336 rust-3488-9197467843878644390_0.profraw
-a---           8/11/2023 10:57 AM        2377336 rust-4996-9197467843878644390_4.profraw
-a---           8/11/2023 10:57 AM        2377336 rust-5820-9197467843878644390_4.profraw
-a---           8/11/2023 10:57 AM        2377336 rust-5936-9197467843878644390_0.profraw

I'm at a loss how to further debug this.

[taiki-e]

Thanks for the report. The report subcommand does not work by itself. Could you provide information on all the commands you have run?

Also, is it possible to provide the information needed to reproduce this, such as code, workflows, compiler version, etc.?

[LilyFoote]

Hi! Sorry for not giving enough detail before. Here's the github actions workflow file (with some irrelevant jobs omitted):

on:
  push:
    paths:
      - .github/workflows/python-package.yml
      - python/**
  pull_request_target:
    paths:
      - .github/workflows/python-package.yml
      - python/**

jobs:
  python-package:
    if:
      (github.actor != 'dependabot[bot]' && github.event_name == 'push') ||
      (github.actor == 'dependabot[bot]' && github.event_name == 'pull_request_target')
    strategy:
      matrix:
        python: ["3.8", "3.9", "3.10", "3.11", "pypy-3.9", "pypy-3.10"]
        django: ["3.2.0", "4.1.0", "4.2.0"]
      fail-fast: false
    runs-on: buildjet-2vcpu-ubuntu-2204	
    defaults:
      run:
        working-directory: ./python
    steps:
    - name: Checkout
      if: ${{ github.event_name != 'pull_request_target' }}
      uses: actions/checkout@v3

    - name: Checkout PR
      if: ${{ github.event_name == 'pull_request_target' }}
      uses: actions/checkout@v3
      with:
        ref: ${{ github.event.pull_request.head.sha }}

    - name: Install Rust toolchain
      uses: dtolnay/rust-toolchain@stable
      with:
        components: llvm-tools-preview

    - name: Install cargo-llvm-cov
      uses: taiki-e/install-action@cargo-llvm-cov

    - uses: Swatinem/rust-cache@v2
      with:
        workspaces: python/rust

    - uses: actions/setup-python@v2
      with:
        python-version: ${{ matrix.python }}
        cache: "pip"
        cache-dependency-path: python/dev-requirements.txt
    - run: python -m venv .venv
    - run: .venv/bin/pip install -r dev-requirements.txt
    - run: .venv/bin/pip install django~=${{ matrix.django }}
    - run: .venv/bin/pip install maturin
    - run: .venv/bin/maturin develop --profile dev --extras=test_generation
      env:
        RUSTFLAGS: -C instrument-coverage -C llvm-args=--instrprof-atomic-counter-update-all --cfg=coverage --cfg=trybuild_no_target
        CARGO_INCREMENTAL: 0
        CARGO_LLVM_COV: 1
    - run: .venv/bin/pip list
    - run: .venv/bin/coverage run -m pytest --ignore=tests/simplepoll
      env:
        PYTHONFAULTHANDLER: 1
        RUST_BACKTRACE: 1
        LLVM_PROFILE_FILE: ${{ github.workspace }}/python/rust/target/rust-%p-%8m.profraw
        CARGO_LLVM_COV_TARGET_DIR: ${{ github.workspace }}/python/rust/target
    - run: .venv/bin/coverage combine
    - run: .venv/bin/coverage xml
    - run: pwd
    - run: head -n 100 coverage.xml
    - run: cargo llvm-cov report --manifest-path=rust/Cargo.toml --codecov --output-path codecov.json
      env:
        LLVM_PROFILE_FILE: ${{ github.workspace }}/python/rust/target/rust-%p-%8m.profraw
        CARGO_LLVM_COV_TARGET_DIR: ${{ github.workspace }}/python/rust/target
      if: ${{ !contains( matrix.python, 'pypy') }}
    - run: |
        . .venv/bin/activate
        cargo llvm-cov --manifest-path=rust/Cargo.toml --all-features --workspace --codecov --output-path codecov-unit.json
      env:
        LLVM_PROFILE_FILE: ${{ github.workspace }}/python/rust/target/rust-%p-%8m.profraw
        CARGO_LLVM_COV_TARGET_DIR: ${{ github.workspace }}/python/rust/target
      if: ${{ !contains( matrix.python, 'pypy') }}
    - name: codecov
      uses: codecov/codecov-action@v3
      with:
        token: ${{ secrets.CODECOV_TOKEN }}
        flags: linux,${{ matrix.python }},${{ matrix.django }}
        verbose: true
        files: codecov.yaml,python/coverage.xml,python/codecov.json,python/codecov-unit.json

  windows:
    if:
      (github.actor != 'dependabot[bot]' && github.event_name == 'push') ||
      (github.actor == 'dependabot[bot]' && github.event_name == 'pull_request_target')
    strategy:
      matrix:
        python: ["3.8", "3.9", "3.10", "3.11", "pypy-3.9", "pypy-3.10"]
      fail-fast: false
    runs-on: windows-latest
    defaults:
      run:
        working-directory: ./python
    steps:
    - name: Checkout
      if: ${{ github.event_name != 'pull_request_target' }}
      uses: actions/checkout@v3

    - name: Checkout PR
      if: ${{ github.event_name == 'pull_request_target' }}
      uses: actions/checkout@v3
      with:
        ref: ${{ github.event.pull_request.head.sha }}
    - name: Install Rust toolchain
      uses: dtolnay/rust-toolchain@stable

    - uses: Swatinem/rust-cache@v2
      with:
        workspaces: python/rust
    - uses: actions/setup-python@v2
      with:
        python-version: ${{ matrix.python }}
        cache: "pip"
        cache-dependency-path: python/dev-requirements.txt
    - run: python -m venv .venv
    - run: .venv/Scripts/pip install -r dev-requirements.txt
    - run: .venv/Scripts/pip install django -U
    - run: .venv/Scripts/pip install maturin
    - run: .venv/Scripts/maturin develop --profile dev --extras=test_generation
    - run: .venv/Scripts/coverage run -m pytest --ignore=tests/simplepoll
      env:
        PYTHONFAULTHANDLER: 1
        RUST_BACKTRACE: 1
    - run: .venv/Scripts/coverage combine
    - run: .venv/Scripts/coverage xml
    - run: pwd
    - run: head -n 100 coverage.xml
    - run: mv coverage.xml ..
    - name: codecov
      uses: codecov/codecov-action@v3
      with:
        token: ${{ secrets.CODECOV_TOKEN }}
        flags: windows,${{ matrix.python }}
        verbose: true

The python-package version collects and reports coverage information, but the windows version leads to the error above.

The code being tested is not straightforward to share, but I will try to create a smaller replication.

[LilyFoote]

Here's a minimised example:

https://github.com/LilyFoote/llvm-cov-windows-bug
https://github.com/LilyFoote/llvm-cov-windows-bug/actions/runs/5857596860/job/15879786951

[taiki-e]

@LilyFoote

Here's a minimised example

https://github.com/LilyFoote/llvm-cov-windows-bug

Thanks. You seem to be manually setting some of the environment variables that cargo llvm-cov show-env sets, would the same problem occur if you used source <(cargo llvm-cov show-env --export-prefix) instead of setting them manually?

[LilyFoote]

@taiki-e That seems to work for linux, but not for windows - I get a parser error:

Run source <(cargo llvm-cov show-env --export-prefix --manifest-path=rust\Cargo.toml)
ParserError: D:\a\_temp\9f4104b9-187b-4c10-acb1-975c3498a176.ps1:2
Line |
   2 |  source <(cargo llvm-cov show-env --export-prefix --manifest-path=rust …
     |         ~
     | The '<' operator is reserved for future use.

https://github.com/LilyFoote/llvm-cov-windows-bug/actions/runs/5877498670/job/15937744529

[taiki-e]

I get a parser error:

@LilyFoote Could you try with shell: bash?

[LilyFoote]

I got that running, but we still have the original error:

https://github.com/LilyFoote/llvm-cov-windows-bug/actions/runs/5877711598/job/15938334102

[taiki-e]

I hid comments unrelated to this issue as off-topic.

• The issue @messense has encountered is misuse due to lack of clarity in the documentation or lack of warnings. The situation here will be improved by adding warnings for such cases. See (and subscribe) Emit runtime warning on cargo llvm-cov nextest run #302 and Emit warning when show-env is used together with cargo llvm-cov commands other than report/clean #305 if you are interested in the status of adding warnings.
• The issue @detly has encountered is the lack of nextest archive support. This is tracked in Any plans to make this work with --archive-file ? #265. The PR supporting it (Use extract-to if using --archive-file with nextest #266) is stalled, but the implementation itself should work (with my patch), so if there is no response from the PR author, I may add a commit to that PR and merge it. See (and subscribe) these issue and PR if you are interested in the status of supporting it.

And one of the reasons this issue has received unrelated additional reports is that the error itself is unclear. This is related to the fact that the error is handled on the llvm-tools side, and reporting the error with some hint on the cargo-llvm-cov side before passing arguments to llvm-tools should improve the situation (#306).

As for the issue @LilyFoote has encountered (the original topic of this issue -- I think that is cargo-llvm-cov or llvm-tools bug), I will fork your reproduction repo later and look into it. I'm guessing it is probably a windows path issue, but I'm still not sure what it really is.

[taiki-e]

Windows issue has been fixed in 0.5.30 (#307).

Also, diagnostics improvements (#302, #305, #306) have been released in 0.5.28 and 0.5.29.

[taiki-e]

• The issue @detly has encountered is the lack of nextest archive support. This is tracked in Any plans to make this work with --archive-file ? #265. The PR supporting it (Use extract-to if using --archive-file with nextest #266) is stalled, but the implementation itself should work (with my patch), so if there is no response from the PR author, I may add a commit to that PR and merge it. See (and subscribe) these issue and PR if you are interested in the status of supporting it.

Btw, this has been supported in v0.6.0.