Ptrace engine + no --implicit-test-threads causes passing quiche tests to fail

[johnDeSilencio]

Problem Statement

Tests in the quiche crate that pass when running cargo test are failing when executed by cargo tarpaulin under the following conditions:

1. ptrace is the coverage tracing backend used
2. The flag --implicit-test-threads is omitted

Expected Behavior

Tests that pass when executed by cargo test should also pass when executed by cargo tarpaulin

Steps to Reproduce

1. Clone the quiche repository
2. Navigate to the quiche/ subdirectory (quiche/quiche/)
3. Run the following command:

cargo tarpaulin --engine Ptrace

The following tests will fail:

failures:
    h3::tests::reset_finished_at_client
    h3::tests::reset_stream
    h3::tests::send_body_truncation_stream_blocked
    tests::collect_streams

test result: FAILED. 464 passed; 4 failed; 0 ignored; 0 measured; 0 filtered out; finished in 55.63s

I've attached the (compressed) ptrace dump to this issue as well as the verbose cargo tarpaulin output.

Workarounds

1. Using the Llvm engine instead of ptrace:

cargo tarpaulin --engine Llvm

# ...

test tests::version_negotiation ... ok
test tests::zero_length_new_token ... ok
test tests::zero_rtt ... ok

test result: ok. 468 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 23.48s

2. Using the --implicit-test-threads flag:

cargo tarpaulin --engine Ptrace --implicit-test-threads

# ...

test tests::version_negotiation ... ok
test tests::zero_length_new_token ... ok
test tests::zero_rtt ... ok

test result: ok. 468 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 43.08s

Tooling & System Versions

cargo-tarpaulin:

cargo-tarpaulin-tarpaulin 0.31.2

quiche:

master:6ae23f39bd997d81cf6e24e5db52108138f4bf83

rustc:

stable-x86_64-unknown-linux-gnu (default)
rustc 1.80.0 (051478957 2024-07-21)

cargo:

cargo 1.80.1 (376290515 2024-07-16)

My system (NixOS):

Linux framework 6.10.5 #1-NixOS SMP PREEMPT_DYNAMIC Wed Aug 14 13:34:38 UTC 2024 x86_64 GNU/Linux

ANSI_COLOR="1;34"
BUG_REPORT_URL="https://github.com/NixOS/nixpkgs/issues"
BUILD_ID="24.05.20240830.6e99f2a"
DOCUMENTATION_URL="https://nixos.org/learn.html"
HOME_URL="https://nixos.org/"
ID=nixos
IMAGE_ID=""
IMAGE_VERSION=""
LOGO="nix-snowflake"
NAME=NixOS
PRETTY_NAME="NixOS 24.05 (Uakari)"
SUPPORT_END="2024-12-31"
SUPPORT_URL="https://nixos.org/community.html"
VERSION="24.05 (Uakari)"
VERSION_CODENAME=uakari
VERSION_ID="24.05

---

I was hoping to get to the bottom of this issue myself, but I am way out of my depth. As someone doing this in his free time, I appreciate the free time of anyone who can look at this issue and evaluate it.

Thank you :) ⛵

[xd009642]

I'll have to look into it deeper to see where it's happening and why but it's not expected that every project will work perfectly with tarpaulin with the default configuration. I try and do a best-case default config that fits the most people.

Unfortunately, if I don't specify test threads explicitly it ends up using a single test thread because of some of the ptrace things. Anyone who's tests rely on CLI arguments or forwards them to another process may as a result experience test failures.

If the test also fails with cargo test -- --test-threads 4 then it shows that the quiche tests aren't working with out-of-the-box cargo test options available. If it's easy to detect and fix in tarpaulin I'll naturally seek to fix it, but if not this'll probably be closed as it's not fully my concern if a library decides to break cargo test features 🤷