Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use nextest for better test isolation #8207

Merged
merged 5 commits into from
Dec 16, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 9 additions & 4 deletions .buildkite/pipeline.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,8 @@ steps:
- label: "cargo test integration-tests*"
command: |
source ~/.cargo/env && set -eux
cargo test --locked -p 'integration-tests'
cargo install cargo-nextest
cargo nextest run --locked -p 'integration-tests'

timeout: 60
agents:
Expand All @@ -28,7 +29,8 @@ steps:
- label: "cargo test not integration-tests*"
command: |
source ~/.cargo/env && set -eux
cargo test --locked --workspace -p '*' --exclude 'integration-tests*'
cargo install cargo-nextest
cargo nextest run --locked --workspace -p '*' --exclude 'integration-tests*'

timeout: 60
agents:
Expand All @@ -38,7 +40,8 @@ steps:
- label: "cargo test nightly integration-tests*"
command: |
source ~/.cargo/env && set -eux
cargo test --features nightly,test_features -p 'integration-tests'
cargo install cargo-nextest
cargo nextest run --features nightly,test_features -p 'integration-tests'

timeout: 60
agents:
Expand All @@ -48,7 +51,9 @@ steps:
- label: "cargo test nightly not integration-tests*"
command: |
source ~/.cargo/env && set -eux
cargo test --workspace --features nightly,test_features,mock_node -p '*' --exclude 'integration-tests*'
cargo install cargo-nextest
cargo nextest run --workspace --features nightly,test_features,mock_node -p '*' --exclude 'integration-tests*'
pompon0 marked this conversation as resolved.
Show resolved Hide resolved
cargo test --doc

timeout: 60
agents:
Expand Down
2 changes: 2 additions & 0 deletions .config/nextest.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
[profile.default]
slow-timeout = { period = "60s", terminate-after = 2, grace-period = "0s" }
15 changes: 9 additions & 6 deletions docs/practices/testing/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,10 @@
4. The general rule of thumb for a reviewer is to first review the tests, and
ensure that they can convince themselves that the code change that passes the
tests must be correct. Only then the code should be reviewed.
5. Have the assertions in the tests as specific as possible. For example, do not
do `assert!(result.is_err())`, expect the specific error instead.
5. Have the assertions in the tests as specific as possible,
however do not make the tests change-detectors of the concrete implementation.
(assert only properties which are required for correctness).
For example, do not do `assert!(result.is_err())`, expect the specific error instead.

# Tests hierarchy

Expand All @@ -21,19 +23,20 @@ In NEAR Reference Client we largely split tests into three categories:
1. Relatively cheap sanity or fast fuzz tests. It includes all the `#[test]`
Rust tests not decorated by features. Our repo is configured in such a way
that all such tests are ran on every PR, and failing at least one of them is
blocking the PR from being pushed.
blocking the PR from being merged.

To run such tests locally run `cargo test --all`
To run such tests locally run `cargo nextest run --all`.
It requires nextest harness which can be installed by running `cargo install cargo-nextest` first.

2. Expensive tests. This includes all the fuzzy tests that run many iterations,
as well as tests that spin up multiple nodes and run them until they reach a
certain condition. Such tests are decorated with
`#[cfg(feature="expensive-tests")]`. It is not trivial to enable features
that are not declared in the top level crate, and thus the easiest way to run
such tests is to enable all the features by passing `--all-features` to
`cargo test`, e.g:
`cargo nextest run`, e.g:

`cargo test --package near-client --test cross_shard_tx
`cargo nextest run --package near-client --test cross_shard_tx
tests::test_cross_shard_tx --all-features`

3. Python tests. We have an infrastructure to spin up nodes, both locally and
Expand Down