use nextest for better test isolation (#8207)

it will allow us to set panic=abort per test (because each test is executed in a separate subprocess) and set test timeouts without developing a custom test harness. Additionally I've set the default test timeout to 2min (period = 1min, killing test after 2 periods), which seems reasonable, given that all presubmit tests execute <1min as of today.

.buildkite/pipeline.yml

@@ -18,7 +18,8 @@ steps:
   - label: "cargo test integration-tests*"
     command: |
       source ~/.cargo/env && set -eux
-      cargo test --locked -p 'integration-tests'
+      cargo install cargo-nextest
+      cargo nextest run --locked -p 'integration-tests'
 
     timeout: 60
     agents:
@@ -28,7 +29,8 @@ steps:
   - label: "cargo test not integration-tests*"
     command: |
       source ~/.cargo/env && set -eux
-      cargo test --locked --workspace -p '*' --exclude 'integration-tests*'
+      cargo install cargo-nextest
+      cargo nextest run --locked --workspace -p '*' --exclude 'integration-tests*'
 
     timeout: 60
     agents:
@@ -38,7 +40,8 @@ steps:
   - label: "cargo test nightly integration-tests*"
     command: |
       source ~/.cargo/env && set -eux
-      cargo test --features nightly,test_features -p 'integration-tests'
+      cargo install cargo-nextest
+      cargo nextest run --features nightly,test_features -p 'integration-tests'
 
     timeout: 60
     agents:
@@ -48,7 +51,9 @@ steps:
   - label: "cargo test nightly not integration-tests*"
     command: |
       source ~/.cargo/env && set -eux
-      cargo test --workspace --features nightly,test_features,mock_node -p '*' --exclude 'integration-tests*'
+      cargo install cargo-nextest
+      cargo nextest run --workspace --features nightly,test_features,mock_node -p '*' --exclude 'integration-tests*'
+      cargo test --doc
 
     timeout: 60
     agents:

.config/nextest.toml

@@ -0,0 +1,2 @@
+[profile.default]
+slow-timeout = { period = "60s", terminate-after = 2, grace-period = "0s" }

docs/practices/testing/README.md

@@ -11,8 +11,10 @@
 4. The general rule of thumb for a reviewer is to first review the tests, and
    ensure that they can convince themselves that the code change that passes the
    tests must be correct. Only then the code should be reviewed.
-5. Have the assertions in the tests as specific as possible. For example, do not
-   do `assert!(result.is_err())`, expect the specific error instead.
+5. Have the assertions in the tests as specific as possible,
+   however do not make the tests change-detectors of the concrete implementation.
+   (assert only properties which are required for correctness).
+   For example, do not do `assert!(result.is_err())`, expect the specific error instead.
 
 # Tests hierarchy
 
@@ -21,19 +23,20 @@ In NEAR Reference Client we largely split tests into three categories:
 1. Relatively cheap sanity or fast fuzz tests. It includes all the `#[test]`
    Rust tests not decorated by features. Our repo is configured in such a way
    that all such tests are ran on every PR, and failing at least one of them is
-   blocking the PR from being pushed.
+   blocking the PR from being merged.
 
-To run such tests locally run `cargo test --all`
+To run such tests locally run `cargo nextest run --all`.
+It requires nextest harness which can be installed by running `cargo install cargo-nextest` first.
 
 2. Expensive tests. This includes all the fuzzy tests that run many iterations,
    as well as tests that spin up multiple nodes and run them until they reach a
    certain condition. Such tests are decorated with
    `#[cfg(feature="expensive-tests")]`. It is not trivial to enable features
    that are not declared in the top level crate, and thus the easiest way to run
    such tests is to enable all the features by passing `--all-features` to
-   `cargo test`, e.g:
+   `cargo nextest run`, e.g:
 
-`cargo test --package near-client --test cross_shard_tx
+`cargo nextest run --package near-client --test cross_shard_tx
 tests::test_cross_shard_tx --all-features`
 
 3. Python tests. We have an infrastructure to spin up nodes, both locally and