ci(test): use cargo nextest to speed up the test #30

pan93412 · 2022-07-08T15:20:39Z

Why introducing this change?

The current cargo test is so slow that can't complete the CI in a reasonable period.

The features of Nextest

Clean, beautiful user interface. Nextest presents its results concisely so you can see which tests passed and failed at a glance.
Up to 60% faster than cargo test. Nextest uses a state-of-the-art execution model for faster, more reliable test runs.
Identify slow tests. Use nextest to detect tests that take a long time to run, and identify bottlenecks during test execution.
Designed for CI. Nextest addresses real-world pain points with continuous integration scenarios.
Use pre-built binaries for quick installation.
Set up CI-specific configuration profiles.
Reuse builds and partition test runs across multiple CI jobs. (Check out this example on GitHub Actions).
Automatically retry failing tests, and mark them as flaky if they pass later.
Print failing output at the end of test runs.
Output information about test runs as JUnit XML.
Cross-platform. Nextest works on Linux and other Unixes, Mac and Windows, so you get the benefits of faster test runs no matter what platform you use.
... and more coming soon!

Source: https://nexte.st

Why separating `nextest` and `doctest`?

cargo nextest hasn't supported doctest.
cargo nextest seems currently failed so cargo test --doc can't be run.

…er#30) crypto-crawler#30

soulmachine · 2022-07-09T00:25:38Z

Would you mind explaining why Nextest is faster than cargo test?

pan93412 · 2022-07-09T00:27:34Z

Would you mind explaining why Nextest is faster than cargo test?

The original cargo test timed out due to a blocking test:

https://github.com/crypto-crawler/crypto-crawler-rs/runs/7249902584?check_suite_focus=true

While nextest ran it in a reasonable time:

https://github.com/pan93412/crypto-crawler-rs/runs/7254465565?check_suite_focus=true

Sorry for forgot pasting the actions link here.

pan93412 · 2022-07-09T01:01:04Z

@soulmachine The cargo nextest run test has been done – It took about 32 minutes instead of infinitely.

soulmachine · 2022-07-09T19:36:43Z

@soulmachine The cargo nextest run test has been done – It took about 32 minutes instead of infinitely.

nextest run many test cases in parallel, I think this is why it is fast. But it caused 429 Too Many Requests, can you reduce its threads?

pan93412 · 2022-07-10T06:33:38Z

blocking: nextest-rs/nextest#158 (comment)

soulmachine · 2022-07-10T09:06:40Z

blocking: nextest-rs/nextest#158 (comment)

Got it , let's wait for a few days. The goal here is to make cargo test pass.

pan93412 · 2022-07-18T05:07:59Z

No 429 issues anymore; however, some tests did not still passed.

Bitmex

        FAIL [   3.811s]     crypto-markets::bitmex fetch_all_symbols

--- STDOUT:                  crypto-markets::bitmex fetch_all_symbols ---

running 1 test
test fetch_all_symbols ... FAILED

failures:

failures:
    fetch_all_symbols

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 13 filtered out; finished in 3.80s


--- STDERR:                  crypto-markets::bitmex fetch_all_symbols ---
thread 'fetch_all_symbols' panicked at 'assertion failed: !symbols.is_empty()', crypto-markets/tests/bitmex.rs:13:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.768s]     crypto-markets::bitmex fetch_quanto_future_symbols

--- STDOUT:                  crypto-markets::bitmex fetch_quanto_future_symbols ---

running 1 test
test fetch_quanto_future_symbols ... FAILED

failures:

failures:
    fetch_quanto_future_symbols

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 13 filtered out; finished in 0.76s


--- STDERR:                  crypto-markets::bitmex fetch_quanto_future_symbols ---
thread 'fetch_quanto_future_symbols' panicked at 'assertion failed: !symbols.is_empty()', crypto-markets/tests/bitmex.rs:85:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

------------
     Summary [ 152.534s] 78 tests run: 76 passed, 2 failed, 2 skipped
error: test run failed

Others

        FAIL [   0.007s]                      crypto-message compact::message::tests::test_calculate_hash

--- STDOUT:                                   crypto-message compact::message::tests::test_calculate_hash ---

running 1 test
test compact::message::tests::test_calculate_hash ... FAILED

failures:

failures:
    compact::message::tests::test_calculate_hash

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 6 filtered out; finished in 0.00s

--- STDERR:                                   crypto-message compact::message::tests::test_calculate_hash ---
thread 'compact::message::tests::test_calculate_hash' panicked at 'assertion failed: `(left == right)`
  left: `12658250145686044913`,
 right: `4200729777880827930`', crypto-message/src/compact/message.rs:527:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

     TIMEOUT [ 120.118s]               crypto-crawler::huobi test_crawl_candlestick::_markettype_spot

--- STDOUT:                            crypto-crawler::huobi test_crawl_candlestick::_markettype_spot ---

running 1 test
test test_crawl_candlestick::_markettype_spot has been running for over 60 seconds

        FAIL [   0.007s]           crypto-msg-parser::okx_v3 l2_event::option_snapshot

--- STDOUT:                        crypto-msg-parser::okx_v3 l2_event::option_snapshot ---

running 1 test
test l2_event::option_snapshot ... FAILED

failures:

failures:
    l2_event::option_snapshot

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 13 filtered out; finished in 0.00s


--- STDERR:                        crypto-msg-parser::okx_v3 l2_event::option_snapshot ---
thread 'l2_event::option_snapshot' panicked at 'assertion failed: `(left == right)`
  left: `906.0`,
 right: `90.60000000000001`', crypto-msg-parser/tests/okx_v3.rs:484:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.006s]           crypto-msg-parser::okx_v3 trade::option

--- STDOUT:                        crypto-msg-parser::okx_v3 trade::option ---

running 1 test
test trade::option ... FAILED

failures:

failures:
    trade::option

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 13 filtered out; finished in 0.00s


--- STDERR:                        crypto-msg-parser::okx_v3 trade::option ---
thread 'trade::option' panicked at 'assertion failed: `(left == right)`
  left: `4.0`,
 right: `0.4`', crypto-msg-parser/tests/okx_v3.rs:179:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.108s]              crypto-rest-client::zb test_spot_l2_snapshot

--- STDOUT:                           crypto-rest-client::zb test_spot_l2_snapshot ---

running 1 test
test test_spot_l2_snapshot ... FAILED

failures:

failures:
    test_spot_l2_snapshot

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 1 filtered out; finished in 0.09s


--- STDERR:                           crypto-rest-client::zb test_spot_l2_snapshot ---
thread 'test_spot_l2_snapshot' panicked at 'called `Result::unwrap()` on an `Err` value: Error("Failed zb spot btc_usdt after retrying 3 times")', crypto-rest-client/tests/zb.rs:9:79
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

     TIMEOUT [ 120.111s]                  crypto-crawler::zb test_crawl_l2_snapshot::_markettype_spot_btc_usdt_

--- STDOUT:                               crypto-crawler::zb test_crawl_l2_snapshot::_markettype_spot_btc_usdt_ ---

running 1 test
test test_crawl_l2_snapshot::_markettype_spot_btc_usdt_ has been running for over 60 seconds

        SLOW [>120.000s]                  crypto-crawler::zb test_crawl_l2_snapshot_without_symbol::_markettype_spot
     TIMEOUT [ 120.111s]                  crypto-crawler::zb test_crawl_l2_snapshot_without_symbol::_markettype_spot

--- STDOUT:                               crypto-crawler::zb test_crawl_l2_snapshot_without_symbol::_markettype_spot ---

running 1 test
test test_crawl_l2_snapshot_without_symbol::_markettype_spot has been running for over 60 seconds

------------
     Summary [ 585.959s] 1609 tests run: 1602 passed, 4 failed, 3 timed out, 118 skipped

Seems all of them should not be related to this PR.

soulmachine · 2022-07-18T13:16:51Z

Hi @pan93412 , thank you for the hard work, we're closer! I'll help you investigate these failed test cases.

.github/workflows/ci.yml

soulmachine · 2022-07-18T13:43:46Z

@pan93412 can you give me push access to https://github.com/pan93412/crypto-crawler-rs so that I can add commits to this PR?

soulmachine · 2022-07-18T13:46:30Z

@pan93412 To make all tests pass, please comment out line 59 and 71 of crypto-crawler/tests/zb.rs

soulmachine

Tested locally, all cases passed

* ci(test): use cargo nextest to speed up the test * ci: separate test to doc-test and nextest * fix(ci/test): slow down the test speed * fix(ci): slow down the bitmex tests * Formatted ci.yml Co-authored-by: Frank Dai <soulmachine@gmail.com>

pan93412 added a commit to pan93412/crypto-crawler-rs that referenced this pull request Jul 8, 2022

merge: ci(test): use cargo nextest to speed up the test (crypto-crawl…

3360e0a

…er#30) crypto-crawler#30

pan93412 marked this pull request as draft July 10, 2022 05:44

pan93412 added 3 commits July 18, 2022 12:51

ci(test): use cargo nextest to speed up the test

f39044b

ci: separate test to doc-test and nextest

83c1c51

fix(ci/test): slow down the test speed

2ad41b1

fix(ci): slow down the bitmex tests

cc6031b

pan93412 force-pushed the up/add-nextest branch from d091ab1 to cc6031b Compare July 18, 2022 05:10

pan93412 marked this pull request as ready for review July 18, 2022 05:11

soulmachine reviewed Jul 18, 2022

View reviewed changes

.github/workflows/ci.yml Outdated Show resolved Hide resolved

soulmachine added 3 commits July 18, 2022 20:38

Merge branch 'main' into up/add-nextest

9d14ab6

Formatted ci.yml

04f013f

Merge branch 'main' into up/add-nextest

4db1e6e

soulmachine force-pushed the main branch from 04f013f to 96d7126 Compare July 19, 2022 04:00

soulmachine approved these changes Jul 19, 2022

View reviewed changes

soulmachine merged commit d221376 into crypto-crawler:main Jul 19, 2022

pan93412 deleted the up/add-nextest branch July 19, 2022 05:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci(test): use cargo nextest to speed up the test #30

ci(test): use cargo nextest to speed up the test #30

pan93412 commented Jul 8, 2022 •

edited

Loading

soulmachine commented Jul 9, 2022

pan93412 commented Jul 9, 2022 •

edited

Loading

pan93412 commented Jul 9, 2022

soulmachine commented Jul 9, 2022 •

edited

Loading

pan93412 commented Jul 10, 2022

soulmachine commented Jul 10, 2022

pan93412 commented Jul 18, 2022

soulmachine commented Jul 18, 2022

soulmachine commented Jul 18, 2022

soulmachine commented Jul 18, 2022

soulmachine left a comment

ci(test): use cargo nextest to speed up the test #30

ci(test): use cargo nextest to speed up the test #30

Conversation

pan93412 commented Jul 8, 2022 • edited Loading

Why introducing this change?

The features of Nextest

Why separating nextest and doctest?

soulmachine commented Jul 9, 2022

pan93412 commented Jul 9, 2022 • edited Loading

pan93412 commented Jul 9, 2022

soulmachine commented Jul 9, 2022 • edited Loading

pan93412 commented Jul 10, 2022

soulmachine commented Jul 10, 2022

pan93412 commented Jul 18, 2022

Bitmex

Others

soulmachine commented Jul 18, 2022

soulmachine commented Jul 18, 2022

soulmachine commented Jul 18, 2022

soulmachine left a comment

Choose a reason for hiding this comment

pan93412 commented Jul 8, 2022 •

edited

Loading

Why separating `nextest` and `doctest`?

pan93412 commented Jul 9, 2022 •

edited

Loading

soulmachine commented Jul 9, 2022 •

edited

Loading