Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Storage prefetch tests are flaky #8252

Closed
pugachAG opened this issue Dec 20, 2022 · 3 comments · Fixed by #8255
Closed

Storage prefetch tests are flaky #8252

pugachAG opened this issue Dec 20, 2022 · 3 comments · Fixed by #8255
Assignees
@pugachAG

Comments

@pugachAG
Copy link
Contributor

pugachAG commented Dec 20, 2022 •
edited
Loading

For more context see the discussion in zulip

Recent changes that might affect this test:

Some failures instances
@pugachAG pugachAG changed the title Storage prefetch tests are falky Storage prefetch tests are flaky Dec 20, 2022
@pugachAG pugachAG self-assigned this Dec 20, 2022
jakmeier added a commit to jakmeier/nearcore that referenced this issue Dec 20, 2022
@jakmeier
fix: flaky prefetcher tests
b054443 
Prefetcher tests have become flaky in CI lately, failing with
`"unexpected number of prefetched values"`.

I was not able to reproduce it locally. But there is a timeout that
is at least one source for flakyness. This commit removes this
source by properly waiting on the background threads to finish
instead of relying on a timeout.

Hopefully resolves  near#8252. (Will have to reopen if not.)
@jakmeier jakmeier mentioned this issue Dec 20, 2022
Merged
near-bulldozer bot pushed a commit that referenced this issue Dec 21, 2022
@jakmeier
fix: flaky prefetcher tests (#8255)
d57ae75 
Prefetcher tests have become flaky in CI lately, failing with
`"unexpected number of prefetched values"`.

I was not able to reproduce it locally. But there is a timeout that
is at least one source for flakyness. This commit removes this
source by properly waiting on the background threads to finish
instead of relying on a timeout.

Hopefully resolves  #8252. (Will have to reopen if not.)
@jakmeier
Copy link
Contributor

jakmeier commented Jan 3, 2023

This failed again, we need to investigate further...
https://buildkite.com/nearprotocol/nearcore/builds/23557#018573b7-8acf-4b34-b7e5-79da4c8c6f55/97-1488

thread 'prefetch::tests::test_prefetch_multiple_accounts' panicked at 'assertion failed: `(left == right)`
  left: `6`,
 right: `10`: unexpected number of prefetched values', runtime/runtime/src/prefetch.rs:287:9
stack backtrace:
   0: rust_begin_unwind
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/panicking.rs:142:14
   2: core::panicking::assert_failed_inner
   3: core::panicking::assert_failed
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/panicking.rs:181:5
   4: node_runtime::prefetch::tests::check_prefetch_account
             at ./src/prefetch.rs:352:9
   5: node_runtime::prefetch::tests::test_prefetch_multiple_accounts
             at ./src/prefetch.rs:287:9
   6: node_runtime::prefetch::tests::test_prefetch_multiple_accounts::{{closure}}
             at ./src/prefetch.rs:239:5
   7: core::ops::function::FnOnce::call_once
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/ops/function.rs:248:5
   8: core::ops::function::FnOnce::call_once
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/ops/function.rs:248:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

@jakmeier jakmeier reopened this Jan 3, 2023
@pugachAG
Copy link
Contributor Author

pugachAG commented Jan 3, 2023

@jakmeier it seems like your PR is based on the outdated master branch which doesn't include the fix:
jakmeier/nearcore@doc-meta-txgas...near:nearcore:master
or am I missing something?

@jakmeier
Copy link
Contributor

jakmeier commented Jan 3, 2023

Oh haha I am stupid, you are right. Good catch @pugachAG , in that case we can close it again. :)

@jakmeier jakmeier closed this as completed Jan 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pugachAG pugachAG

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: flaky prefetcher tests jakmeier/nearcore
2 participants
@pugachAG @jakmeier