kv/kvnemesis: TestKVNemesisMultiNode failed #118471
Assignees
Labels
A-testing
Testing tools and infrastructure
branch-release-23.2
Used to mark GA and release blockers, technical advisories, and bugs for 23.2
C-bug
Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.
C-test-failure
Broken test (automatically or manually discovered).
O-robot
Originated from a bot.
P-3
Issues/test failures with no fix SLA
T-kv
KV Team
Milestone
Comments
cockroach-teamcity
added
branch-release-23.2
|
Duplicate of #118017. |
kvoli
added
X-duplicate
|
Pretty sure this is a test issue, as described here: #118017 (comment) @pav-kv Can you confirm, and remove the blocker label in that case? |
|
There is no raft log tail for this one to confirm 100%. But the signature looks the same. A LAI 3 proposal got reassigned LAI 1 (by this test), and accidentally overtaken a legit LAI 2 proposal. |
pav-kv
removed
the
release-blocker
This was referenced Feb 2, 2024
|
kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on release-23.2 @ 11ad4340c4949d16546de5f4064357d7c39882fe: Fatal error: Stack: Log preceding fatal error
Parameters:
Same failure on other branches
|
|
kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on release-23.2 @ 0df66c4306badb5116db98f43baa4ea39aade92f: Parameters:
Same failure on other branches
|
|
kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on release-23.2 @ 3790ccda4a0d705525ace94aaff1793f2a5e9c2f: Fatal error: Stack: Log preceding fatal error
Parameters:
Same failure on other branches
|
This was referenced Apr 8, 2024
|
kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on release-23.2 @ e61e140a75b2f7dde9f5410c1d90e557d009c437: Parameters:
Same failure on other branches
|
|
kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on release-23.2 @ d91ca866b265a837f832bc93b59481847d721f18: Fatal error: Stack: Log preceding fatal error
Parameters:
|
|
I'm surprised this failed again given we backported #122067 to 23.2. The failure mode looks similar at first glance, but I'll dig in more. |
arulajmani
added
P-3
|
kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on release-23.2 @ a36883a03ce0b30af0c025ff2b0de2f3845ad8d7: Fatal error: Stack: Log preceding fatal error
Parameters:
|
nvanbenschoten
added
C-bug
|
We have marked this test failure issue as stale because it has been |
|
kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on release-23.2 @ f53be59b1ef3f769fda678812d2c5a17fd817a63: Fatal error: Stack: Log preceding fatal error
Parameters:
Same failure on other branches
|
|
kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on release-23.2 @ 9e7ed1f7ee5346ea2357389f06963e411fd49306: Fatal error: Stack: Log preceding fatal error
Parameters:
Same failure on other branches
|
|
kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on release-23.2 @ 52d7f3db720359153214894c6dd3b4f96bd8be6b: Parameters:
Same failure on other branches
|
This was referenced Sep 19, 2024
|
I looked through the test failures here -- most seem to be because of CTS regressions, for which I've created a new tracking issue (#131450). Others are because of inconsistent range stats, which should be addressed by #131446. The only failure that doesn't fall into this bucket is #118471 (comment). However, the artifacts for it are long gone. I'll close this issue out now that we have #131450 to track most of the failures. That way, new KVNemesis failures will pop up in our triage queues. |
Should be fixed by #131503. |
craig bot
pushed a commit
that referenced
this issue
Sep 27, 2024
131503: apply: fix initial snapshot assertion r=arulajmani a=pav-kv The broad assertion here is that all the entries composing a snapshot must have been applied by someone before (at least the sender of the snapshot has done it, otherwise it couldn’t have sent this snapshot). And then there is an additional assertion (which failed occasionally before this PR) that there can’t be a snapshot without at least one entry applied. This is generally true, except that there are entries not registered by the "apply" stack: particularly, empty entries that raft leader appends at the start of its term. There is a comment to this extent in the `asserter.go` code: https://github.com/cockroachdb/cockroach/blob/5b0371b44009ac2ae3f58ea7ed95b290dd4e8227/pkg/kv/kvserver/apply/asserter.go#L254-L259 This commit modifies the snapshot assertion to correctly handle this case. It is possible that an initial snapshot sent from leader to followers contains only this dummy entry, and there were no "real" proposals applied. Fixes the false assertion failure in #118471 (comment) Related to #116319 Co-authored-by: Pavel Kalinnikov <pavel@cockroachlabs.com>
This was referenced Sep 27, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on release-23.2 @ 1a497ceea8c72d2a853404a9c5a0020f7509fadf:
Fatal error:
Stack:
Log preceding fatal error
Parameters:
TAGS=bazel,gss,deadlockstress=trueHelp
See also: How To Investigate a Go Test Failure (internal)
Same failure on other branches
This test on roachdash | Improve this report!
Jira issue: CRDB-35752
The text was updated successfully, but these errors were encountered: