Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kv/kvnemesis: TestKVNemesisSingleNode failed #98473

Closed
cockroach-teamcity opened this issue Mar 13, 2023 · 1 comment · Fixed by #98519
Closed

kv/kvnemesis: TestKVNemesisSingleNode failed #98473

cockroach-teamcity opened this issue Mar 13, 2023 · 1 comment · Fixed by #98519
Assignees
@itsbilal
Labels
A-storage Relating to our storage engine (Pebble) on-disk storage. branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-storage Storage Team
Milestone
23.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Mar 13, 2023 •
edited by cockroach-jira-scripts
Loading

kv/kvnemesis.TestKVNemesisSingleNode failed with artifacts on master @ e4924e2b9be4a36d466beab53a80df9241df4783:

=== RUN   TestKVNemesisSingleNode
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode255520637
    test_log_scope.go:79: use -show-logs to present logs inline
    kvnemesis_test.go:180: seed: 2337241152468748031
    kvnemesis_test.go:124: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis4042989563
    kvnemesis.go:165: range 99 (/Table/100/"0b55994aa55092f8") RANGE_CONSISTENT_STATS_INCORRECT:
        stats: {ContainsEstimates:0 LastUpdateNanos:1678692209549415201 IntentAge:0 GCBytesAge:33881 LiveBytes:0 LiveCount:0 KeyBytes:1710 KeyCount:34 ValBytes:1258 ValCount:83 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:3 RangeKeyBytes:154 RangeValCount:4 RangeValBytes:42 SysBytes:3732 SysCount:25 AbortSpanBytes:1534}
        delta (stats-computed): {ContainsEstimates:0 LastUpdateNanos:1678692209549415201 IntentAge:0 GCBytesAge:-1760 LiveBytes:0 LiveCount:0 KeyBytes:0 KeyCount:0 ValBytes:0 ValCount:0 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:-1 RangeKeyBytes:-68 RangeValCount:-2 RangeValBytes:-20 SysBytes:0 SysCount:0 AbortSpanBytes:0}
    kvnemesis.go:185: failures(verbose): /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis4042989563/failures
        repro steps: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis4042989563/repro.go
        rangefeed KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis4042989563/kvs-rangefeed.txt
        scan KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis4042989563/kvs-scan.txt
    kvnemesis_test.go:207: 
        	Error Trace:	/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2297/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/kv/kvnemesis/kvnemesis_test_/kvnemesis_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvnemesis/kvnemesis_test.go:207
        	            				/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2297/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/kv/kvnemesis/kvnemesis_test_/kvnemesis_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvnemesis/kvnemesis_test.go:148
        	Error:      	Should be zero, but was 1
        	Test:       	TestKVNemesisSingleNode
        	Messages:   	kvnemesis detected failures
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode255520637
--- FAIL: TestKVNemesisSingleNode (25.43s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-25284
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Mar 13, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.1 milestone Mar 13, 2023
@blathers-crl blathers-crl bot added the T-kv KV Team label Mar 13, 2023
@erikgrinaker
Copy link
Contributor

@itsbilal Seems like #98426 didn't fully fix things, going to disable AddSSTable for now.

@erikgrinaker erikgrinaker added T-storage Storage Team and removed T-kv KV Team labels Mar 13, 2023
@blathers-crl blathers-crl bot added the A-storage Relating to our storage engine (Pebble) on-disk storage. label Mar 13, 2023
This was referenced Mar 13, 2023
Closed
Closed
Merged
craig bot pushed a commit that referenced this issue Mar 13, 2023
@samiskin @erikgrinaker
Merge #98472 #98475
6a41a2f 
98472: changefeedccl: enable TestChangefeedPropagatesTerminalError r=samiskin a=samiskin

Fixes: #95057

The test TestChangefeedPropagatesTerminalError no longer seems to flake after running it under stress on a GCE worker for a while, so this change re-enables it.

Release note: None

98475: kvnemesis: disable `AddSSTable` range keys r=erikgrinaker a=erikgrinaker

These trigger MVCC stats bugs in `CheckSSTConflicts`.

Touches #94141.
Touches #98473.
Touches #94876.

Epic: none
Release note: None

Co-authored-by: Shiranka Miskin <shiranka.miskin@gmail.com>
Co-authored-by: Erik Grinaker <grinaker@cockroachlabs.com>
itsbilal added a commit to itsbilal/cockroach that referenced this issue Mar 13, 2023
@itsbilal
storage: More CheckSSTConflicts fixes
34bffa5 
A few additional fixes around CheckSSTConflicts, stats
calculations, and Next()ing logic, caught by kvnemesis.
Hopefully the last of its kind.

Fixes cockroachdb#94141.
Fixes cockroachdb#98473.
Informs cockroachdb#94876.

Epic: none

Release note: None
@itsbilal itsbilal mentioned this issue Mar 13, 2023
Merged
@nvanbenschoten nvanbenschoten mentioned this issue Mar 13, 2023
Closed
craig bot pushed a commit that referenced this issue Mar 14, 2023
@ajwerner @itsbilal @msbutler
Merge #98368 #98519 #98567
ca5ae38 
98368: multiregionccl,server: use cached sqlliveness.Reader, deflake ColdStartLatencyTest r=ajwerner a=ajwerner

#### multitenantccl: deflake ColdStartLatencyTest
This test was flakey due to the closed timestamp sometimes not leading far
for global tables due to overload, and due to a cached liveness reader
not being used in distsql. The former was fixed in previous commits. The
latter is fixed here.

Fixes: #96334


#### sql: use CachedReader for uses with sqlinstance and the sql builtins
The CachedReader won't block, which in multi-region clusters is good. It will
mean that in some cases, it'll state that a sessions is alive when it most
certainly is not. Currently, nobody needs synchronous semantics.

This is a major part of fixing the TestColdStartLatency as sometimes
distsql planning would block. That's not acceptable -- the idea that
query physical planning can need to wait for a cross-region RPC is
unacceptable.

#### sqlliveness: re-arrange APIs to clarify when the API was blocking

By "default" the implementation of Reader was blocking and there was a method
to get a handle to a non-blocking CachedReader(). This asymmetry does not aid
understandability. The non-blocking reader came later, but it is generally the
more desirable interface.

Release note: None

98519: storage: More CheckSSTConflicts fixes r=erikgrinaker a=itsbilal

A few additional fixes around CheckSSTConflicts, stats calculations, and Next()ing logic, caught by kvnemesis. Hopefully the last of its kind.

Also re-enable kvnemesis testing for range keys in AddSSTable, reverting #98475.

Fixes #94141.
Fixes #98473.
Informs #94876.

Epic: none

Release note: None

98567: backupccl: use correct version gate for restore checkpointing r=adityamaru a=msbutler

PR #97862 introduced a subtle bug which allowed the new restore checkpointing policy to take effect before the 23_1 migrations occured. This patch ensures the new policy only takes effect after all migrations occur.

Release note: None

Epic: None

Co-authored-by: ajwerner <awerner32@gmail.com>
Co-authored-by: Bilal Akhtar <bilal@cockroachlabs.com>
Co-authored-by: Michael Butler <butler@cockroachlabs.com>
@craig craig bot closed this as completed in 9ec7760 Mar 14, 2023
@jbowens jbowens moved this to Done in [Deprecated] Storage Jun 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@itsbilal itsbilal

Labels
A-storage Relating to our storage engine (Pebble) on-disk storage. branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-storage Storage Team
Projects
No open projects
[Deprecated] Storage
Archived in project
Milestone
23.1
Development

Successfully merging a pull request may close this issue.

storage: More CheckSSTConflicts fixes itsbilal/cockroach
3 participants
@itsbilal @erikgrinaker @cockroach-teamcity