Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: clearrange/checks=false failed #129026

Closed
cockroach-teamcity opened this issue Aug 15, 2024 · 1 comment · Fixed by #129088
Closed

roachtest: clearrange/checks=false failed #129026

cockroach-teamcity opened this issue Aug 15, 2024 · 1 comment · Fixed by #129088
Assignees
@itsbilal
Labels
A-storage Relating to our storage engine (Pebble) on-disk storage. branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-storage Storage Team

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Aug 15, 2024 •
edited by cockroach-jira-scripts
Loading

roachtest.clearrange/checks=false failed with artifacts on master @ 575cdd4696dfcac8f311d1ea546683271102f73e:

(cluster.go:2473).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
unexpected node event: n1: cockroach process for system interface died (exit code 134)
test artifacts and logs in: /artifacts/clearrange/checks=false/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

Jira issue: CRDB-41335
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-storage Storage Team labels Aug 15, 2024
@blathers-crl blathers-crl bot added the A-storage Relating to our storage engine (Pebble) on-disk storage. label Aug 15, 2024
@github-project-automation github-project-automation bot moved this to Incoming in Storage Aug 15, 2024
@jbowens
Copy link
Collaborator

jbowens commented Aug 15, 2024 

a panic has occurred!
panic: ‹pebble: start key (/Table/106/1/13229609/0,0) < flushed key (/Table/106/1/13268834/0/1723704620.106208117,0)›
(1) attached stack trace
  -- stack trace:
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:770
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover
  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:770
  | github.com/cockroachdb/pebble/internal/keyspan.(*Fragmenter).Add
  | 	github.com/cockroachdb/pebble/internal/keyspan/external/com_github_cockroachdb_pebble/internal/keyspan/fragmenter.go:146
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*multiSSTWriter).PutRangeKey
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:516
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:809
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1649
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:213
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:210
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:621
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267

@itsbilal looks like there's an out-of-order issue with the splitting of snapshot ingest sstables

This was referenced Aug 15, 2024
Closed
Closed
This was referenced Aug 15, 2024
Merged
Merged
@itsbilal itsbilal mentioned this issue Aug 15, 2024
Merged
itsbilal added a commit to itsbilal/cockroach that referenced this issue Aug 15, 2024
@itsbilal
kvserver: disable snapshot splitting of sstables for now
0801a13 
Informs cockroachdb#129026.

Epic: none

Release note: None
craig bot pushed a commit that referenced this issue Aug 15, 2024
@renatolabs @itsbilal
Merge #128849 #129063
a5b8e0c 
128849: roachtest: make graceful stop an option instead of a function r=DarrylWong a=renatolabs

Previously, a few tests would use the `StopCockroachGracefullyOnNode`
function when they wanted a graceful shutdown. However, the start/stop
functions are already able to take options from the caller.

In this commit, we replace the custom function to perform a graceful
shutdown with a `Graceful` option that can be passed to `Stop`. This
is now composable with other options that can also be passed to `Stop`.

Epic: none

Release note: None

129063: kvserver: disable snapshot splitting of sstables for now r=RaduBerinde a=itsbilal

Informs #129026.

Epic: none

Release note: None

Co-authored-by: Renato Costa <renato@cockroachlabs.com>
Co-authored-by: Bilal Akhtar <bilal@cockroachlabs.com>
@itsbilal itsbilal mentioned this issue Aug 15, 2024
Merged
itsbilal added a commit to itsbilal/cockroach that referenced this issue Aug 15, 2024
@itsbilal
kvserver: reenable splitting of snapshot sstables
6c1e4e2 
This change updates the snapshot strategy's sender side
to iterate over points and ranges together, instead of only
iterating on points first, then only ranges. This allows us to
more efficiently split snapshot sstables on the receiver side.
To avoid the need to add a version gate on the receiver side, we
propagate a bool, RangeKeysInOrder, to the receiver which is a signal
to it to enable sstable splits.

Fixes cockroachdb#129026.

Epic: none

Release note: None
itsbilal added a commit to itsbilal/cockroach that referenced this issue Aug 16, 2024
@itsbilal
kvserver: reenable splitting of snapshot sstables
a0f0dd3 
This change updates the snapshot strategy's sender side
to iterate over points and ranges together, instead of only
iterating on points first, then only ranges. This allows us to
more efficiently split snapshot sstables on the receiver side.
To avoid the need to add a version gate on the receiver side, we
propagate a bool, RangeKeysInOrder, to the receiver which is a signal
to it to enable sstable splits.

Fixes cockroachdb#129026.

Epic: none

Release note: None
craig bot pushed a commit that referenced this issue Aug 16, 2024
@itsbilal
Merge #129088
bdfba7a 
129088: kvserver: reenable splitting of snapshot sstables r=aadityasondhi a=itsbilal

This change updates the snapshot strategy's sender side to iterate over points and ranges together, instead of only iterating on points first, then only ranges. This allows us to more efficiently split snapshot sstables on the receiver side. To avoid the need to add a version gate on the receiver side, we propagate a bool, RangeKeysInOrder, to the receiver which is a signal to it to enable sstable splits.

Fixes #129026.

Epic: none

Release note: None

Co-authored-by: Bilal Akhtar <bilal@cockroachlabs.com>
@craig craig bot closed this as completed in ef16081 Aug 16, 2024
@github-project-automation github-project-automation bot moved this from Incoming to Done in Storage Aug 16, 2024
@jbowens jbowens mentioned this issue Aug 22, 2024
Closed
itsbilal added a commit that referenced this issue Sep 18, 2024
@itsbilal
kvserver: reenable splitting of snapshot sstables
94fb782 
This change updates the snapshot strategy's sender side
to iterate over points and ranges together, instead of only
iterating on points first, then only ranges. This allows us to
more efficiently split snapshot sstables on the receiver side.
To avoid the need to add a version gate on the receiver side, we
propagate a bool, RangeKeysInOrder, to the receiver which is a signal
to it to enable sstable splits.

Fixes #129026.

Epic: none

Release note: None
itsbilal added a commit to itsbilal/cockroach that referenced this issue Sep 30, 2024
@itsbilal
kvserver: reenable splitting of snapshot sstables
6abec1b 
This change updates the snapshot strategy's sender side
to iterate over points and ranges together, instead of only
iterating on points first, then only ranges. This allows us to
more efficiently split snapshot sstables on the receiver side.
To avoid the need to add a version gate on the receiver side, we
propagate a bool, RangeKeysInOrder, to the receiver which is a signal
to it to enable sstable splits.

Fixes cockroachdb#129026.

Epic: none

Release note: None
itsbilal added a commit to itsbilal/cockroach that referenced this issue Sep 30, 2024
@itsbilal
kvserver: reenable splitting of snapshot sstables
8c76056 
This change updates the snapshot strategy's sender side
to iterate over points and ranges together, instead of only
iterating on points first, then only ranges. This allows us to
more efficiently split snapshot sstables on the receiver side.
To avoid the need to add a version gate on the receiver side, we
propagate a bool, RangeKeysInOrder, to the receiver which is a signal
to it to enable sstable splits.

Fixes cockroachdb#129026.

Epic: none

Release note: None
itsbilal added a commit to itsbilal/cockroach that referenced this issue Nov 7, 2024
@itsbilal
kvserver: reenable splitting of snapshot sstables
6cc931b 
This change updates the snapshot strategy's sender side
to iterate over points and ranges together, instead of only
iterating on points first, then only ranges. This allows us to
more efficiently split snapshot sstables on the receiver side.
To avoid the need to add a version gate on the receiver side, we
propagate a bool, RangeKeysInOrder, to the receiver which is a signal
to it to enable sstable splits.

Fixes cockroachdb#129026.

Epic: none

Release note: None
itsbilal added a commit to itsbilal/cockroach that referenced this issue Nov 7, 2024
@itsbilal
kvserver: reenable splitting of snapshot sstables
e17115b 
This change updates the snapshot strategy's sender side
to iterate over points and ranges together, instead of only
iterating on points first, then only ranges. This allows us to
more efficiently split snapshot sstables on the receiver side.
To avoid the need to add a version gate on the receiver side, we
propagate a bool, RangeKeysInOrder, to the receiver which is a signal
to it to enable sstable splits.

Fixes cockroachdb#129026.

Epic: none

Release note: None
blathers-crl bot pushed a commit that referenced this issue Nov 12, 2024
@itsbilal
kvserver: reenable splitting of snapshot sstables
86ffb4e 
This change updates the snapshot strategy's sender side
to iterate over points and ranges together, instead of only
iterating on points first, then only ranges. This allows us to
more efficiently split snapshot sstables on the receiver side.
To avoid the need to add a version gate on the receiver side, we
propagate a bool, RangeKeysInOrder, to the receiver which is a signal
to it to enable sstable splits.

Fixes #129026.

Epic: none

Release note: None
itsbilal added a commit to itsbilal/cockroach that referenced this issue Nov 13, 2024
@itsbilal
kvserver: reenable splitting of snapshot sstables
73fc907 
This change updates the snapshot strategy's sender side
to iterate over points and ranges together, instead of only
iterating on points first, then only ranges. This allows us to
more efficiently split snapshot sstables on the receiver side.
To avoid the need to add a version gate on the receiver side, we
propagate a bool, RangeKeysInOrder, to the receiver which is a signal
to it to enable sstable splits.

Fixes cockroachdb#129026.

Epic: none

Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@itsbilal itsbilal

Labels
A-storage Relating to our storage engine (Pebble) on-disk storage. branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-storage Storage Team
Projects
Storage
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

kvserver: reenable splitting of snapshot sstables itsbilal/cockroach
3 participants
@itsbilal @jbowens @cockroach-teamcity