Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sentry: panic.go:914: panic: must not be holding latches (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:914 | github.com/cockroachdb/cockroach/pkg/kv... #134785

Open
cockroach-sentry opened this issue Nov 10, 2024 · 0 comments
Labels
branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.

Comments

@cockroach-sentry
Copy link
Collaborator

cockroach-sentry commented Nov 10, 2024

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/6055108610/?referrer=webhooks_plugin

Panic Message:

panic.go:914: panic: must not be holding latches
(1) attached stack trace
  -- stack trace:
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:914
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes.func1
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:109
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:920
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:304
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:241
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:468
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:189
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:202
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
  | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1328
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
  | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1462
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:704
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12
  | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:169
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:714
  | github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func8
  | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:228
  | github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:899
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:784
  | github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:907
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:211
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:189
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2423
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1920
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1488
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendParallelCommit
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1287
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1102
Wraps: (2) panic: must not be holding latches
  | this is tracked in github.com/cockroachdb/cockroach/issues/77663; please comment if seen
  | eval_kind=0, holding_latches=true, branch=1, first_iteration=true, stack=
  | goroutine 950307 [running]:
  | runtime/debug.Stack()
  | 	GOROOT/src/runtime/debug/stack.go:24 +0x64
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard(0x4003ccdae0, {0x70575e8, 0x4009b4e9c0}, 0x400b791440, 0x0?)
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:307 +0x84c
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq(0x0?, {0x70575e8, 0x4009b4e9c0}, 0x400de98400?, {0x400a3ae480, {0x18068d9f53d7180b, 0x0, 0x0}, 0x0, 0x0, ...}, ...)
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:241 +0x228
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0x4003cf9200, {0x70575e8, 0x4009b4e9c0}, 0x400a3ae6c0, 0x5d74490)
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:468 +0x1e8
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes(0x4003cf9200, {0x70575e8?, 0x4009b4e990?}, 0x400a3ae6c0)
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:189 +0x4f4
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes(0x4002078000, {0x70575e8?, 0x4009b4e960?}, 0x400a3ae6c0)
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193 +0x5c0
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes(0x40008e3260?, {0x70575e8, 0x4009b4e960}, 0x400a3ae6c0)
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:202 +0xac
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0x400a858000, {0x70575e8?, 0x4009b4e930?}, {0x400a852360?}, 0x400a3ae6c0)
  | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1328 +0x388
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0x400a858000, {0x70575e8, 0x4009b4e8d0}, 0x400a3ae6c0)
  | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1462 +0x224
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1({0x70575e8?, 0x4009b4e8d0?}, {0x58f63a0?, 0x400a3ae6c0?})
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:704 +0x54
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12({0x70575e8, 0x4009b4e8d0}, {0x58f63a0, 0x400a3ae6c0}, 0x400369b420, 0x40036a1ad0)
  | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97 +0x32c
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0x70575e8?, 0x4009b4e8d0?}, {0x58f63a0?, 0x400a3ae6c0?})
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x40
  | github.com/co...
Stacktrace (expand for inline code snippets):

if withParallelCommit {
rpl, pErr = ds.divideAndSendParallelCommit(ctx, ba, rs, isReverse, 0 /* batchIdx */)
} else {

// isReverse flag.
br, pErr = ds.divideAndSendBatchToRanges(ctx, ba, rs, isReverse, true /* withCommit */, batchIdx)

if !ri.NeedAnother(rs) {
resp := ds.sendPartialBatch(
ctx, ba, rs, isReverse, withCommit, batchIdx, ri.Token(),

prevTok = routingTok
reply, err = ds.sendToReplicas(ctx, ba, routingTok, withCommit)

br, err = transport.SendNext(ctx, ba)
ds.metrics.updateCrossLocalityMetricsOnReplicaAddressedBatchResponse(comparisonResult, int64(br.Size()))

}
return gt.sendBatch(ctx, r.NodeID, iface, ba)
}

log.VEvent(ctx, 2, "sending batch request")
reply, err := iface.Batch(ctx, ba)
log.VEvent(ctx, 2, "received batch response")

https://github.com/cockroachdb/cockroach/blob/57ab10786d6ed277959c8990e2823772893ac873/pkg/rpc/pkg/rpc/context.go#L906-L908
https://github.com/cockroachdb/cockroach/blob/57ab10786d6ed277959c8990e2823772893ac873/pkg/rpc/pkg/rpc/context.go#L783-L785
https://github.com/cockroachdb/cockroach/blob/57ab10786d6ed277959c8990e2823772893ac873/pkg/rpc/pkg/rpc/context.go#L898-L900
if localRequest {
return invoker(ctx, method, req, resp, cc, opts...)
}

https://github.com/cockroachdb/cockroach/blob/57ab10786d6ed277959c8990e2823772893ac873/pkg/rpc/pkg/rpc/context.go#L713-L715
https://github.com/cockroachdb/cockroach/blob/57ab10786d6ed277959c8990e2823772893ac873/pkg/rpc/pkg/rpc/context.go#L814-L816
https://github.com/cockroachdb/cockroach/blob/57ab10786d6ed277959c8990e2823772893ac873/pkg/rpc/pkg/rpc/context.go#L133-L135
return f(ctx)
}

https://github.com/cockroachdb/cockroach/blob/57ab10786d6ed277959c8990e2823772893ac873/pkg/rpc/pkg/rpc/context.go#L135-L137
https://github.com/cockroachdb/cockroach/blob/57ab10786d6ed277959c8990e2823772893ac873/pkg/rpc/pkg/rpc/context.go#L814-L816
https://github.com/cockroachdb/cockroach/blob/57ab10786d6ed277959c8990e2823772893ac873/pkg/rpc/pkg/rpc/context.go#L168-L170
https://github.com/cockroachdb/cockroach/blob/57ab10786d6ed277959c8990e2823772893ac873/pkg/rpc/pkg/rpc/context.go#L814-L816
if methodExcludedFromTracing(info.FullMethod) {
return handler(ctx, req)
}

https://github.com/cockroachdb/cockroach/blob/57ab10786d6ed277959c8990e2823772893ac873/pkg/rpc/pkg/rpc/context.go#L703-L705

cockroach/pkg/server/node.go

Lines 1461 to 1463 in 57ab107

br, err := n.batchInternal(ctx, tenantID, args)

cockroach/pkg/server/node.go

Lines 1327 to 1329 in 57ab107

var pErr *kvpb.Error
br, writeBytes, pErr = n.stores.SendWithWriteBytes(ctx, args)
if pErr != nil {

https://github.com/cockroachdb/cockroach/blob/57ab10786d6ed277959c8990e2823772893ac873/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L201-L203
https://github.com/cockroachdb/cockroach/blob/57ab10786d6ed277959c8990e2823772893ac873/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L192-L194
https://github.com/cockroachdb/cockroach/blob/57ab10786d6ed277959c8990e2823772893ac873/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L188-L190
https://github.com/cockroachdb/cockroach/blob/57ab10786d6ed277959c8990e2823772893ac873/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L467-L469
g.EvalKind = evalKind
resp, err := m.sequenceReqWithGuard(ctx, g, branch)
if resp != nil || err != nil {

if g.EvalKind != PessimisticAfterFailedOptimisticEval {
panic(redact.Safe(fmt.Sprintf("must not be holding latches\n"+
"this is tracked in github.com/cockroachdb/cockroach/issues/77663; please comment if seen\n"+

GOROOT/src/runtime/panic.go#L919-L921
https://github.com/cockroachdb/cockroach/blob/57ab10786d6ed277959c8990e2823772893ac873/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L108-L110
GOROOT/src/runtime/panic.go#L913-L915

pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).Send at line 1102
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendParallelCommit at line 1287
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges at line 1488
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch at line 1920
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas at line 2423
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext at line 189
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch at line 211
pkg/rpc/pkg/rpc/context.go in pkg/rpc.internalClientAdapter.Batch at line 907
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func3 at line 784
pkg/rpc/pkg/rpc/context.go in pkg/rpc.getChainUnaryInvoker.func1 at line 899
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewContext.ClientInterceptor.func8 at line 228
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func2 at line 714
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 815
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1 at line 134
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 336
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1.1 at line 136
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 815
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func3 at line 169
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 815
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewServerEx.ServerInterceptor.func12 at line 97
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func1 at line 704
pkg/server/node.go in pkg/server.(*Node).Batch at line 1462
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 1328
pkg/kv/kvserver/pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).SendWithWriteBytes at line 202
pkg/kv/kvserver/pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 193
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 189
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 468
pkg/kv/kvserver/concurrency/concurrency_manager.go in pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq at line 241
pkg/kv/kvserver/concurrency/concurrency_manager.go in pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard at line 304
GOROOT/src/runtime/panic.go in runtime.gopanic at line 920
pkg/kv/kvserver/pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes.func1 at line 109
GOROOT/src/runtime/panic.go in runtime.gopanic at line 914

Tags

Tag Value
Command start-single-node
Environment v23.2.5
Go Version go1.21.9 X:nocoverageredesign
Platform linux arm64
Distribution CCL
Cockroach Release v23.2.5
Cockroach SHA 57ab107
# of CPUs 11
# of Goroutines 525

Jira issue: CRDB-44239

@cockroach-sentry cockroach-sentry added branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. labels Nov 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.
Projects
None yet
Development

No branches or pull requests

1 participant