Sentry: error.go:78: unexpected error from the vectorized engine: node unavailable; try another peer (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/sql/colflow/col...

[cockroach-sentry]

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/5986876064/?referrer=webhooks_plugin

Panic Message:

error.go:78: unexpected error from the vectorized engine: node unavailable; try another peer
(1) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Inbox).Next.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/inbox.go:324
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:770
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.ExpectedError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:299
  | github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Inbox).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/inbox.go:385
  | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecjoin.(*mergeJoinInnerOp).Next
  | 	github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/sql/colexec/colexecjoin/mergejoiner_inner.eg.go:10929
  | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase/simple_project.go:124
  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*ParallelUnorderedSynchronizer).Init.(*ParallelUnorderedSynchronizer).Init.func1.func2
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/parallel_unordered_synchronizer.go:200
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:152
  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*ParallelUnorderedSynchronizer).init.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/parallel_unordered_synchronizer.go:261
  | runtime.goexit
  | 	src/runtime/asm_amd64.s:1695
Wraps: (2)
Wraps: (3) tags: [n74,client=10.245.1.44:55184,hostssl,user=×,f×,distsql.stmt=×,distsql.gateway=74,distsql.appname=×,distsql.txn=×,streamID=×,received-error]
Wraps: (4) tags: [n×,f×,distsql.stmt=×,distsql.gateway=×,distsql.appname=×,distsql.txn=×,streamID=×,sent-error=]
Wraps: (5)
Wraps: (6) assertion failure
Wraps: (7)
  | (opaque error wrapper)
  | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
  | reportable 0:
  |
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:78
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:770
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.InternalError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:291
  | github.com/cockroachdb/cockroach/pkg/sql/colfetcher.(*ColIndexJoin).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colfetcher/index_join.go:226
  | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils.(*CancelChecker).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils/cancel_checker.go:59
  | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecsel.(*selEQInt16Int64ConstOp).Next
  | 	github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/sql/colexec/colexecsel/selection_ops.eg.go:2559
  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*isNullSelOp).Next
  | 	github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/sql/colexec/is_null_ops.eg.go:218
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*HashRouter).processNextBatch
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/routers.go:667
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*HashRouter).Run.func1.1
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/routers.go:591
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:152
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*HashRouter).Run.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/routers.go:629
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:152
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*HashRouter).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/routers.go:586
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlowCreator).setupRouter.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:825
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlowCreator).setupRouter.(*vectorizedFlowCreator).accumulateAsyncComponent.func2.1
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:728
  | runtime.goexit
  | 	src/runtime/asm_amd64.s:1695
Wraps: (8) unexpected error from the vectorized engine
Wraps: (9) node unavailable; try another peer
  | -- cause hidden behind barrier
  | node unavailable; try another peer
  | (1) node unavailable; try another peer
  | Error types: (1) *kvpb.NodeUnavailableError
Error types: (1) *withstack.withStack (2) *colexecerror.notInternalError (3) *contexttags.withContext (4) *contexttags.withContext (5) *colexecerror.notInternalError (6) *assert.withAssertionFailure (7) *errbase.opaqueWrapper (8) *errutil.withPrefix (9) *barriers.barrierErr
-- report composition:
*barriers.barrierErr: masked error: node unavailable; try another peer
*errutil.withPrefix: unexpected error from the vectorized engine
error.go:78: *withstack.withStack (top exception)
*assert.withAssertionFailure
*colexecerror.notInternalError
*contexttags.withContext: n75
*contexttags.withContext: n74
*colexecerror.notInternalError
inbox.go:324: *withstack.withStack (1)
(check the extra data payloads)

Stacktrace (expand for inline code snippets):

src/runtime/asm_amd64.s#L1694-L1696

cockroach/pkg/sql/colexec/parallel_unordered_synchronizer.go

Lines 260 to 262 in 8fb18e7

	case parallelUnorderedSynchronizerStateRunning:
	if err := colexecerror.CatchVectorizedRuntimeError(s.nextBatch[inputIdx]); err != nil {
	if s.getState() == parallelUnorderedSynchronizerStateDraining && s.Ctx.Err() == nil && s.cancelLocalInput[inputIdx] != nil {

cockroach/pkg/sql/colexecerror/error.go

Lines 151 to 153 in 8fb18e7

	}()
	operation()
	return retErr

cockroach/pkg/sql/colexec/parallel_unordered_synchronizer.go

Lines 199 to 201 in 8fb18e7

	return func() {
	s.batches[inputIdx] = inputOp.Next()
	}

cockroach/pkg/sql/colexec/colexecbase/simple_project.go

Lines 123 to 125 in 8fb18e7

	func (d *simpleProjectOp) Next() coldata.Batch {
	batch := d.Input.Next()
	if batch.Length() == 0 {

https://github.com/cockroachdb/cockroach/blob/8fb18e7ae08fdd481b959f74330e273a2500e1df/bazel-out/k8-opt/bin/pkg/sql/colexec/colexecjoin/mergejoiner_inner.eg.go#L10928-L10930

cockroach/pkg/sql/colflow/colrpc/inbox.go

Lines 384 to 386 in 8fb18e7

	if receivedErr != nil {
	colexecerror.ExpectedError(receivedErr)
	}

cockroach/pkg/sql/colexecerror/error.go

Lines 298 to 300 in 8fb18e7

	func ExpectedError(err error) {
	panic(newNotInternalError(err))
	}

GOROOT/src/runtime/panic.go#L769-L771

cockroach/pkg/sql/colflow/colrpc/inbox.go

Lines 323 to 325 in 8fb18e7

	}
	err := logcrash.PanicAsError(0, panicObj)
	log.VEventf(i.Ctx, 1, "Inbox encountered an error in Next: %v", err)

src/runtime/asm_amd64.s#L1694-L1696

cockroach/pkg/sql/colflow/vectorized_flow.go

Lines 727 to 729 in 8fb18e7

	defer wg.Done()
	run(ctx, flowCtxCancel)
	}()

cockroach/pkg/sql/colflow/vectorized_flow.go

Lines 824 to 826 in 8fb18e7

	runRouter := func(ctx context.Context, _ context.CancelFunc) {
	router.Run(logtags.AddTag(ctx, "hashRouterID", streamIDs))
	}

cockroach/pkg/sql/colflow/routers.go

Lines 585 to 587 in 8fb18e7

	// well for more fine-grained control of error propagation.
	if err := colexecerror.CatchVectorizedRuntimeError(func() {
	r.Input.Init(ctx)

cockroach/pkg/sql/colexecerror/error.go

Lines 151 to 153 in 8fb18e7

	}()
	operation()
	return retErr

cockroach/pkg/sql/colflow/routers.go

Lines 628 to 630 in 8fb18e7

	if err := colexecerror.CatchVectorizedRuntimeError(processNextBatch); err != nil {
	r.cancelOutputs(ctx, err)

cockroach/pkg/sql/colexecerror/error.go

Lines 151 to 153 in 8fb18e7

	}()
	operation()
	return retErr

cockroach/pkg/sql/colflow/routers.go

Lines 590 to 592 in 8fb18e7

	processNextBatch := func() {
	done = r.processNextBatch(ctx)
	}

cockroach/pkg/sql/colflow/routers.go

Lines 666 to 668 in 8fb18e7

	func (r *HashRouter) processNextBatch(ctx context.Context) bool {
	b := r.Input.Next()
	n := b.Length()

https://github.com/cockroachdb/cockroach/blob/8fb18e7ae08fdd481b959f74330e273a2500e1df/bazel-out/k8-opt/bin/pkg/sql/colexec/is_null_ops.eg.go#L217-L219
https://github.com/cockroachdb/cockroach/blob/8fb18e7ae08fdd481b959f74330e273a2500e1df/bazel-out/k8-opt/bin/pkg/sql/colexec/colexecsel/selection_ops.eg.go#L2558-L2560

cockroach/pkg/sql/colexec/colexecutils/cancel_checker.go

Lines 58 to 60 in 8fb18e7

	c.CheckEveryCall()
	return c.Input.Next()
	}

cockroach/pkg/sql/colfetcher/index_join.go

Lines 225 to 227 in 8fb18e7

	); err != nil {
	colexecerror.InternalError(err)
	}

cockroach/pkg/sql/colexecerror/error.go

Lines 290 to 292 in 8fb18e7

	func InternalError(err error) {
	panic(newInternalError(err))
	}

GOROOT/src/runtime/panic.go#L769-L771

cockroach/pkg/sql/colexecerror/error.go

Lines 77 to 79 in 8fb18e7

	if code := pgerror.GetPGCode(retErr); code == pgcode.Uncategorized {
	retErr = errors.NewAssertionErrorWithWrappedErrf(
	retErr, "unexpected error from the vectorized engine",

src/runtime/asm_amd64.s in runtime.goexit at line 1695
pkg/sql/colexec/parallel_unordered_synchronizer.go in pkg/sql/colexec.(*ParallelUnorderedSynchronizer).init.func1 at line 261
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 152
pkg/sql/colexec/parallel_unordered_synchronizer.go in pkg/sql/colexec.(*ParallelUnorderedSynchronizer).Init.(*ParallelUnorderedSynchronizer).Init.func1.func2 at line 200
pkg/sql/colexec/colexecbase/simple_project.go in pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next at line 124
bazel-out/k8-opt/bin/pkg/sql/colexec/colexecjoin/mergejoiner_inner.eg.go in pkg/sql/colexec/colexecjoin.(*mergeJoinInnerOp).Next at line 10929
pkg/sql/colflow/colrpc/inbox.go in pkg/sql/colflow/colrpc.(*Inbox).Next at line 385
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.ExpectedError at line 299
GOROOT/src/runtime/panic.go in runtime.gopanic at line 770
pkg/sql/colflow/colrpc/inbox.go in pkg/sql/colflow/colrpc.(*Inbox).Next.func1 at line 324
src/runtime/asm_amd64.s in runtime.goexit at line 1695
pkg/sql/colflow/vectorized_flow.go in pkg/sql/colflow.(*vectorizedFlowCreator).setupRouter.(*vectorizedFlowCreator).accumulateAsyncComponent.func2.1 at line 728
pkg/sql/colflow/vectorized_flow.go in pkg/sql/colflow.(*vectorizedFlowCreator).setupRouter.func1 at line 825
pkg/sql/colflow/routers.go in pkg/sql/colflow.(*HashRouter).Run at line 586
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 152
pkg/sql/colflow/routers.go in pkg/sql/colflow.(*HashRouter).Run.func1 at line 629
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 152
pkg/sql/colflow/routers.go in pkg/sql/colflow.(*HashRouter).Run.func1.1 at line 591
pkg/sql/colflow/routers.go in pkg/sql/colflow.(*HashRouter).processNextBatch at line 667
bazel-out/k8-opt/bin/pkg/sql/colexec/is_null_ops.eg.go in pkg/sql/colexec.(*isNullSelOp).Next at line 218
bazel-out/k8-opt/bin/pkg/sql/colexec/colexecsel/selection_ops.eg.go in pkg/sql/colexec/colexecsel.(*selEQInt16Int64ConstOp).Next at line 2559
pkg/sql/colexec/colexecutils/cancel_checker.go in pkg/sql/colexec/colexecutils.(*CancelChecker).Next at line 59
pkg/sql/colfetcher/index_join.go in pkg/sql/colfetcher.(*ColIndexJoin).Next at line 226
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.InternalError at line 291
GOROOT/src/runtime/panic.go in runtime.gopanic at line 770
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1 at line 78

Tags

Tag	Value
Command	server
Environment	v24.2.2
Go Version	go1.22.5 X:nocoverageredesign
Platform	linux amd64
Distribution	CCL
Cockroach Release	v24.2.2
Cockroach SHA	8fb18e7
# of CPUs	4
# of Goroutines	1631

Jira issue: CRDB-43127

[yuzefovich]

dup of #128649