Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: multitenant/distsql/instances=20/bundle=on/timeout=0 failed #114295

Closed
cockroach-teamcity opened this issue Nov 11, 2023 · 5 comments
Closed

roachtest: multitenant/distsql/instances=20/bundle=on/timeout=0 failed #114295

cockroach-teamcity opened this issue Nov 11, 2023 · 5 comments
Assignees
@michae2
Labels
branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 C-test-failure Broken test (automatically or manually discovered). GA-blocker O-roachtest O-robot Originated from a bot. T-sql-queries SQL Queries Team
Milestone
23.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Nov 11, 2023 •
edited by cockroach-jira-scripts
Loading

roachtest.multitenant/distsql/instances=20/bundle=on/timeout=0 failed with artifacts on release-23.2 @ 495a09bec6289a4a15f92e7edc7e793972c79722:

(assertions.go:333).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:225
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
	            				main/pkg/cmd/roachtest/test_runner.go:1090
	            				src/runtime/asm_amd64.s:1598
	Error:      	Received unexpected error:
	            	pq: internal error: unexpected error from the vectorized engine: runtime error: index out of range [36] with length 36
	Test:       	multitenant/distsql/instances=20/bundle=on/timeout=0
(require.go:1360).NoError: FailNow called
(cluster.go:2239).Run: context canceled
(cluster.go:2239).Run: context canceled
(cluster.go:2239).Run: context canceled
(cluster.go:2239).Run: context canceled
(cluster.go:2239).Run: context canceled
(cluster.go:2239).Run: context canceled
(cluster.go:2239).Run: context canceled
(cluster.go:2239).Run: context canceled
(cluster.go:2239).Run: context canceled
(cluster.go:2239).Run: context canceled
(cluster.go:2239).Run: context canceled
(cluster.go:2239).Run: context canceled
(cluster.go:2239).Run: context canceled
(cluster.go:2239).Run: context canceled
(cluster.go:2239).Run: context canceled
(cluster.go:2239).Run: context canceled
(cluster.go:2239).Run: context canceled
(cluster.go:2239).Run: context canceled
(cluster.go:2239).Run: context canceled
(cluster.go:2239).Run: context canceled
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=0/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_metamorphicBuild=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/sql-queries

This test on roachdash | Improve this report!

Jira issue: CRDB-33438
@cockroach-teamcity cockroach-teamcity added branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-queries SQL Queries Team labels Nov 11, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.2 milestone Nov 11, 2023
@github-project-automation github-project-automation bot moved this to Triage in SQL Queries Nov 11, 2023
@DrewKimball
Copy link
Collaborator 

E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277  encountered internal error:
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +unexpected error from the vectorized engine: runtime error: index out of range [36] with length 36
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +(1) ‹Statement diagnostics bundle generated. Download using the SQL shell or command›
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +‹  | line.›
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +‹  | SQL shell: \statement-diag download 916244795059929089›
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +‹  | Command line: cockroach statement-diag download 916244795059929089›
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +Wraps: (2) attached stack trace
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  -- stack trace:
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Inbox).Next.func1
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/inbox.go:324
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | runtime.gopanic
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	GOROOT/src/runtime/panic.go:884
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.ExpectedError
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:203
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Inbox).Next
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/inbox.go:385
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*invariantsChecker).Next
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/invariants_checker.go:93
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:118
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:126
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*ParallelUnorderedSynchronizer).Init.func1.1
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/parallel_unordered_synchronizer.go:198
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*ParallelUnorderedSynchronizer).init.func1
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/parallel_unordered_synchronizer.go:259
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | runtime.goexit
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	src/runtime/asm_amd64.s:1598
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +Wraps: (3)
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +Wraps: (4) tags: [nsql1,client=34.86.44.98:45858,hostssl,user=root,f‹c9a92cff›,distsql.stmt=‹EXPLAIN ANALYZE (DEBUG) SELECT DISTINCT i FROM t›,distsql.gateway=1,distsql.txn=‹edda394b-bc19-4e06-9aed-160dff3f1da0›,streamID=‹433›,received-error]
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +Wraps: (5) tags: [n‹sql8›,f‹c9a92cff›,distsql.stmt=‹EXPLAIN ANALYZE (DEBUG) SELECT DISTINCT i FROM t›,distsql.gateway=‹1›,distsql.txn=‹edda394b-bc19-4e06-9aed-160dff3f1da0›,streamID=‹433›,sent-error=]
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +Wraps: (6) assertion failure
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +Wraps: (7)
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | (opaque error wrapper)
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | reportable 0:
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  |
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:89
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | runtime.gopanic
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	GOROOT/src/runtime/panic.go:884
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | runtime.goPanicIndex
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	GOROOT/src/runtime/panic.go:113
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexechash.(*HashTable).updateSel
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/sql/colexec/colexechash/hashtable_distinct.eg.go:5528
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexechash.(*HashTable).RemoveDuplicates
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/colexechash/hashtable.go:698
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexechash.(*HashTable).DistinctBuild
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/colexechash/hashtable.go:632
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*UnorderedDistinct).Next
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/unordered_distinct.go:110
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecdisk.(*diskSpillerBase).Next.func1
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecdisk/disk_spiller.go:202
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecdisk.(*diskSpillerBase).Next
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecdisk/disk_spiller.go:200
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*invariantsChecker).Next
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/invariants_checker.go:93
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:118
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:126
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils.(*deselectorOp).Next
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils/deselector.go:53
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Outbox).sendBatches.func1
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/outbox.go:285
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Outbox).sendBatches
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/outbox.go:276
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Outbox).runWithStream
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/outbox.go:414
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Outbox).Run
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/outbox.go:237
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlowCreator).setupRemoteOutputStream.func1
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:772
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlowCreator).accumulateAsyncComponent.func1.1
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:734
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | runtime.goexit
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | 	src/runtime/asm_amd64.s:1598
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +Wraps: (8) unexpected error from the vectorized engine
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +Wraps: (9) runtime error: index out of range [36] with length 36
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | -- cause hidden behind barrier
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | ‹runtime error: index out of range [36] with length 36›
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | (1) ‹runtime error: index out of range [36] with length 36›
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  |   |
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  |   | (opaque error leaf)
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  |   | type name: runtime/runtime.boundsError
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +  | Error types: (1) *errbase.opaqueLeaf
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +Error types: (1) *hintdetail.withDetail (2) *withstack.withStack (3) *colexecerror.notInternalError (4) *contexttags.withContext (5) *contexttags.withContext (6) *assert.withAssertionFailure (7) *errbase.opaqueWrapper (8) *errutil.withPrefix (9) *barriers.barrierErr
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +HINT: ‹You have encountered an unexpected error.›
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +‹Please check the public issue tracker to check whether this problem is›
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +‹already tracked. If you cannot find it there, please report the error›
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +‹with details by creating a new issue.›
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +‹If you would rather not post publicly, please contact us directly›
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +‹using the support form.›
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +
E231111 07:02:01.199445 4602 sql/sqltelemetry/report.go:57 ⋮ [T11,Vcluster-11,nsql1,client=34.86.44.98:45858,hostssl,user=root] 277 +‹We appreciate your feedback.›

@mgartner mgartner added GA-blocker and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Nov 13, 2023
@mgartner
Copy link
Collaborator

Marking a GA-blocker instead of release-blocker until we know more. This shouldn't hold up the beta though because it's unlikely to be a regression.

@mgartner mgartner moved this from Triage to Active in SQL Queries Nov 14, 2023
@michae2
Copy link
Collaborator

michae2 commented Nov 14, 2023

[triage] @yuzefovich and @DrewKimball suggest this might have to do with different batch sizes between tenant and system

@yuzefovich
Copy link
Member

(More precisely: different batch sizes between SQL pods (or "instances") of the same tenant.)

@renatolabs
Copy link
Contributor

Test fails when using metamorphic cockroach builds. These runs were disabled (#114618) until we have a stable release. We'll re-assess test failures when this feature is enabled again.

In this particular instance, I believe this discrepancy happened because multitenant roachtests use their own mechanisms to start and stop pods outside of roachprod (multitenant_utils.go), and that was not updated to work well with metamorphic builds. We are currently working to unify this so that all multitenant tests are able to use the standard c.Start, c.Stop APIs in which case all pods should have the same view of metamorphic constants.

@github-project-automation github-project-automation bot moved this from Active to Done in SQL Queries Nov 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@michae2 michae2

Labels
branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 C-test-failure Broken test (automatically or manually discovered). GA-blocker O-roachtest O-robot Originated from a bot. T-sql-queries SQL Queries Team
Projects
SQL Queries
Archived in project
Milestone
23.2
Development

No branches or pull requests
6 participants
@mgartner @yuzefovich @cockroach-teamcity @michae2 @DrewKimball @renatolabs