pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed

[cockroach-teamcity]

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed on release-24.3 @ 60b41ceffef285d73a0dfe9b93264e33a2dc400a:

        	            	Wraps: (5) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).randOp
        	            	  | 	pkg/workload/schemachange/operation_generator.go:219
        	            	  | [...repeated from below...]
        	            	Wraps: (6) failed generating operation: alterTableSetColumnNotNull
        	            	Wraps: (7) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.CollectOne[...]
        	            	  | 	pkg/workload/schemachange/query_util.go:176
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.Scan[...]
        	            	  | 	pkg/workload/schemachange/query_util.go:152
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).scanBool
        	            	  | 	pkg/workload/schemachange/error_screening.go:84
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).columnContainsNull
        	            	  | 	pkg/workload/schemachange/error_screening.go:756
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).setColumnNotNull
        	            	  | 	pkg/workload/schemachange/operation_generator.go:2503
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).randOp
        	            	  | 	pkg/workload/schemachange/operation_generator.go:206
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
        	            	  | 	pkg/workload/schemachange/schemachange.go:489
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
        	            	  | 	pkg/workload/schemachange/schemachange.go:599
        	            	  | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func5.func7
        	            	  | 	pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:153
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (8) CollectOne: CollectOneRow: Query: SELECT EXISTS (
        	            	  | 		SELECT col34_w1_49
        	            	  | 		  FROM public.table_w1_34
        	            	  | 	   WHERE col34_w1_49 IS NULL
        	            	  | 	) Args: []
        	            	Wraps: (9) ERROR: internal error: requested column col34_w1_40 not in index (SQLSTATE XX000)
        	            	Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *withstack.withStack (8) *errutil.withPrefix (9) *pgconn.PgError
        	Test:       	TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 43
Total Schema Statements Succeeded = 43
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 28
Total Transactions Rolled Back = 23
Total Transactions Executed = 51
    schema_change_external_test.go:117: backup, tracing data, and system table dumps in /mnt/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/testlogs/pkg/ccl/testccl/workload/schemachange/schemachange_test/run_4_of_10/test.outputs/logTestWorkload2681773018
    schema_change_external_test.go:126: running DROP with use_declarative_schema_changer = on
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestWorkload2681773018
--- FAIL: TestWorkload (10.55s)

Parameters:

• attempt=1
• run=4
• shard=1

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

• pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed #160318 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [C-test-failure O-robot T-sql-foundations branch-release-26.1 release-blocker]

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-58347

[spilchen]

We have seen this before (see #153226). Basically, if there is a concurrent ALTER PK then this missing column error can show up:

internal error: requested column col34_w1_40 not in index (SQLSTATE XX000)

There were a few fixes for ALTER PK that aren't in 24.3. The most likely one is #141850. Closing this as it doesn't look like we will be backporting that issue since it has some DSC opgen changes that can't easily be backported.