roachtest: schemachange/mixed-versions failed [already contains column]

[cockroach-teamcity]

roachtest.schemachange/mixed-versions failed with artifacts on release-25.2 @ c616af291289bef46024b55af834079028b955d1:

(mixedversion.go:829).Run: mixed-version test failure while running step 16 (run "run schemachange workload and validation in mixed version"): full command output in run_065301.438370653_n4_COCKROACHRANDOMSEED3.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=4
• encrypted=false
• metamorphicBufferedSender=false
• mvtDeploymentMode=shared-process
• mvtVersions=v25.1.6 → release-25.2
• runtimeAssertionsBuild=false
• ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-50954

[Dedej-Bergin]

Seems to be similar to #146567.

  ],
  "previousStatements": [
   "ALTER TABLE schema_w4_45.table_w2_298 ALTER PRIMARY KEY USING COLUMNS (col211_w1_212)"
  ]
 }
}
Schema Workload Stats
Total Schema Statements Executed = 25
Total Schema Statements Succeeded = 25
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 13
Total Transactions Rolled Back = 24
Total Transactions Executed = 37
E250525 06:53:04.594906 1 workload/cli/run.go:583  [-] 3  workload run error: ***UNEXPECTED COMMIT ERROR; Received an unexpected commit error: ERROR: transaction committed but schema change aborted with error: (42701): error executing PostCommitPhase stage 8 of 15 with 26 MutationType ops: relation "table_w2_298" (244): index "table_w1_211_expr_idx" already contains column "col211_w1_212" (SQLSTATE 42701)
E250525 06:53:04.594906 1 workload/cli/run.go:583  [-] 3 +(1) forced error mark
E250525 06:53:04.594906 1 workload/cli/run.go:583  [-] 3 +  | "fatal error when running txn"
E250525 06:53:04.594906 1 workload/cli/run.go:583  [-] 3 +  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
E250525 06:53:04.594906 1 workload/cli/run.go:583  [-] 3 +Wraps: (2)
E250525 06:53:04.594906 1 workload/cli/run.go:583  [-] 3 +Wraps: (3) attached stack trace
E250525 06:53:04.594906 1 workload/cli/run.go:583  [-] 3 +  -- stack trace:
E250525 06:53:04.594906 1 workload/cli/run.go:583  [-] 3 +  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
E250525 06:53:04.594906 1 workload/cli/run.go:583  [-] 3 +  | 	pkg/workload/schemachange/schemachange.go:681
E250525 06:53:04.594906 1 workload/cli/run.go:583  [-] 3 +  | github.com/cockroachdb/cockroach/pkg/workload/cli.workerRun
E250525 06:53:04.594906 1 workload/cli/run.go:583  [-] 3 +  | 	pkg/workload/cli/run.go:290
E250525 06:53:04.594906 1 workload/cli/run.go:583  [-] 3 +  | github.com/cockroachdb/cockroach/pkg/workload/cli.runRun.func5.1
E250525 06:53:04.594906 1 workload/cli/run.go:583  [-] 3 +  | 	pkg/workload/cli/run.go:542
E250525 06:53:04.594906 1 workload/cli/run.go:583  [-] 3 +  | runtime.goexit
E250525 06:53:04.594906 1 workload/cli/run.go:583  [-] 3 +  | 	src/runtime/asm_amd64.s:1700
E250525 06:53:04.594906 1 workload/cli/run.go:583  [-] 3 +Wraps: (4) ***UNEXPECTED COMMIT ERROR; Received an unexpected commit error
E250525 06:53:04.594906 1 workload/cli/run.go:583  [-] 3 +Wraps: (5) ERROR: transaction committed but schema change aborted with error: (42701): error executing PostCommitPhase stage 8 of 15 with 26 MutationType ops: relation "table_w2_298" (244): index "table_w1_211_expr_idx" already contains column "col211_w1_212" (SQLSTATE 42701)
E250525 06:53:04.594906 1 workload/cli/run.go:583  [-] 3 +Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *pgconn.PgError
Error: ***UNEXPECTED COMMIT ERROR; Received an unexpected commit error: ERROR: transaction committed but schema change aborted with error: (42701): error executing PostCommitPhase stage 8 of 15 with 26 MutationType ops: relation "table_w2_298" (244): index "table_w1_211_expr_idx" already contains column "col211_w1_212" (SQLSTATE 42701)
run_065301.438370653_n4_COCKROACHRANDOMSEED3: 2025/05/25 06:53:04 cluster.go:2545: > result: COMMAND_PROBLEM: exit status 1

[cockroach-teamcity]

roachtest.schemachange/mixed-versions failed with artifacts on release-25.2 @ a2b8b65ed24f9bc9dffcba0dcc1a1d000888c223:

(mixedversion.go:829).Run: mixed-version test failure while running step 8 (run "run schemachange workload and validation in mixed version"): full command output in run_123449.498489986_n4_COCKROACHRANDOMSEED5.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/cpu_arch=arm64/run_1

Parameters:

• arch=arm64
• cloud=gce
• coverageBuild=false
• cpu=4
• encrypted=false
• fs=ext4
• localSSD=false
• metamorphicBufferedSender=true
• mvtDeploymentMode=system-only
• mvtVersions=v25.1.6 → release-25.2
• runtimeAssertionsBuild=false
• ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

[cockroach-teamcity]

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.schemachange/mixed-versions failed with artifacts on release-25.2 @ a25a36211e8d14ac4b26d22d586243042af79bfb:

(mixedversion.go:829).Run: mixed-version test failure while running step 14 (run "run schemachange workload and validation in mixed version"): full command output in run_064351.173843741_n4_COCKROACHRANDOMSEED7.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=4
• encrypted=false
• metamorphicBufferedSender=false
• mvtDeploymentMode=separate-process
• mvtVersions=v25.1.6 → release-25.2
• runtimeAssertionsBuild=true
• ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

[fqazi]

This only seems to occur in a mixed version state:

CREATE SCHEMA schema_w4_45;
CREATE TABLE schema_w4_45.table_w2_298 (col211_w1_212 STRING NOT NULL, col211_w1_213 INT8, col211_w1_214 FLOAT8 NOT NULL, col211_w1_215 VECTOR NOT NULL, col211_w1_216 FLOAT8 NOT NULL, col211_w1_217 GEOGRAPHY NOT NULL, col211_w1_218 STRING NOT NULL, col211_w1_219 INT8 AS (abs(col211_w1_213)) STORED, col211_w1_220 FLOAT8 NOT NULL AS (abs(col211_w1_216)) VIRTUAL, col211_w1_221 FLOAT8 NOT NULL AS (col211_w1_216 + col211_w1_214) VIRTUAL, col211_w1_222 STRING NOT NULL AS (lower(col211_w1_218)) VIRTUAL, col211_w1_223 FLOAT8 NOT NULL AS (abs(col211_w1_216)) VIRTUAL, INDEX (lower(CAST(col211_w1_215 AS STRING)) DESC) STORING (col211_w1_212, col211_w1_219), FAMILY (col211_w1_217, col211_w1_212, col211_w1_219, col211_w1_215, col211_w1_218), FAMILY (col211_w1_213), FAMILY (col211_w1_216, col211_w1_214));
ALTER TABLE schema_w4_45.table_w2_298 ALTER PRIMARY KEY USING COLUMNS (col211_w1_212);

I'll see if I can adjust the test to tolerate these errors better

[fqazi]

The actual PR that fixed this was: #141850, which isn't safe to backport