pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed #112390

cockroach-teamcity · 2023-10-15T08:54:26Z

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed with artifacts on release-23.2 @ 87a94f6ac59d4e01c822bb2128db858176600dd6:

1, 29, rangelog, 13
1, 29, region_liveness, 9
1, 29, replication_constraint_stats, 25
1, 29, replication_critical_localities, 26
1, 29, replication_stats, 27
1, 29, reports_meta, 28
1, 29, role_id_seq, 48
1, 29, role_members, 23
1, 29, role_options, 33
1, 29, scheduled_jobs, 37
1, 29, settings, 6
1, 29, span_configurations, 47
1, 29, span_stats_buckets, 55
1, 29, span_stats_samples, 56
1, 29, span_stats_tenant_boundaries, 57
1, 29, span_stats_unique_keys, 54
1, 29, sql_instances, 46
1, 29, sqlliveness, 39
1, 29, statement_activity, 60
1, 29, statement_bundle_chunks, 34
1, 29, statement_diagnostics, 36
1, 29, statement_diagnostics_requests, 35
1, 29, statement_execution_insights, 65
1, 29, statement_statistics, 42
1, 29, table_statistics, 20
1, 29, task_payloads, 58
1, 29, tenant_id_seq, 62
1, 29, tenant_settings, 50
1, 29, tenant_tasks, 59
1, 29, tenant_usage, 45
1, 29, tenants, 8
1, 29, transaction_activity, 61
1, 29, transaction_execution_insights, 64
1, 29, transaction_statistics, 43
1, 29, ui, 14
1, 29, users, 4
1, 29, web_sessions, 19
1, 29, zones, 5
100, 0, public, 101
102, 0, public, 103
104, 0, public, 105
104, 0, schema93, 123
104, 105, _enum37, 116
104, 105, _enum67, 120
104, 105, enum37, 115
104, 105, enum67, 119
    schema_change_external_test.go:82: backup in /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload469890276
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload469890276
--- FAIL: TestWorkload (51.70s)

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [error dropping enum type] #111540 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [error dropping enum type] [C-test-failure O-robot O-schema-testing T-sql-foundations branch-master db-cy-23]

/cc @cockroachdb/sql-foundations _{This test on roachdash | Improve this report!

Jira issue: CRDB-32392}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2023-10-17T10:04:17Z

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed with artifacts on release-23.2 @ dda421d56ace4a4c04604d0b99c3227f3948f45e:

Fatal error:

panic: connection has timed out &{0xc0094a0fa0 0xc012ec9600 178edca5342ff31a0000000000000001 0xc0173d45a0 { true} { false} 0}

Stack:

goroutine 4176618 [running]:
github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc0173cb7a0, {0x7a7a598, 0xc0098cafa0})
	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:115 +0x26d
created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:133 +0x139

Log preceding fatal error

   "sql": "DROP SCHEMA \"public\" CASCADE",
   "expectedExecErr": "3F000"
  }
 ],
 "expectedExecErrors": "3F000",
 "expectedCommitErrors": "",
 "message": "ROLLBACK; Successfully got expected execution error.: ERROR: cannot drop schema \"public\" (SQLSTATE 3F000)"
}
{
 "workerId": 0,
 "clientTimestamp": "09:59:49.590841",
 "ops": [
  "BEGIN",
  {
   "sql": "CREATE SEQUENCE IF NOT EXISTS public.seq179"
  },
  "COMMIT"
 ],
 "expectedExecErrors": "",
 "expectedCommitErrors": "",
 "message": ""
}
{
 "workerId": 0,
 "clientTimestamp": "09:59:50.225835",
 "ops": [
  "BEGIN",
  {
   "sql": "DROP SEQUENCE public.seq177"
  },
  "COMMIT"
 ],
 "expectedExecErrors": "",
 "expectedCommitErrors": "",
 "message": ""
}
{
 "workerId": 0,
 "clientTimestamp": "09:59:51.658481",
 "ops": [
  "BEGIN",
  {
   "sql": "DROP SCHEMA \"public\" CASCADE",
   "expectedExecErr": "3F000"
  }
 ],
 "expectedExecErrors": "3F000",
 "expectedCommitErrors": "",
 "message": "ROLLBACK; Successfully got expected execution error.: ERROR: cannot drop schema \"public\" (SQLSTATE 3F000)"
}

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [error dropping enum type] #111540 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [error dropping enum type] [C-test-failure O-robot O-schema-testing T-sql-foundations branch-master db-cy-23]

_{This test on roachdash | Improve this report!}

rafiss · 2023-10-17T18:14:31Z

@chrisseto can you check if this might be related to any new changes added to the schema change workload?

cockroach-teamcity · 2023-10-18T09:49:54Z

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed with artifacts on release-23.2 @ 26fbd322059f7e42084648e3cba99e697958f1b8:

1, 29, mvcc_statistics, 63
1, 29, namespace, 30
1, 29, privileges, 51
1, 29, protected_ts_meta, 31
1, 29, protected_ts_records, 32
1, 29, rangelog, 13
1, 29, region_liveness, 9
1, 29, replication_constraint_stats, 25
1, 29, replication_critical_localities, 26
1, 29, replication_stats, 27
1, 29, reports_meta, 28
1, 29, role_id_seq, 48
1, 29, role_members, 23
1, 29, role_options, 33
1, 29, scheduled_jobs, 37
1, 29, settings, 6
1, 29, span_configurations, 47
1, 29, span_stats_buckets, 55
1, 29, span_stats_samples, 56
1, 29, span_stats_tenant_boundaries, 57
1, 29, span_stats_unique_keys, 54
1, 29, sql_instances, 46
1, 29, sqlliveness, 39
1, 29, statement_activity, 60
1, 29, statement_bundle_chunks, 34
1, 29, statement_diagnostics, 36
1, 29, statement_diagnostics_requests, 35
1, 29, statement_execution_insights, 65
1, 29, statement_statistics, 42
1, 29, table_statistics, 20
1, 29, task_payloads, 58
1, 29, tenant_id_seq, 62
1, 29, tenant_settings, 50
1, 29, tenant_tasks, 59
1, 29, tenant_usage, 45
1, 29, tenants, 8
1, 29, transaction_activity, 61
1, 29, transaction_execution_insights, 64
1, 29, transaction_statistics, 43
1, 29, ui, 14
1, 29, users, 4
1, 29, web_sessions, 19
1, 29, zones, 5
100, 0, public, 101
102, 0, public, 103
104, 0, public, 105
    schema_change_external_test.go:82: backup in /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload782688638
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload782688638
--- FAIL: TestWorkload (18.42s)

Parameters: TAGS=bazel,gss , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [error dropping enum type] #111540 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [error dropping enum type] [C-test-failure O-robot O-schema-testing T-sql-foundations branch-master db-cy-23]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-10-26T09:59:58Z

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed with artifacts on release-23.2 @ ff055942cbd72f699764a4c7d185af79d878e036:

1, 29, rangelog, 13
1, 29, region_liveness, 9
1, 29, replication_constraint_stats, 25
1, 29, replication_critical_localities, 26
1, 29, replication_stats, 27
1, 29, reports_meta, 28
1, 29, role_id_seq, 48
1, 29, role_members, 23
1, 29, role_options, 33
1, 29, scheduled_jobs, 37
1, 29, settings, 6
1, 29, span_configurations, 47
1, 29, span_stats_buckets, 55
1, 29, span_stats_samples, 56
1, 29, span_stats_tenant_boundaries, 57
1, 29, span_stats_unique_keys, 54
1, 29, sql_instances, 46
1, 29, sqlliveness, 39
1, 29, statement_activity, 60
1, 29, statement_bundle_chunks, 34
1, 29, statement_diagnostics, 36
1, 29, statement_diagnostics_requests, 35
1, 29, statement_execution_insights, 65
1, 29, statement_statistics, 42
1, 29, table_statistics, 20
1, 29, task_payloads, 58
1, 29, tenant_id_seq, 62
1, 29, tenant_settings, 50
1, 29, tenant_tasks, 59
1, 29, tenant_usage, 45
1, 29, tenants, 8
1, 29, transaction_activity, 61
1, 29, transaction_execution_insights, 64
1, 29, transaction_statistics, 43
1, 29, ui, 14
1, 29, users, 4
1, 29, web_sessions, 19
1, 29, zones, 5
100, 0, public, 101
102, 0, public, 103
104, 0, public, 105
104, 0, schema165, 157
104, 105, _enum114, 138
104, 105, enum114, 137
104, 105, seq156, 151
104, 157, seq171, 161
    schema_change_external_test.go:82: backup in /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload4074330552
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload4074330552
--- FAIL: TestWorkload (109.02s)

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed #112834 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [C-test-failure O-robot T-sql-foundations branch-release-23.1]
pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [error dropping enum type] #111540 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [error dropping enum type] [C-test-failure O-robot O-schema-testing T-sql-foundations branch-master db-cy-23]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-10-28T09:42:36Z

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed with artifacts on release-23.2 @ 877e24c91113ef9fe67fae44511f9f77cf2650c2:

1, 29, privileges, 51
1, 29, protected_ts_meta, 31
1, 29, protected_ts_records, 32
1, 29, rangelog, 13
1, 29, region_liveness, 9
1, 29, replication_constraint_stats, 25
1, 29, replication_critical_localities, 26
1, 29, replication_stats, 27
1, 29, reports_meta, 28
1, 29, role_id_seq, 48
1, 29, role_members, 23
1, 29, role_options, 33
1, 29, scheduled_jobs, 37
1, 29, settings, 6
1, 29, span_configurations, 47
1, 29, span_stats_buckets, 55
1, 29, span_stats_samples, 56
1, 29, span_stats_tenant_boundaries, 57
1, 29, span_stats_unique_keys, 54
1, 29, sql_instances, 46
1, 29, sqlliveness, 39
1, 29, statement_activity, 60
1, 29, statement_bundle_chunks, 34
1, 29, statement_diagnostics, 36
1, 29, statement_diagnostics_requests, 35
1, 29, statement_execution_insights, 65
1, 29, statement_statistics, 42
1, 29, table_statistics, 20
1, 29, task_payloads, 58
1, 29, tenant_id_seq, 62
1, 29, tenant_settings, 50
1, 29, tenant_tasks, 59
1, 29, tenant_usage, 45
1, 29, tenants, 8
1, 29, transaction_activity, 61
1, 29, transaction_execution_insights, 64
1, 29, transaction_statistics, 43
1, 29, ui, 14
1, 29, users, 4
1, 29, web_sessions, 19
1, 29, zones, 5
100, 0, public, 101
102, 0, public, 103
104, 0, public, 105
104, 105, seq30, 111
104, 105, seq45, 114
    schema_change_external_test.go:82: backup in /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload1378507392
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload1378507392
--- FAIL: TestWorkload (11.32s)

Parameters: TAGS=bazel,gss , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed #112834 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [C-test-failure O-robot T-sql-foundations branch-release-23.1]
pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [error dropping enum type] #111540 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [error dropping enum type] [C-test-failure O-robot O-schema-testing T-sql-foundations branch-master db-cy-23]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-10-28T09:48:34Z

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed with artifacts on release-23.2 @ 877e24c91113ef9fe67fae44511f9f77cf2650c2:

1, 29, rangelog, 13
1, 29, region_liveness, 9
1, 29, replication_constraint_stats, 25
1, 29, replication_critical_localities, 26
1, 29, replication_stats, 27
1, 29, reports_meta, 28
1, 29, role_id_seq, 48
1, 29, role_members, 23
1, 29, role_options, 33
1, 29, scheduled_jobs, 37
1, 29, settings, 6
1, 29, span_configurations, 47
1, 29, span_stats_buckets, 55
1, 29, span_stats_samples, 56
1, 29, span_stats_tenant_boundaries, 57
1, 29, span_stats_unique_keys, 54
1, 29, sql_instances, 46
1, 29, sqlliveness, 39
1, 29, statement_activity, 60
1, 29, statement_bundle_chunks, 34
1, 29, statement_diagnostics, 36
1, 29, statement_diagnostics_requests, 35
1, 29, statement_execution_insights, 65
1, 29, statement_statistics, 42
1, 29, table_statistics, 20
1, 29, task_payloads, 58
1, 29, tenant_id_seq, 62
1, 29, tenant_settings, 50
1, 29, tenant_tasks, 59
1, 29, tenant_usage, 45
1, 29, tenants, 8
1, 29, transaction_activity, 61
1, 29, transaction_execution_insights, 64
1, 29, transaction_statistics, 43
1, 29, ui, 14
1, 29, users, 4
1, 29, web_sessions, 19
1, 29, zones, 5
100, 0, public, 101
102, 0, public, 103
104, 0, public, 105
104, 105, _enum65, 124
104, 105, enum65, 123
104, 105, seq106, 132
104, 105, seq130, 133
104, 105, seq135, 135
    schema_change_external_test.go:82: backup in /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload2777807237
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload2777807237
--- FAIL: TestWorkload (69.31s)

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed #112834 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [C-test-failure O-robot T-sql-foundations branch-release-23.1]
pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [error dropping enum type] #111540 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [error dropping enum type] [C-test-failure O-robot O-schema-testing T-sql-foundations branch-master db-cy-23]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-10-30T09:51:38Z

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed with artifacts on release-23.2 @ 1e5be546f219b64b171cb1f39985706b5809d79b:

Fatal error:

panic: connection has timed out &{0xc00917e460 0xc005eb1e60 1792d986a13e8c4f0000000000000001 0xc00a3a8960 { true} { false} 0}

Stack:

goroutine 22268 [running]:
github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc007f67560, {0x7a7c0e8, 0xc0081d9630})
	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:115 +0x26d
created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:133 +0x139

Log preceding fatal error

   "expectedExecErr": "3F000",
   "potentialExecErr": "0A000,42601,42704"
  }
 ],
 "expectedExecErrors": "3F000",
 "expectedCommitErrors": "",
 "message": "ROLLBACK; Successfully got expected execution error.: ERROR: cannot create \"schema158.table159\" because the target database or schema does not exist (SQLSTATE 3F000)"
}
{
 "workerId": 1,
 "clientTimestamp": "09:46:37.946204",
 "ops": [
  "BEGIN",
  {
   "sql": "DROP SCHEMA \"public\" CASCADE",
   "expectedExecErr": "3F000"
  }
 ],
 "expectedExecErrors": "3F000",
 "expectedCommitErrors": "",
 "message": "ROLLBACK; Successfully got expected execution error.: ERROR: cannot drop schema \"public\" (SQLSTATE 3F000)"
}
{
 "workerId": 1,
 "clientTimestamp": "09:46:37.996376",
 "ops": [
  "BEGIN",
  {
   "sql": "ALTER TABLE public.table178 ADD COLUMN IrrelevantColumnName string",
   "expectedExecErr": "42P01"
  }
 ],
 "expectedExecErrors": "42P01",
 "expectedCommitErrors": "",
 "message": "ROLLBACK; Successfully got expected execution error.: ERROR: relation \"public.table178\" does not exist (SQLSTATE 42P01)"
}
{
 "workerId": 1,
 "clientTimestamp": "09:46:38.030767",
 "ops": [
  "BEGIN",
  {
   "sql": "DROP SEQUENCE public.seq155"
  },
  "COMMIT"
 ],
 "expectedExecErrors": "",
 "expectedCommitErrors": "",
 "message": ""
}

Parameters: TAGS=bazel,gss , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed #112834 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [C-test-failure O-robot T-sql-foundations branch-release-23.1]
pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [error dropping enum type] #111540 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [error dropping enum type] [C-test-failure O-robot O-schema-testing T-sql-foundations branch-master db-cy-23]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-10-30T10:13:11Z

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed with artifacts on release-23.2 @ 1e5be546f219b64b171cb1f39985706b5809d79b:

Fatal error:

panic: connection has timed out &{0xc009455cc0 0xc01256da60 1792daab4aede20f0000000000000001 0xc018a17c80 { true} { false} 0}

Stack:

goroutine 4007567 [running]:
github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc00d6923c0, {0x7a81208, 0xc009bb54a0})
	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:115 +0x26d
created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:133 +0x139

Log preceding fatal error

 "message": "ROLLBACK; Successfully got expected execution error.: ERROR: cannot drop schema \"public\" (SQLSTATE 3F000)"
}
{
 "workerId": 1,
 "clientTimestamp": "10:09:03.928053",
 "ops": [
  "BEGIN",
  {
   "sql": "DROP SEQUENCE public.seq152"
  },
  "COMMIT"
 ],
 "expectedExecErrors": "",
 "expectedCommitErrors": "",
 "message": ""
}
{
 "workerId": 1,
 "clientTimestamp": "10:09:05.061173",
 "ops": [
  "BEGIN",
  {
   "sql": "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()"
  },
  {
   "sql": "DROP VIEW IF EXISTS public.view163"
  },
  {
   "sql": "ALTER TABLE public.table164 ALTER COLUMN IrrelevantColumnName SET NOT NULL",
   "expectedExecErr": "42P01"
  }
 ],
 "expectedExecErrors": "42P01",
 "expectedCommitErrors": "",
 "message": "ROLLBACK; Successfully got expected execution error.: ERROR: relation \"public.table164\" does not exist (SQLSTATE 42P01)"
}
{
 "workerId": 1,
 "clientTimestamp": "10:09:05.707917",
 "ops": [
  "BEGIN",
  {
   "sql": "DROP SCHEMA \"public\" CASCADE",
   "expectedExecErr": "3F000"
  }
 ],
 "expectedExecErrors": "3F000",
 "expectedCommitErrors": "",
 "message": "ROLLBACK; Successfully got expected execution error.: ERROR: cannot drop schema \"public\" (SQLSTATE 3F000)"
}

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed #112834 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [C-test-failure O-robot T-sql-foundations branch-release-23.1]
pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [error dropping enum type] #111540 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [error dropping enum type] [C-test-failure O-robot O-schema-testing T-sql-foundations branch-master db-cy-23]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-11-01T10:04:41Z

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed with artifacts on release-23.2 @ 45bfb4bf0c88330c273363039b8063d6ed516da4:

Fatal error:

panic: connection has timed out &{0xc005385ea0 0xc012d02f00 1793774d77df3e3b0000000000000001 0xc00bd00f60 { true} { false} 0}

Stack:

goroutine 10100394 [running]:
github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc01a7bade0, {0x7a88e88, 0xc004811360})
	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:115 +0x26d
created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:133 +0x139

Log preceding fatal error

  "COMMIT"
 ],
 "expectedExecErrors": "",
 "expectedCommitErrors": "",
 "message": ""
}
{
 "workerId": 0,
 "clientTimestamp": "09:59:35.465367",
 "ops": [
  "BEGIN",
  {
   "sql": "DROP SEQUENCE public.seq174"
  },
  "COMMIT"
 ],
 "expectedExecErrors": "",
 "expectedCommitErrors": "",
 "message": ""
}
{
 "workerId": 0,
 "clientTimestamp": "09:59:36.964557",
 "ops": [
  "BEGIN",
  {
   "sql": "SELECT * FROM public.table175 AS t0  FETCH FIRST 1 ROWS ONLY",
   "expectedExecErr": "42P01",
   "potentialExecErr": "53100,53200,57014"
  }
 ],
 "expectedExecErrors": "42P01",
 "expectedCommitErrors": "",
 "message": "ROLLBACK; Successfully got expected execution error.: ERROR: relation \"public.table175\" does not exist (SQLSTATE 42P01)"
}
{
 "workerId": 0,
 "clientTimestamp": "09:59:37.43157",
 "ops": [
  "BEGIN",
  {
   "sql": "SELECT * FROM public.table176 AS t0 ,public.table177 AS t1  FETCH FIRST 1 ROWS ONLY",
   "expectedExecErr": "42P01",
   "potentialExecErr": "53100,53200,57014"
  }
 ],
 "expectedExecErrors": "42P01",
 "expectedCommitErrors": "",
 "message": "ROLLBACK; Successfully got expected execution error.: ERROR: relation \"public.table176\" does not exist (SQLSTATE 42P01)"
}

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed #113557 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [C-test-failure O-robot T-sql-foundations branch-master release-blocker]
pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed #112834 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [C-test-failure O-robot T-sql-foundations branch-release-23.1]

_{This test on roachdash | Improve this report!}

fqazi · 2023-11-01T14:45:37Z

The bug here is typedesc schema change coupled with a DROP operation colliding together, which leads to the legacy job hitting and getting hung on:

E231030 09:49:16.885234 59105 jobs/registry.go:1621 â‹® [T1,Vsystem,n2] 11792  TYPEDESC SCHEMA CHANGE job 912879752934260737: stepping through state reverting with unexpected error: descriptor is being dropped

Previously, the declarative schema changer only waited for legacy schema changer mutations/jobs on relations and did not correctly account for type schema changes. As a result, it was possible to drop a type concurrently while a schema change was trying to mutate a type, which could lead to type change jobs hanging. To address this, this patch causes declarative schema changes to wait for type modifications. Fixes: cockroachdb#112390, cockroachdb#111540 Release note (bug fix): ALTER TYPE could get stuck if DROP TYPE was executed concurrently.

113639: sql/schemachanger: prevent concurrent type desc changes r=fqazi a=fqazi Previously, the declarative schema changer only waited for legacy schema changer mutations/jobs on relations and did not correctly account for type schema changes. As a result, it was possible to drop a type concurrently while a schema change was trying to mutate a type, which could lead to type change jobs hanging. To address this, this patch causes declarative schema changes to wait for type modifications. Fixes: #112390, #111540 Release note (bug fix): ALTER TYPE could get stuck if DROP TYPE was executed concurrently. Co-authored-by: Faizan Qazi <faizan@cockroachlabs.com>

Previously, the declarative schema changer only waited for legacy schema changer mutations/jobs on relations and did not correctly account for type schema changes. As a result, it was possible to drop a type concurrently while a schema change was trying to mutate a type, which could lead to type change jobs hanging. To address this, this patch causes declarative schema changes to wait for type modifications. Fixes: #112390, #111540 Release note (bug fix): ALTER TYPE could get stuck if DROP TYPE was executed concurrently.

cockroach-teamcity added this to the 23.2 milestone Oct 15, 2023

cockroach-teamcity mentioned this issue Oct 17, 2023

pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [error dropping enum type] #111540

Closed

rafiss assigned chrisseto Oct 17, 2023

cockroach-teamcity mentioned this issue Oct 22, 2023

pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed #112834

Closed

rafiss assigned fqazi and unassigned chrisseto Oct 24, 2023

rafiss removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Oct 25, 2023

cockroach-teamcity mentioned this issue Nov 1, 2023

pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed #113557

Closed

fqazi mentioned this issue Nov 1, 2023

sql/schemachanger: prevent concurrent type desc changes #113639

Merged

craig bot closed this as completed in b443797 Nov 2, 2023

blathers-crl bot mentioned this issue Nov 2, 2023

release-23.2: sql/schemachanger: prevent concurrent type desc changes #113644

Merged

github-project-automation bot added this to SQL Foundations Aug 28, 2024

github-project-automation bot moved this to Done in SQL Foundations Aug 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed #112390

pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed #112390

cockroach-teamcity commented Oct 15, 2023 •

edited by cockroach-jira-scripts

Loading

cockroach-teamcity commented Oct 17, 2023

rafiss commented Oct 17, 2023

cockroach-teamcity commented Oct 18, 2023

cockroach-teamcity commented Oct 26, 2023

cockroach-teamcity commented Oct 28, 2023

cockroach-teamcity commented Oct 28, 2023

cockroach-teamcity commented Oct 30, 2023

cockroach-teamcity commented Oct 30, 2023

cockroach-teamcity commented Nov 1, 2023

fqazi commented Nov 1, 2023

pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed #112390

pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed #112390

Comments

cockroach-teamcity commented Oct 15, 2023 • edited by cockroach-jira-scripts Loading

cockroach-teamcity commented Oct 17, 2023

rafiss commented Oct 17, 2023

cockroach-teamcity commented Oct 18, 2023

cockroach-teamcity commented Oct 26, 2023

cockroach-teamcity commented Oct 28, 2023

cockroach-teamcity commented Oct 28, 2023

cockroach-teamcity commented Oct 30, 2023

cockroach-teamcity commented Oct 30, 2023

cockroach-teamcity commented Nov 1, 2023

fqazi commented Nov 1, 2023

cockroach-teamcity commented Oct 15, 2023 •

edited by cockroach-jira-scripts

Loading