schema: TestRevertingJobsOnDatabasesAndSchemas/failed_due_to_injected_error/drop_database_cascade flaky

[RichardJCai]

TestRevertingJobsOnDatabasesAndSchemas/failed_due_to_injected_error/drop_database_cascade is flaky, see the test history: https://teamcity.cockroachdb.com/project.html?projectId=Cockroach_UnitTests&testNameId=-1013794384827983388&tab=testDetails

Going to skip for now.

Removing test in #77049

Jira issue: CRDB-13384

[postamar]

 Error:        Received unexpected error:
                        query 'SELECT num_runs > 3 FROM crdb_internal.jobs WHERE status = 'reverting' AND description ~ '^DROP DATABASE db_drop CASCADE$'': expected:
                        (1) attached stack trace
                          -- stack trace:
                          | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).CheckQueryResultsRetry.func1
                          |   /go/src/github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:238
                          | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1
                          |   /go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:71
                          | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
                          |   /go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:207
                          | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError
                          |   /go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:77
                          | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).succeedsWithin
                          |   /go/src/github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:74
                          | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).CheckQueryResultsRetry
                          |   /go/src/github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:235
                          | github.com/cockroachdb/cockroach/pkg/sql_test.TestRevertingJobsOnDatabasesAndSchemas.func1.2
                          |   /go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer_test.go:6850
                          | testing.tRunner
                          |   /usr/local/go/src/testing/testing.go:1259
                          | runtime.goexit
                          |   /usr/local/go/src/runtime/asm_amd64.s:1581
                        Wraps: (2) query 'SELECT num_runs > 3 FROM crdb_internal.jobs WHERE status = 'reverting' AND description ~ '^DROP DATABASE db_drop CASCADE$'': expected:
                          | true
                          | got:
                          |
                        Error types: (1) *withstack.withStack (2) *errutil.leafError
          Test:         TestRevertingJobsOnDatabasesAndSchemas/failed_due_to_injected_error/drop_database_cascade

[postamar]

My understanding is that this test asserts on side effects from the old schema changer, which might not apply to the new. For a start in the new schema changer these jobs should be non-revertible.

[mgartner]

I'm reopening this issue because I witnessed this test fail on a recent CI build: https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_UnitTests/5966731?showRootCauses=true&expandBuildChangesSection=true&expandBuildProblemsSection=true&expandBuildTestsSection=true

------- Stdout: -------
=== RUN   TestRevertingJobsOnDatabasesAndSchemas/failed_due_to_injected_error/drop_schema
    sql_runner.go:101:
          Error Trace:  sql_runner.go:114
                              sql_runner.go:284
                              schema_changer_test.go:6931
          Error:        Received unexpected error:
                        query 'SELECT num_runs > 3 FROM crdb_internal.jobs WHERE status = 'reverting' AND description ~ '^DROP SCHEMA db_drop_schema.sc$'': expected:
                        (1) attached stack trace
                          -- stack trace:
                          | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).CheckQueryResultsRetry.func1
                          |   /go/src/github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:287
                          | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1
                          |   /go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:69
                          | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
                          |   /go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:207
                          | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError
                          |   /go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
                          | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).succeedsWithin
                          |   /go/src/github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:114
                          | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).CheckQueryResultsRetry
                          |   /go/src/github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:284
                          | github.com/cockroachdb/cockroach/pkg/sql_test.TestRevertingJobsOnDatabasesAndSchemas.func1.2
                          |   /go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer_test.go:6931
                          | testing.tRunner
                          |   /usr/local/go/src/testing/testing.go:1439
                          | runtime.goexit
                          |   /usr/local/go/src/runtime/asm_amd64.s:1571
                        Wraps: (2) query 'SELECT num_runs > 3 FROM crdb_internal.jobs WHERE status = 'reverting' AND description ~ '^DROP SCHEMA db_drop_schema.sc$'': expected:
                          | true
                          | got:
                          |
                        Error types: (1) *withstack.withStack (2) *errutil.leafError
        --- FAIL: TestRevertingJobsOnDatabasesAndSchemas/failed_due_to_injected_error/drop_schema (45.38s)