pkg/ccl/testccl/sqlccl/sqlccl_test: TestTenantTempTableCleanup failed

[cockroach-teamcity]

pkg/ccl/testccl/sqlccl/sqlccl_test.TestTenantTempTableCleanup failed with artifacts on master @ f1623e0a2cffc81580ee3c3b71035efd854d89c9:

          | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowExWithCols
          | 	github.com/cockroachdb/cockroach/pkg/sql/internal.go:694
          | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowEx
          | 	github.com/cockroachdb/cockroach/pkg/sql/internal.go:680
          | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRow
          | 	github.com/cockroachdb/cockroach/pkg/sql/internal.go:664
          | github.com/cockroachdb/cockroach/pkg/upgrade/migrationstable.CheckIfMigrationCompleted
          | 	github.com/cockroachdb/cockroach/pkg/upgrade/migrationstable/migrations_table.go:118
          | github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).RunPermanentUpgrades.func2
          | 	github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:220
          | github.com/cockroachdb/cockroach/pkg/util/startup.RunIdempotentWithRetryEx[...]
          | 	github.com/cockroachdb/cockroach/pkg/util/startup/retry.go:142
          | github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).RunPermanentUpgrades
          | 	github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:216
          | github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).preStart
          | 	github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1601
          | github.com/cockroachdb/cockroach/pkg/server.(*SQLServerWrapper).PreStart
          | 	github.com/cockroachdb/cockroach/pkg/server/tenant.go:754
          | github.com/cockroachdb/cockroach/pkg/server.(*SQLServerWrapper).Start
          | 	github.com/cockroachdb/cockroach/pkg/server/tenant.go:921
          | github.com/cockroachdb/cockroach/pkg/server.(*testServer).StartTenant
          | 	github.com/cockroachdb/cockroach/pkg/server/testserver.go:1507
          | github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartTenant
          | 	github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:277
          | pkg/ccl/testccl/sqlccl/sqlccl_test_test.TestTenantTempTableCleanup
          | 	pkg/ccl/testccl/sqlccl/sqlccl_test_test/pkg/ccl/testccl/sqlccl/temp_table_clean_test.go:163
          | [...repeated from below...]
        Wraps: (2) migration-job-find-already-completed
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/rpc/nodedialer.(*Dialer).dial
          | 	github.com/cockroachdb/cockroach/pkg/rpc/nodedialer/nodedialer.go:181
          | github.com/cockroachdb/cockroach/pkg/rpc/nodedialer.(*Dialer).Dial
          | 	github.com/cockroachdb/cockroach/pkg/rpc/nodedialer/nodedialer.go:103
          | github.com/cockroachdb/cockroach/pkg/sql.runnerRequest.run
          | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:112
          | github.com/cockroachdb/cockroach/pkg/sql.(*runnerCoordinator).init.func1
          | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:152
          | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
          | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484
          | runtime.goexit
          | 	GOROOT/src/runtime/asm_amd64.s:1594
        Wraps: (4) failed to connect to n1 at 127.0.0.1:34173
        Wraps: (5) initial connection heartbeat failed
        Wraps: (6) grpc: client requested node ID 1 doesn't match server node ID 3 [code 2/Unknown]
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *netutil.InitialHeartbeatFailedError (6) *status.Error
I230807 08:09:21.697240 63957 2@rpc/peer.go:537  [T10,nsql3,rnode=3,raddr=127.0.0.1:34173,class=default,rpc] 2090  connection is now healthy
W230807 08:09:21.723780 63817 sql/temporary_schema.go:466  [T10,nsql3] 2091  error during schema cleanup, retrying: fan out rpc failed with failed to dial into node 1 (NODE_STATUS_LIVE): initial connection heartbeat failed: grpc: client requested node ID 1 doesn't match server node ID 3 [code 2/Unknown] on node 1
E230807 08:09:21.727402 63960 2@rpc/peer.go:611  [T10,nsql3,rnode=1,raddr=127.0.0.1:34173,class=default,rpc] 2092  failed connection attempt (last connected 33ms ago): grpc: client requested node ID 1 doesn't match server node ID 3 [code 2/Unknown]
--- FAIL: TestTenantTempTableCleanup (14.39s)

Parameters: TAGS=bazel,gss , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-queries

This test on roachdash | Improve this report!

Jira issue: CRDB-30404