roachtest: autoupgrade failed

[cockroach-teamcity]

(roachtest).autoupgrade failed on master@3d01fda046c609ee5947e2f41c099dae35b206c1:

	cluster.go:1666,context.go:140,cluster.go:1655,test_runner.go:841: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2597708-1611128699-19-n5cpu4 --oneshot --ignore-empty-nodes: exit status 1 5: 4502
		1: 5170
		2: 4339
		3: dead
		4: dead
		Error: UNCLASSIFIED_PROBLEM: 3: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) secondary error attachment
		  | 4: dead
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | main.glob..func14
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1147
		  |   | main.wrap.func1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:271
		  |   | github.com/spf13/cobra.(*Command).execute
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  |   | github.com/spf13/cobra.(*Command).ExecuteC
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  |   | github.com/spf13/cobra.(*Command).Execute
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  |   | main.main
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1852
		  |   | runtime.main
		  |   | 	/usr/local/go/src/runtime/proc.go:204
		  |   | runtime.goexit
		  |   | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		  | Wraps: (2) 4: dead
		  | Error types: (1) *withstack.withStack (2) *errutil.leafError
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1147
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:271
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1852
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (4) 3: dead
		Error types: (1) errors.Unclassified (2) *secondary.withSecondaryError (3) *withstack.withStack (4) *errutil.leafError

More

Artifacts: /autoupgrade
Related:

• roachtest: autoupgrade failed #59036 roachtest: autoupgrade failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

[asubiotto]

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1d7c8d2]

goroutine 1570 [running]:
panic(0x42b1e80, 0x775d060)
	/usr/local/go/src/runtime/panic.go:1064 +0x545 fp=0xc000baf4b8 sp=0xc000baf3f0 pc=0x4e1825
runtime.panicmem(...)
	/usr/local/go/src/runtime/panic.go:212
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:742 +0x413 fp=0xc000baf4e8 sp=0xc000baf4b8 pc=0x4f8573
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Version(0xc0005b7800, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica.go:805 +0x72 fp=0xc000baf528 sp=0xc000baf4e8 pc=0x1d7c8d2
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).PurgeOutdatedReplicas.func1(0xc0005b7800, 0xc002350901)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store.go:2813 +0x8c fp=0xc000baf5c8 sp=0xc000baf528 pc=0x1e4890c
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*storeReplicaVisitor).Visit(0xc002350930, 0xc000baf688)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store.go:396 +0x151 fp=0xc000baf630 sp=0xc000baf5c8 pc=0x1dfd031
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).VisitReplicas(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store.go:2012
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).PurgeOutdatedReplicas(0xc00124fc00, 0x56471a0, 0xc0023508a0, 0x200000014, 0xe00000000, 0xc0013458c8, 0xc000fe8990)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store.go:2812 +0x1bf fp=0xc000baf6e8 sp=0xc000baf630 pc=0x1e0911f
github.com/cockroachdb/cockroach/pkg/server.(*migrationServer).PurgeOutdatedReplicas.func1(0xc00124fc00, 0x4baa45, 0xc000baf788)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/migration.go:184 +0x65 fp=0xc000baf730 sp=0xc000baf6e8 pc=0x3948865
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).VisitStores.func1(0x4, 0xc00124fc00, 0xc000baf788)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/stores.go:131 +0x38 fp=0xc000baf760 sp=0xc000baf730 pc=0x1e4c418
github.com/cockroachdb/cockroach/pkg/util/syncutil.(*IntMap).Range(0xc000a99220, 0xc000baf818)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/syncutil/int_map.go:352 +0x130 fp=0xc000baf7f8 sp=0xc000baf760 pc=0x7318f0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).VisitStores(0xc000a991f0, 0xc000baf888, 0x496000d, 0x17)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/stores.go:130 +0x75 fp=0xc000baf840 sp=0xc000baf7f8 pc=0x1e2a995
github.com/cockroachdb/cockroach/pkg/server.(*migrationServer).PurgeOutdatedReplicas(0xc0006c2c20, 0x56471a0, 0xc002350810, 0xc000011528, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/migration.go:183 +0x15e fp=0xc000baf8c8 sp=0xc000baf840 pc=0x390bb3e
github.com/cockroachdb/cockroach/pkg/server/serverpb._Migration_PurgeOutdatedReplicas_Handler.func1(0x56471a0, 0xc002350810, 0x4754920, 0xc000011528, 0x0, 0x0, 0x6, 0xc000a88270)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/serverpb/migration.pb.go:487 +0x89 fp=0xc000baf910 sp=0xc000baf8c8 pc=0x239f449
github.com/cockroachdb/cockroach/pkg/util/tracing.ServerInterceptor.func1(0x56471a0, 0xc002350810, 0x4754920, 0xc000011528, 0xc00234e880, 0xc00234e8a0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/tracing/grpc_interceptor.go:124 +0x4bb fp=0xc000bafa08 sp=0xc000baf910 pc=0xe693fb
google.golang.org/grpc.getChainUnaryHandler.func1(0x56471a0, 0xc002350810, 0x4754920, 0xc000011528, 0xc0023f5ac8, 0xce1a48, 0x462aa60, 0xc002356140)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:921 +0xe7 fp=0xc000bafa88 sp=0xc000bafa08 pc=0xcf4a07
github.com/cockroachdb/cockroach/pkg/rpc.NewServer.func1(0x56471a0, 0xc002350810, 0x4754920, 0xc000011528, 0xc00234e880, 0xc002356140, 0xc002356140, 0x20, 0x4397dc0, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:176 +0xa8 fp=0xc000bafad8 sp=0xc000bafa88 pc=0x196fde8
google.golang.org/grpc.chainUnaryServerInterceptors.func1(0x56471a0, 0xc002350810, 0x4754920, 0xc000011528, 0xc00234e880, 0xc00234e8a0, 0xc0023f5ba0, 0x6b5be6, 0x46469e0, 0xc002350810)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:907 +0xd0 fp=0xc000bafb40 sp=0xc000bafad8 pc=0xcf48b0
github.com/cockroachdb/cockroach/pkg/server/serverpb._Migration_PurgeOutdatedReplicas_Handler(0x463d980, 0xc0006c2c20, 0x56471a0, 0xc002350810, 0xc0023452c0, 0xc0017eac20, 0x56471a0, 0xc002350810, 0xc00131ccf6, 0xa)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/serverpb/migration.pb.go:489 +0x150 fp=0xc000bafbb0 sp=0xc000bafb40 pc=0x2302a50
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0009409c0, 0x56ae760, 0xc001f20000, 0xc001e5bb00, 0xc0004020f0, 0x778a288, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1082 +0x522 fp=0xc000bafe40 sp=0xc000bafbb0 pc=0xce2022
google.golang.org/grpc.(*Server).handleStream(0xc0009409c0, 0x56ae760, 0xc001f20000, 0xc001e5bb00, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1405 +0xcc5 fp=0xc000baff68 sp=0xc000bafe40 pc=0xce6125
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc00011b3a0, 0xc0009409c0, 0x56ae760, 0xc001f20000, 0xc001e5bb00)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:746 +0xa5 fp=0xc000baffb8 sp=0xc000baff68 pc=0xcf4565
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc000baffc0 sp=0xc000baffb8 pc=0x51ac01
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:744 +0xa5

[asubiotto]

Looks like another instance of #58523. Can I close as dup @tbg?

[tbg]

Yes, #59194 should avoid that failure mode.

[asubiotto]

OK, thanks. If we now avoid the NPE, can we remove the alpha release blocker tag on the other issue or do we think there's a serious underlying problem?

[tbg]

Yes

[tbg]

There's a serious conceptual problem, but an old one that isn't going to start wreaking havoc just now, not a release blocker.

…

On Thu, Jan 21, 2021 at 2:46 PM Alfonso Subiotto Marqués < ***@***.***> wrote: Closed #59187 <#59187>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#59187 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGXPZDM4TS5XRR5C4XZP6TS3AV4PANCNFSM4WKJUAZA> .