roachtest: acceptance/gossip/restart-node-one failed

[cockroach-teamcity]

(roachtest).acceptance/gossip/restart-node-one failed on master@8eb5063da9db0fc17e1216da2c73186f55ba1b4f:

		  |   | * - Your cluster is open for any client that can access 127.0.0.1.
		  |   | * - Any user, even root, can log in without providing a password.
		  |   | * - Any user, connecting as root, can read or write any data in your cluster.
		  |   | * - There is no network encryption nor authentication, and thus no confidentiality.
		  |   | * 
		  |   | * Check out how to secure your cluster: https://www.cockroachlabs.com/docs/v20.2/secure-a-cluster.html
		  |   | *
		  |   | *
		  |   | * ERROR: ERROR: cockroach server exited with error: failed to create engines: pebble: error when replaying WAL: pebble/record: invalid chunk
		  |   | *
		  |   | ERROR: cockroach server exited with error: failed to create engines: pebble: error when replaying WAL: pebble/record: invalid chunk
		  |   | Failed running "start"
		  |   | E200910 05:24:30.318479 1 cli/error.go:394  ERROR: exit status 1
		  |   | ERROR: exit status 1
		  |   | Failed running "start"
		  | Wraps: (3) exit status 1
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *exec.ExitError: 
		  | I200910 05:24:30.324045 1 cluster_synced.go:1733  command failed
		  |
		  | stdout:
		  | local: starting nodes
		Wraps: (2) exit status 1
		Error types: (1) *main.withCommandDetails (2) *exec.ExitError

	cluster.go:1651,context.go:135,cluster.go:1640,test_runner.go:823: dead node detection: /go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor local --oneshot --ignore-empty-nodes: exit status 1 1: dead
		4: 14653
		2: 14483
		3: 14568
		Error: UNCLASSIFIED_PROBLEM: 1: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
		  | main.main
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (3) 1: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

More

Artifacts: /acceptance/gossip/restart-node-one
Related:

• roachtest: acceptance/gossip/restart-node-one failed #54142 roachtest: acceptance/gossip/restart-node-one failed C-test-failure O-roachtest O-robot branch-release-20.2 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

[cockroach-teamcity]

(roachtest).acceptance/gossip/restart-node-one failed on master@6227db03fb84e83cf2e0519a70de963fb24d457c:

		  |   | * - Your cluster is open for any client that can access 127.0.0.1.
		  |   | * - Any user, even root, can log in without providing a password.
		  |   | * - Any user, connecting as root, can read or write any data in your cluster.
		  |   | * - There is no network encryption nor authentication, and thus no confidentiality.
		  |   | * 
		  |   | * Check out how to secure your cluster: https://www.cockroachlabs.com/docs/v20.2/secure-a-cluster.html
		  |   | *
		  |   | *
		  |   | * ERROR: ERROR: cockroach server exited with error: failed to create engines: pebble: error when replaying WAL: pebble/record: invalid chunk
		  |   | *
		  |   | ERROR: cockroach server exited with error: failed to create engines: pebble: error when replaying WAL: pebble/record: invalid chunk
		  |   | Failed running "start"
		  |   | E200910 18:50:50.882903 1 cli/error.go:394  ERROR: exit status 1
		  |   | ERROR: exit status 1
		  |   | Failed running "start"
		  | Wraps: (3) exit status 1
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *exec.ExitError: 
		  | I200910 18:50:50.888234 1 cluster_synced.go:1733  command failed
		  |
		  | stdout:
		  | local: starting nodes
		Wraps: (2) exit status 1
		Error types: (1) *main.withCommandDetails (2) *exec.ExitError

	cluster.go:1651,context.go:135,cluster.go:1640,test_runner.go:823: dead node detection: /go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor local --oneshot --ignore-empty-nodes: exit status 1 1: dead
		2: 15328
		3: 15414
		4: 15500
		Error: UNCLASSIFIED_PROBLEM: 1: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
		  | main.main
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (3) 1: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

More

Artifacts: /acceptance/gossip/restart-node-one
Related:

• roachtest: acceptance/gossip/restart-node-one failed #54142 roachtest: acceptance/gossip/restart-node-one failed C-test-failure O-roachtest O-robot branch-release-20.2 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

[irfansharif]

From internally:

@jbowens: It means one of Pebble’s write-ahead logs was deemed corrupt during startup, in this case erroneously. There’s a fix committed to Pebble, but I haven’t bumped cockroach master’s pebble version yet. I’ll put that PR up now