Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

workload: run random causes panic: testing: Verbose called before Init #46569

Closed
awoods187 opened this issue Mar 25, 2020 · 0 comments · Fixed by #46573 · May be fixed by #46590
Closed

workload: run random causes panic: testing: Verbose called before Init #46569

awoods187 opened this issue Mar 25, 2020 · 0 comments · Fixed by #46573 · May be fixed by #46590
Assignees
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. S-2-temp-unavailability Temp crashes or other availability problems. Can be worked around or resolved by restarting.

Comments

@awoods187
Copy link
Contributor

awoods187 commented Mar 25, 2020

I was testing the idea in #44713 with run random when I hit a panic in the workload generator:

export CLUSTER=andy-3node
roachprod create $CLUSTER -n 4 --clouds=aws  --aws-machine-type-ssd=c5d.4xlarge
roachprod run $CLUSTER --  "sudo umount /mnt/data1; sudo mount -o discard,defaults,nobarrier /mnt/data1/; mount | grep /mnt/data1"
roachprod stage $CLUSTER:1-3 cockroach
roachprod stage $CLUSTER:4 workload
roachprod start $CLUSTER:1-3
roachprod adminurl --open $CLUSTER:1
roachprod sql $CLUSTER:1
CREATE TABLE t0(c0 INT);
select * from t0;
  c0
------
(0 rows)

Time: 5.039517ms

Then, in another shell, I tried to populate this with data:

roachprod run $CLUSTER:4 "./workload run rand --table t0 --db defaultdb {pgurl:1-3}"

panic: testing: Verbose called before Init

goroutine 1 [running]:
panic(0x3a9b5c0, 0x490b620)
	/usr/local/go/src/runtime/panic.go:722 +0x2c2 fp=0xc000e3f150 sp=0xc000e3f0c0 pc=0x72f862
testing.Verbose(...)
	/usr/local/go/src/testing/testing.go:389
github.com/cockroachdb/cockroach/pkg/workload/rand.(*random).Ops(0xc000711960, 0xc000711f10, 0x3, 0x7, 0xc0000d80c0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/workload/rand/rand.go:265 +0x2217 fp=0xc000e3f688 sp=0xc000e3f150 pc=0x16e13f7
github.com/cockroachdb/cockroach/pkg/workload/cli.runRun(0x49a5a80, 0xc000711960, 0xc000711f10, 0x3, 0x7, 0x7fff7e4bad92, 0x9, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/workload/cli/run.go:354 +0x5d0 fp=0xc000e3fc30 sp=0xc000e3f688 pc=0x1031420
github.com/cockroachdb/cockroach/pkg/workload/cli.CmdHelper.func1(0xc0006cc780, 0xc000711f10, 0x3, 0x7, 0x4, 0xdc3d70)
	/go/src/github.com/cockroachdb/cockroach/pkg/workload/cli/run.go:182 +0x23a fp=0xc000e3fd28 sp=0xc000e3fc30 pc=0x1033b0a
github.com/cockroachdb/cockroach/pkg/workload/cli.HandleErrs.func1(0xc0006cc780, 0xc000711f10, 0x3, 0x7)
	/go/src/github.com/cockroachdb/cockroach/pkg/workload/cli/cli.go:74 +0x51 fp=0xc000e3fd88 sp=0xc000e3fd28 pc=0x1032d51
github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute(0xc0006cc780, 0xc000711e30, 0x7, 0x7, 0xc0006cc780, 0xc000711e30)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766 +0x2aa fp=0xc000e3fe60 sp=0xc000e3fd88 pc=0xdbd2da
github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc000474780, 0xc000474780, 0xb2e00, 0xc00007a0b8)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852 +0x2ea fp=0xc000e3ff30 sp=0xc000e3fe60 pc=0xdbdeca
github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute(...)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
main.main()
	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/workload/main.go:22 +0x34 fp=0xc000e3ff60 sp=0xc000e3ff30 pc=0x327ca74
runtime.main()
	/usr/local/go/src/runtime/proc.go:203 +0x21e fp=0xc000e3ffe0 sp=0xc000e3ff60 pc=0x7315ae
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc000e3ffe8 sp=0xc000e3ffe0 pc=0x75fdc1

Here is a public gist of the remaining output https://gist.github.com/awoods187/f50c789955bff3b5d5ce5fb36fb849ab
V20.1.0-BETA.2-1072-G6C3C5D4

@awoods187 awoods187 added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. S-2-temp-unavailability Temp crashes or other availability problems. Can be worked around or resolved by restarting. labels Mar 25, 2020
@rohany rohany assigned rohany and unassigned jordanlewis Mar 26, 2020
@jordanlewis jordanlewis assigned jordanlewis and unassigned rohany Jun 3, 2020
@craig craig bot closed this as completed in d411572 Jun 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. S-2-temp-unavailability Temp crashes or other availability problems. Can be worked around or resolved by restarting.
Projects
None yet
3 participants