Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: version/mixed/nodes=5 failed #69459

Closed
cockroach-teamcity opened this issue Aug 27, 2021 · 2 comments · Fixed by #69592
Closed

roachtest: version/mixed/nodes=5 failed #69459

cockroach-teamcity opened this issue Aug 27, 2021 · 2 comments · Fixed by #69592
Assignees
@Azhng
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)

Comments

@cockroach-teamcity
Copy link
Member

roachtest.version/mixed/nodes=5 failed with artifacts on master @ 8cae60f603ccc4d83137167b3b31cab09be9d41a:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/version/mixed/nodes=5/run_1
	monitor.go:128,version.go:211,version.go:224,test_runner.go:777: monitor failure: unexpected node event: 4: dead (exit status 134)
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:211
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:224
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 4: dead (exit status 134)
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

	cluster.go:1249,context.go:89,cluster.go:1237,test_runner.go:866: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3366874-1630046181-38-n6cpu4 --oneshot --ignore-empty-nodes: exit status 1 4: dead (exit status 134)
		6: skipped
		5: 17200
		1: 21767
		3: 22362
		2: 20320
		Error: UNCLASSIFIED_PROBLEM: 4: dead (exit status 134)
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1173
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:281
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2107
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:225
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (3) 4: dead (exit status 134)
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
Reproduce

See: roachtest README

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Aug 27, 2021
@tbg
Copy link
Member

tbg commented Aug 28, 2021 

I210827 09:23:53.598678 1 util/log/flags.go:180  [-] 1  stderr capture started
panic: Payload.Type called on a payload with an unknown details type: <nil>

goroutine 2254823 [running]:
panic(0x47c9220, 0xc0008cd4b0)
	/usr/local/go/src/runtime/panic.go:1064 +0x545 fp=0xc007da7980 sp=0xc007da78b8 pc=0x48b725
github.com/cockroachdb/cockroach/pkg/jobs/jobspb.DetailsType(0x0, 0x0, 0x14)
	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobspb/wrap.go:89 +0x385 fp=0xc007da79e0 sp=0xc007da7980 pc=0x1f1bfc5
github.com/cockroachdb/cockroach/pkg/jobs/jobspb.(*Payload).Type(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/jobspb/wrap.go:56
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).createResumer(0xc000615a00, 0xc006ed2140, 0xc0002d0000, 0x0, 0x0, 0x0, 0x4c3b57c)
	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1146 +0x91 fp=0xc007da7b98 sp=0xc007da79e0 pc=0x23cda71
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob(0xc000615a00, 0x5a99a00, 0xc001760ac0, 0x98c9510c6658001, 0x5a43c40, 0xc000e1ec60, 0xc001465400, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:209 +0x6b5 fp=0xc007da7f10 sp=0xc007da7b98 pc=0x23bb355
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeClaimedJobs.func3(0xc006780d80, 0xc000615a00, 0x5a99a00, 0xc001760ac0, 0x5a43c40, 0xc000e1ec60, 0x98c9510c6658001)
	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:123 +0xaa fp=0xc007da7fa8 sp=0xc007da7f10 pc=0x23d81aa
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc007da7fb0 sp=0xc007da7fa8 pc=0x4c4b41
created by github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeClaimedJobs
	/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:121 +0x1aa

cc @ajwerner

@blathers-crl blathers-crl bot added the T-sql-schema-deprecated Use T-sql-foundations instead label Aug 28, 2021
@ajwerner
Copy link
Contributor

This is the same as #69544 (comment).

@ajwerner ajwerner removed their assignment Aug 29, 2021
@Azhng Azhng mentioned this issue Aug 30, 2021
Merged
Azhng added a commit to Azhng/cockroach that referenced this issue Aug 30, 2021
@Azhng
sql: proper version gate sql stats
e20f0af 
Previously, SQL Stats's implementation for version gating is faulty.
This means that SQL Stats's job monitor would attempt to start sql
stats compaction job in an incompatible cluster.
This commit fixed the faulty implementation.

Resolves cockroachdb#69459
Resolves cockroachdb#69541
Resolves cockroachdb#69544
Resolves cockroachdb#69565

Release justification: Category 2: Bug fixes and low-risk updates to
new functionality

Release note: None
Azhng added a commit to Azhng/cockroach that referenced this issue Aug 30, 2021
@Azhng
sql: proper version gate sql stats
412f804 
Previously, SQL Stats's implementation for version gating is faulty.
This means that SQL Stats's job monitor would attempt to start sql
stats compaction job in an incompatible cluster.
This commit fixed the faulty implementation.

Resolves cockroachdb#69459
Resolves cockroachdb#69541
Resolves cockroachdb#69544
Resolves cockroachdb#69565

Release justification: Category 2: Bug fixes and low-risk updates to
new functionality

Release note: None
Azhng added a commit to Azhng/cockroach that referenced this issue Aug 30, 2021
@Azhng
sql: proper version gate sql stats
4c5c0aa 
Previously, SQL Stats's implementation for version gating is faulty.
This means that SQL Stats's job monitor would attempt to start sql
stats compaction job in an incompatible cluster.
This commit fixed the faulty implementation.

Resolves cockroachdb#69459
Resolves cockroachdb#69541
Resolves cockroachdb#69544
Resolves cockroachdb#69565

Release justification: Category 2: Bug fixes and low-risk updates to
new functionality

Release note: None
Azhng added a commit to Azhng/cockroach that referenced this issue Aug 30, 2021
@Azhng
sql: proper version gate sql stats
8c3bbd6 
Previously, SQL Stats's implementation for version gating is faulty.
This means that SQL Stats's job monitor would attempt to start sql
stats compaction job in an incompatible cluster.
This commit fixed the faulty implementation.

Resolves cockroachdb#69459
Resolves cockroachdb#69541
Resolves cockroachdb#69544
Resolves cockroachdb#69565

Release justification: Category 2: Bug fixes and low-risk updates to
new functionality

Release note: None
craig bot pushed a commit that referenced this issue Aug 30, 2021
@xinhaoz @Azhng
Merge #69591 #69592
e76382b 
69591: sql: deflake TestStatusAPIStatements and TestStatusAPICombinedStatements r=xinhaoz a=xinhaoz

Fixes: #69557

As mentioned in #69533, we have a race condition in tests
where we request statements stats with start=now. In these
tests we expect to see no results, but because in-memory
stats have the aggregated_ts field set on iterator return,
depending on the time the test is run we might see
results returned.

For example, suppose we insert stats at 15:45. If we then request
stats at 16:00, requesting only stats aggregated after or at the
current time, i.e. start=16:00, the aggregated_ts for in-memory
stats will be set to 16:00 and thus we will see results
returned.

To deflake these tests, we stub the aggregated_ts field to
a predetermined value.

Release justification: non-production code changes
Release note: None

69592: sql: proper version gate sql stats r=maryliag,ajwerner a=Azhng

Previously, SQL Stats's implementation for version gating is faulty.
This means that SQL Stats's job monitor would attempt to start sql
stats compaction job in an incompatible cluster.
This commit fixed the faulty implementation.

Resolves #69459
Resolves #69544
Resolves #69565

Release justification: Category 2: Bug fixes and low-risk updates to
new functionality

Release note: None

Co-authored-by: Xin Hao Zhang <xzhang@cockroachlabs.com>
Co-authored-by: Azhng <archer.xn@gmail.com>
@craig craig bot closed this as completed in 3ea128d Aug 30, 2021
@exalate-issue-sync exalate-issue-sync bot removed the T-sql-schema-deprecated Use T-sql-foundations instead label May 10, 2023
@exalate-issue-sync exalate-issue-sync bot added T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) and removed T-sql-observability labels May 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Azhng Azhng

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

sql: proper version gate sql stats Azhng/cockroach
4 participants
@ajwerner @tbg @Azhng @cockroach-teamcity