Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: follower-reads/survival=zone/locality=global/reads=exact-staleness failed #69952

Closed
cockroach-teamcity opened this issue Sep 9, 2021 · 4 comments · Fixed by #75739
Closed

roachtest: follower-reads/survival=zone/locality=global/reads=exact-staleness failed #69952

cockroach-teamcity opened this issue Sep 9, 2021 · 4 comments · Fixed by #75739
Assignees
@andreimatei
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. skipped-test T-kv KV Team

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Sep 9, 2021 •
edited by exalate-issue-sync bot
Loading

roachtest.follower-reads/survival=zone/locality=global/reads=exact-staleness failed with artifacts on master @ 2a7ef521a38d923cca9ec96e229237bbedc0c5fa:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/follower-reads/survival=zone/locality=global/reads=exact-staleness/run_1
	follower_reads.go:657,follower_reads.go:319,follower_reads.go:57,test_runner.go:777: too many intervals with more than 2 nodes with low follower read ratios: 5 intervals > 4 threshold. Bad intervals:
		interval 09:44:40-09:44:50: n1 ratio: 0.601 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 n5 ratio: 2.430 n6 ratio: 1.000 
		interval 09:44:50-09:45:00: n1 ratio: 0.000 n2 ratio: 0.658 n3 ratio: 1.000 n4 ratio: 0.000 n5 ratio: 2.441 n6 ratio: 1.000 
		interval 09:45:20-09:45:30: n1 ratio: 0.679 n2 ratio: 1.000 n3 ratio: 0.895 n4 ratio: 0.000 n5 ratio: 2.430 n6 ratio: 1.000 
		interval 09:45:40-09:45:50: n1 ratio: 0.648 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 n5 ratio: 2.450 n6 ratio: 1.000 
		interval 09:45:50-09:46:00: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.768 n4 ratio: 0.000 n5 ratio: 2.430 n6 ratio: 1.000
Reproduce

See: roachtest README

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Epic CRDB-10559
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Sep 9, 2021
@tbg
Copy link
Member

tbg commented Sep 9, 2021

cc @nvanbenschoten

@nvanbenschoten nvanbenschoten mentioned this issue Sep 14, 2021
Closed
@nvanbenschoten nvanbenschoten removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Sep 21, 2021
@cockroach-teamcity
Copy link
Member Author

roachtest.follower-reads/survival=zone/locality=global/reads=exact-staleness failed with artifacts on master @ 5f53feb1ce7070453e9ce52d99013934718cc9d7:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/follower-reads/survival=zone/locality=global/reads=exact-staleness/run_1
	follower_reads.go:657,follower_reads.go:319,follower_reads.go:57,test_runner.go:777: too many intervals with more than 2 nodes with low follower read ratios: 5 intervals > 4 threshold. Bad intervals:
		interval 09:44:50-09:45:00: n1 ratio: 1.000 n2 ratio: 0.472 n3 ratio: 0.699 n4 ratio: 0.000 n5 ratio: 4.524 n6 ratio: 1.000 
		interval 09:45:10-09:45:20: n1 ratio: 0.415 n2 ratio: 0.842 n3 ratio: 1.000 n4 ratio: 0.000 n5 ratio: 4.498 n6 ratio: 1.000 
		interval 09:45:30-09:45:40: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.863 n4 ratio: 0.000 n5 ratio: 4.466 n6 ratio: 1.000 
		interval 09:45:40-09:45:50: n1 ratio: 0.697 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 n5 ratio: 4.506 n6 ratio: 1.000 
		interval 09:46:30-09:46:40: n1 ratio: 1.000 n2 ratio: 0.288 n3 ratio: 0.895 n4 ratio: 0.000 n5 ratio: 4.506 n6 ratio: 1.000
Reproduce

See: roachtest README

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.follower-reads/survival=zone/locality=global/reads=exact-staleness failed with artifacts on master @ aef9aa08e12d50ca6401440cac5b57d5a5d6b720:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/follower-reads/survival=zone/locality=global/reads=exact-staleness/run_1
	follower_reads.go:752,follower_reads.go:364,follower_reads.go:71,test_runner.go:777: too many intervals with more than 2 nodes with low follower read ratios: 5 intervals > 4 threshold. Bad intervals:
		interval 10:21:50-10:22:00: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 1.987 n5 ratio: 1.000 n6 ratio: 0.748 
		interval 10:22:00-10:22:10: n1 ratio: 0.494 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 2.000 n5 ratio: 1.000 n6 ratio: 0.000 
		interval 10:22:50-10:23:00: n1 ratio: 0.770 n2 ratio: 0.403 n3 ratio: 0.000 n4 ratio: 2.000 n5 ratio: 1.000 n6 ratio: 1.000 
		interval 10:24:00-10:24:10: n1 ratio: 0.631 n2 ratio: 0.186 n3 ratio: 0.000 n4 ratio: 1.974 n5 ratio: 1.000 n6 ratio: 1.000 
		interval 10:24:40-10:24:50: n1 ratio: 0.517 n2 ratio: 0.654 n3 ratio: 0.000 n4 ratio: 1.985 n5 ratio: 1.000 n6 ratio: 1.000
Reproduce

See: roachtest README

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

@exalate-issue-sync exalate-issue-sync bot added the T-kv KV Team label Oct 11, 2021
@cockroach-teamcity
Copy link
Member Author

roachtest.follower-reads/survival=zone/locality=global/reads=exact-staleness failed with artifacts on master @ a5158c4c93e9dca858b38ef7e94321f0a0a2b5c5:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/follower-reads/survival=zone/locality=global/reads=exact-staleness/run_1
	follower_reads.go:270,follower_reads.go:72,test_runner.go:780: failed to get follower read counts: Get "http://34.75.241.29:26258/_status/vars": dial tcp 34.75.241.29:26258: connect: connection refused
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

andreimatei added a commit to andreimatei/cockroach that referenced this issue Jan 31, 2022
@andreimatei
sql: fix the sql.trace.stmt.enable_threshold cluster setting
13c3b7c 
This setting had rotted, leading to a span use-after-Finish.

This was caught by the recently-unskipped follower reads roachtests
(roachtests panic on use-after-finish), accounting for all the issues
below. The last entry in these issues refers to the nodes crashing
because of the bug fixed here. Some of these issues have been open for a
long time for more relevant failures - those have been fixed in cockroachdb#72296.

Fixes cockroachdb#75716
Fixes cockroachdb#75715
Fixes cockroachdb#75714
Fixes cockroachdb#71244
Fixes cockroachdb#71126
Fixes cockroachdb#70818
Fixes cockroachdb#70191
Fixes cockroachdb#70011
Fixes cockroachdb#70010
Fixes cockroachdb#69952
Fixes cockroachdb#69951

Release note: None
@andreimatei andreimatei mentioned this issue Jan 31, 2022
Merged
andreimatei added a commit to andreimatei/cockroach that referenced this issue Jan 31, 2022
@andreimatei
sql: fix the sql.trace.stmt.enable_threshold cluster setting
e85fe73 
This setting had rotted, leading to a span use-after-Finish.

This was caught by the recently-unskipped follower reads roachtests
(roachtests panic on use-after-finish), accounting for all the issues
below. The last entry in these issues refers to the nodes crashing
because of the bug fixed here. Some of these issues have been open for a
long time for more relevant failures - those have been fixed in cockroachdb#72296.

Fixes cockroachdb#75716
Fixes cockroachdb#75715
Fixes cockroachdb#75714
Fixes cockroachdb#71244
Fixes cockroachdb#71126
Fixes cockroachdb#70818
Fixes cockroachdb#70191
Fixes cockroachdb#70011
Fixes cockroachdb#70010
Fixes cockroachdb#69952
Fixes cockroachdb#69951

Release note: None
craig bot pushed a commit that referenced this issue Feb 1, 2022
@andreimatei @yuzefovich
Merge #75739 #75764
19a5d53 
75739: sql: fix the sql.trace.stmt.enable_threshold cluster setting  r=andreimatei a=andreimatei

This setting had rotted, leading to a span use-after-Finish.

This was caught by the recently-unskipped follower reads roachtests
(roachtests panic on use-after-finish), accounting for all the issues
below. The last entry in these issues refers to the nodes crashing
because of the bug fixed here. Some of these issues have been open for a
long time for more relevant failures - those have been fixed in #72296.

Fixes #75716
Fixes #75715
Fixes #75714
Fixes #71244
Fixes #71126
Fixes #70818
Fixes #70191
Fixes #70011
Fixes #70010
Fixes #69952
Fixes #69951

Release note: None

75764: roachtest: add a variant of tpch_concurrency with no admission control r=yuzefovich a=yuzefovich

Informs: #74836.

Release note: None

Co-authored-by: Andrei Matei <andrei@cockroachlabs.com>
Co-authored-by: Yahor Yuzefovich <yahor@cockroachlabs.com>
@craig craig bot closed this as completed in #75739 Feb 1, 2022
@craig craig bot closed this as completed in f5d78e2 Feb 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@andreimatei andreimatei

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. skipped-test T-kv KV Team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

sql: fix the sql.trace.stmt.enable_threshold cluster setting andreimatei/cockroach
4 participants
@andreimatei @tbg @nvanbenschoten @cockroach-teamcity