Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: admission-control/disk-bandwidth-limiter failed #135056

Closed
cockroach-teamcity opened this issue Nov 13, 2024 · 12 comments · Fixed by #135680
Closed

roachtest: admission-control/disk-bandwidth-limiter failed #135056

cockroach-teamcity opened this issue Nov 13, 2024 · 12 comments · Fixed by #135680
Assignees
@aadityasondhi
Labels
A-storage Relating to our storage engine (Pebble) on-disk storage. branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. T-storage Storage Team

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Nov 13, 2024 •
edited by cockroach-jira-scripts
Loading

roachtest.admission-control/disk-bandwidth-limiter failed with artifacts on master @ 28e473bec7cb6f8b0a1c8147efce4a662bcdf124:

(admission_control_disk_bandwidth_overload.go:187).3: write bandwidth 79.069844 exceeded threshold of 78.750000, read bandwidth: 12.930156, total bandwidth: 92.000000
(cluster.go:2452).Run: context canceled
(cluster.go:2452).Run: context canceled
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/admission-control/disk-bandwidth-limiter/cpu_arch=arm64/run_1

Parameters:

  • ROACHTEST_arch=arm64
  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

Jira issue: CRDB-44339
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-storage Storage Team labels Nov 13, 2024
@blathers-crl blathers-crl bot added the A-storage Relating to our storage engine (Pebble) on-disk storage. label Nov 13, 2024
@cockroach-teamcity cockroach-teamcity mentioned this issue Nov 13, 2024
Closed
@aadityasondhi aadityasondhi removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Nov 13, 2024
@aadityasondhi aadityasondhi self-assigned this Nov 13, 2024
@aadityasondhi
Copy link
Collaborator

Might need this patch as well. #133310

@aadityasondhi
Copy link
Collaborator

Or actually, these blips should be smoothed out like we do for our other AC test assertions. I will try smoothing it out first.

@cockroach-teamcity
Copy link
Member Author

roachtest.admission-control/disk-bandwidth-limiter failed with artifacts on master @ 39e43b85ec3b02bc760df10fce1c19d09419d6f2:

(disk_stall.go:74).Setup: cluster needs ReusePolicyNone to support disk stalls
test artifacts and logs in: /artifacts/admission-control/disk-bandwidth-limiter/run_1

Parameters:

  • arch=amd64
  • cloud=gce
  • coverageBuild=false
  • cpu=8
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • metamorphicLeases=expiration
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.admission-control/disk-bandwidth-limiter failed with artifacts on master @ 39e43b85ec3b02bc760df10fce1c19d09419d6f2:

(disk_stall.go:74).Setup: cluster needs ReusePolicyNone to support disk stalls
test artifacts and logs in: /artifacts/admission-control/disk-bandwidth-limiter/cpu_arch=arm64/run_1

Parameters:

  • arch=arm64
  • cloud=aws
  • coverageBuild=false
  • cpu=8
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • metamorphicLeases=default
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.admission-control/disk-bandwidth-limiter failed with artifacts on master @ 6610d705724a21c836f3521f75972e65d9e9e2d4:

(disk_stall.go:74).Setup: cluster needs ReusePolicyNone to support disk stalls
test artifacts and logs in: /artifacts/admission-control/disk-bandwidth-limiter/run_1

Parameters:

  • arch=amd64
  • cloud=gce
  • coverageBuild=false
  • cpu=8
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • metamorphicLeases=default
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.admission-control/disk-bandwidth-limiter failed with artifacts on master @ 6610d705724a21c836f3521f75972e65d9e9e2d4:

(disk_stall.go:74).Setup: cluster needs ReusePolicyNone to support disk stalls
test artifacts and logs in: /artifacts/admission-control/disk-bandwidth-limiter/run_1

Parameters:

  • arch=amd64
  • cloud=aws
  • coverageBuild=false
  • cpu=8
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • metamorphicLeases=expiration
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@pav-kv
Copy link
Collaborator

pav-kv commented Nov 15, 2024 

(disk_stall.go:74).Setup: cluster needs ReusePolicyNone to support disk stalls
test artifacts and logs in: /artifacts/admission-control/disk-bandwidth-limiter/run_1
commented on existing GitHub issue #135056

@aadityasondhi need to add spec.ReuseNone() to MakeClusterSpec invocation in this roachtest.

@cockroach-teamcity
Copy link
Member Author

roachtest.admission-control/disk-bandwidth-limiter failed with artifacts on master @ e83bc46aa42f2476b4b11b9703b8038c660dc980:

(disk_stall.go:74).Setup: cluster needs ReusePolicyNone to support disk stalls
test artifacts and logs in: /artifacts/admission-control/disk-bandwidth-limiter/run_1

Parameters:

  • arch=amd64
  • cloud=gce
  • coverageBuild=false
  • cpu=8
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • metamorphicLeases=default
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.admission-control/disk-bandwidth-limiter failed with artifacts on master @ e83bc46aa42f2476b4b11b9703b8038c660dc980:

(disk_stall.go:74).Setup: cluster needs ReusePolicyNone to support disk stalls
test artifacts and logs in: /artifacts/admission-control/disk-bandwidth-limiter/cpu_arch=arm64/run_1

Parameters:

  • arch=arm64
  • cloud=aws
  • coverageBuild=false
  • cpu=8
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • metamorphicLeases=default
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Nov 16, 2024
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.admission-control/disk-bandwidth-limiter failed with artifacts on master @ a5ab06046e817627cd0b64390f4ede0c4a5d82ef:

(disk_stall.go:74).Setup: cluster needs ReusePolicyNone to support disk stalls
test artifacts and logs in: /artifacts/admission-control/disk-bandwidth-limiter/run_1

Parameters:

  • arch=amd64
  • cloud=gce
  • coverageBuild=false
  • cpu=8
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • metamorphicLeases=expiration
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.admission-control/disk-bandwidth-limiter failed with artifacts on master @ a5ab06046e817627cd0b64390f4ede0c4a5d82ef:

(disk_stall.go:74).Setup: cluster needs ReusePolicyNone to support disk stalls
test artifacts and logs in: /artifacts/admission-control/disk-bandwidth-limiter/cpu_arch=arm64/run_1

Parameters:

  • arch=arm64
  • cloud=aws
  • coverageBuild=false
  • cpu=8
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • metamorphicLeases=expiration
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.admission-control/disk-bandwidth-limiter failed with artifacts on master @ 9927a9a1f0827daa734d5eb718017cf260dfe676:

(disk_stall.go:74).Setup: cluster needs ReusePolicyNone to support disk stalls
test artifacts and logs in: /artifacts/admission-control/disk-bandwidth-limiter/run_1

Parameters:

  • arch=amd64
  • cloud=gce
  • coverageBuild=false
  • cpu=8
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • metamorphicLeases=default
  • runtimeAssertionsBuild=false
  • ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@pav-kv pav-kv mentioned this issue Nov 19, 2024
Merged
@craig craig bot closed this as completed in f873ef8 Nov 19, 2024
@github-project-automation github-project-automation bot moved this from Incoming to Done in [Deprecated] Storage Nov 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aadityasondhi aadityasondhi

Labels
A-storage Relating to our storage engine (Pebble) on-disk storage. branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. T-storage Storage Team
Projects
No open projects
[Deprecated] Storage
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

roachtest: fix disk-bandwidth-limiter AC test pav-kv/cockroach
3 participants
@pav-kv @aadityasondhi @cockroach-teamcity