roachtest: disk-stalled/fuse/log=false,data=false failed

[cockroach-teamcity]

roachtest.disk-stalled/fuse/log=false,data=false failed with artifacts on release-22.2 @ fc7ffbb5b0d14a2fbf62bba29090afc6b70fd6f7:

test artifacts and logs in: /artifacts/disk-stalled/fuse/log=false_data=false/run_1
(ts_util.go:51).mustGetMetrics: Post "http://34.74.5.153:26258/ts/query": dial tcp 34.74.5.153:26258: connect: connection refused
(cluster.go:1977).Run: cluster.RunE: context canceled
(cluster.go:1977).Run: cluster.RunE: context canceled
(cluster.go:1977).Run: output in run_093321.807248688_n4_cockroach-workload-r: ./cockroach workload run kv --read-percent 50 --duration 10m --concurrency 256 --max-rate 2048 --tolerate-errors  --min-block-bytes=512 --max-block-bytes=512 {pgurl:1-3} returned: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

• roachtest: disk-stalled/fuse/log=false,data=false failed #99372 roachtest: disk-stalled/fuse/log=false,data=false failed [C-test-failure O-roachtest O-robot T-storage branch-master]

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

Jira issue: CRDB-26074

Epic CRDB-20293

[nicktrav]

Looks like this was a disk stall on n2.

teamcity-9283345-1679980651-46-n4cpu4-0002> F230328 09:40:31.984774 286168 storage/pebble.go:1055 ⋮ [n2] 620 disk stall detected: pebble unable to write to ‹×› in 12.00 seconds
stack trace:
goroutine 286168 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0x1)
	github.com/cockroachdb/cockroach/pkg/util/log/get_stacks.go:25 +0x89
github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(0xc0008b9680, {{{0xc0006a5ec0, 0x24}, {0x53d52df, 0x1}, {0x0, 0x0}, {0x0, 0x0}}, 0x17508bda8d18ea70, ...})
	github.com/cockroachdb/cockroach/pkg/util/log/clog.go:262 +0x97
github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x6228748, 0xc00086de60}, 0x2, 0x4, 0x0, 0x0?, {0x54b6aec, 0x41}, {0xc002bebe40, 0x2, ...})
	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:106 +0x645
github.com/cockroachdb/cockroach/pkg/util/log.logfDepth(...)
	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:39
github.com/cockroachdb/cockroach/pkg/util/log.Fatalf(...)
	github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/util/log/log_channels_generated.go:848
github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).makeMetricEtcEventListener.func3({{0xc00b7447e0?, 0xc003ddef74?}, 0xc005e47860?})
	github.com/cockroachdb/cockroach/pkg/storage/pebble.go:1055 +0x271
github.com/cockroachdb/pebble.TeeEventListener.func4({{0xc00b7447e0?, 0xb891994?}, 0xedbb4ad0c?})
	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/event.go:633 +0x3e
github.com/cockroachdb/cockroach/pkg/storage.wrapFilesystemMiddleware.func1({0xc00b7447e0?, 0x9c960e0?}, 0x2cb06b24f?)
	github.com/cockroachdb/cockroach/pkg/storage/pebble.go:598 +0x26
github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).Create.func2(0xc100cb630b891994?)
	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:430 +0x2f
github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:85 +0x211
created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker
	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:66 +0x5d

#99869 should hopefully help here.

Closing this out, and marking as an infra flake. Linking to #97968.