roachtest: kv95/enc=false/nodes=3/cpu=32/size=64kb/no-admission failed

[cockroach-teamcity]

roachtest.kv95/enc=false/nodes=3/cpu=32/size=64kb/no-admission failed with artifacts on release-22.2 @ 86279b015967e35804ba03036259e77999981d74:

test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/cpu=32/size=64kb/no-admission/run_1
(cluster.go:1977).Run: output in run_134151.926748431_n4_workload-run-kv-tole: ./workload run kv --tolerate-errors --init --histograms=perf/stats.json --concurrency=192 --splits=1000 --duration=30m0s --read-percent=95 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-3} returned: context canceled
(monitor.go:127).Wait: monitor failure: monitor command failure: unexpected node event: 1: dead (exit status 7)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=32 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-25976

[renatolabs]

Error when closing the http listener in server_http.go [1]:

F230326 14:05:47.264579 340 1@server/server_http.go:223 ⋮ [n1] 2399  close tcp ‹[::]:26258›: ‹use of closed network connection›

@cockroachdb/obs-inf-prs please take a look.

[1]

cockroach/pkg/server/server_http.go

Lines 222 to 224 in 86279b0

	if err := httpLn.Close(); err != nil {
	log.Ops.Fatalf(ctx, "%v", err)
	}

[aadityasondhi]

I looked through the logs from the test artifacts where this error is initially logged, and could not find anything suggesting why this happened "use of closed network connection" (suggests that we are trying to close httpLn when it is already closed.

Looking through the code surrounding the block here, the only other place where this http listener is closed is https://github.com/cockroachdb/cmux/blob/master/cmux.go#L157-L159. The error handler by default is a function that always returns true so not really sure how this failure happened https://github.com/cockroachdb/cmux/blob/master/cmux.go#L66.

It also seems like this test has not failed since first reported so could have been flake. I will try to repro myself using roachprod but if I cannot repro and it doesn't fail here again in the next few days, I think we can remove the release blocker tag.

[renatolabs]

We also saw this on master btw (#98523). We should probably backport #98600 to 22.2.

[aadityasondhi]

@dhartunian feel free to close this issue once #98600 is backported to 22.2.

[renatolabs]

Do we want to fix this in time for 22.2.8 (see release announcement email)?

[knz]

I am tempted to say this is not a release blocker. @dhartunian should confirm

[dhartunian]

Not a release blocker. Removed the labels.

[srosenberg]

Fix was successfully backported into 22.2 [1], closing.

[1] #100772