net/http: Various tests flaky on Dragonfly builder

[timdarbydotnet]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

1.10

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

DragonflyBSD amd64

What did you do?

The Dragonfly builder very sporadically fails on a few different tests in net/http. This is a recent example:
https://build.golang.org/log/25f7cc4010ecf00d0bdb9afb5aa6471ef12e7c9b

[ianlancetaylor]

Some more examples:

2018-01-23T15:59:15-cafb36b/dragonfly-amd64
2018-02-14T15:51:46-eab06e6/dragonfly-amd64
2018-04-01T06:55:54-fdf3373/dragonfly-amd64
2018-04-05T23:14:14-3f483e6/dragonfly-amd64
2018-04-19T20:33:49-1c439e6/dragonfly-amd64
2018-05-31T18:21:53-462c182/dragonfly-amd64
2018-06-07T14:58:07-cf901a7/dragonfly-amd64

[timdarbydotnet]

example of a failure on TestRetryRequestsOnError:
https://build.golang.org/log/acf2959a136589575226bb5b6345d7fd904db592

[timdarbydotnet]

I don't know if this is related or not, but I saw this error today:

builder: dragonfly-amd64
rev: 72ce047
buildlet: http://dragonflybsd-amd64 reverse peer dragonflybsd-amd64/10.240.0.4:51023 for host type host-dragonfly-amd64-tdfbsd
started: 2018-06-13 16:23:26.554450402 +0000 UTC m=+495931.005849026
ended: 2018-06-13 16:35:46.033695842 +0000 UTC m=+496670.485094427
success: false

Events:
2018-06-13T16:23:26Z checking_for_snapshot
2018-06-13T16:23:26Z finish_checking_for_snapshot after 0s
2018-06-13T16:23:26Z get_buildlet
2018-06-13T16:23:26Z wait_static_builder host-dragonfly-amd64-tdfbsd
2018-06-13T16:23:26Z waiting_machine_in_use
2018-06-13T16:34:46Z finish_wait_static_builder after 11m19.2s; host-dragonfly-amd64-tdfbsd
2018-06-13T16:34:46Z clean_buildlet http://dragonflybsd-amd64 reverse peer dragonflybsd-amd64/10.240.0.4:51023 for host type host-dragonfly-amd64-tdfbsd
2018-06-13T16:34:46Z finish_clean_buildlet after 62.7ms; http://dragonflybsd-amd64 reverse peer dragonflybsd-amd64/10.240.0.4:51023 for host type host-dragonfly-amd64-tdfbsd
2018-06-13T16:34:46Z finish_get_buildlet after 11m19.5s
2018-06-13T16:34:46Z using_buildlet dragonflybsd-amd64
2018-06-13T16:34:46Z write_version_tar
2018-06-13T16:34:46Z get_source
2018-06-13T16:34:46Z finish_get_source after 0s
2018-06-13T16:34:46Z write_go_src_tar
2018-06-13T16:35:46Z finish_write_go_src_tar after 59.4s; err=writing tarball from Gerrit: Put http://dragonflybsd-amd64/writetgz?dir=go: net/http: HTTP/1.x transport connection broken: write tcp 10.200.4.4:443->10.240.0.4:51023: use of closed network connection

Build log:

Error: writing tarball from Gerrit: Put http://dragonflybsd-amd64/writetgz?dir=go: net/http: HTTP/1.x transport connection broken: write tcp 10.200.4.4:443->10.240.0.4:51023: use of closed network connection

[bcmills]

The TestIdleConnChannelLeak failure mode appears to have been fixed in 2020.

My experience with the net/http tests is that they tend to have test-specific root causes, so I'm going to close out this “various tests” issue in favor of tracking failures in specific tests as we identify them.

greplogs --dashboard -md -l -e 'FAIL: TestIdleConnChannelLeak'

2020-01-09T19:50:42-10bfe5b/dragonfly-amd64
2019-12-06T01:11:02-a037582/dragonfly-amd64
2019-10-23T18:41:38-7833302/dragonfly-amd64
2019-07-03T14:12:22-1c1e517/dragonfly-amd64