testing: run Cleanup if SIGINT occurs

[katiehockman]

Currently, testing.T.Cleanup and testing.B.Cleanup are only run if the test and all of its subtests have finished running. This means it won't run if the test is stopped with a SIGINT (e.g. Ctrl+C). There are several things that may be in such a Cleanup function that would be helpful to run regardless of whether or not the test fully completed.

This will be especially unfortunate in the case of fuzzing (http://golang.org/s/draft-fuzzing-design). Take for example:

func FuzzFoo(f *testing.F) {
	f.Cleanup(func() {
		// Check for things like stray go routines, etc
		// Delete tmp resources that may be left behind
	})
	f.Fuzz(func(t *testing.T, b []byte) {
		// some code being fuzzed
	})
}

It may be a very common use case that someone could run go test -fuzz FuzzFoo and give a SIGINT after a few hours. If no crash occurs, there would be no way to run any cleanup code or do any checks after fuzzing completed since the fuzzing runs in an infinite loop.

I would like to expand the behavior of Cleanup to not only run if tests finished, but also run in the case of a SIGINT. This could of course be special cased to fuzzing, but I can see good reasons why it should be a part of unit and benchmark testing as well (e.g. if the test hangs for some reason).

/cc @jayconrod @bcmills @FiloSottile

[katiehockman]

A note on possible complexity from @jayconrod: "One problem is that we can't handle SIGINT on Windows. It looks like there a couple similar things (CTRL_CLOSE_EVENT). Not sure if we can hook into that without cgo."

[jayconrod]

Not sure whether we should handle SIGINT in all tests and benchmarks. User code may install signal handlers (for example, in an initializer in a cgo library), and we might accidentally replace those.

It makes more sense for fuzzing, specifically when running the fuzzing engine in worker processes, since 1) we don't need to be compatible with existing fuzz targets because there are none, and 2) it will be common for users to stop fuzzing with ^C.

[bcmills]

Can the test function itself register the SIGINT handler, rather than having the testing package do that?

For example, something like: https://play.golang.org/p/ZsRDGgDMgGz

[jayconrod]

We were looking for something more general purpose with fuzzing. The fuzzing coordinator process (started by go test) should handle SIGINT and should flush pending updates to the corpus. It should also stop its worker processes.

[bcmills]

I don't think Cleanup is appropriate for that without coordination from the test itself, though.

The Cleanup function can normally assume that it does not race with any subtests, and can also assume that it is only called after the subtests are complete (compare #40908). So it seems to me that if you want to run Cleanup functions on SIGINT, you fundamentally have to thread that signal down to the test functions and have them cooperatively complete.

[ianlancetaylor]

Note that the last time we tried to handle SIGINT in the testing package, for #19397, we had to roll back the change because it broke tests that were testing code that itself handled SIGINT. See #21000.

[dprotaso]

Related #39382

[dprotaso]

Knative have written some test-infra code that deals with handling interrupts but it's not ideal to maintain.

Generally

• calls to our custom 'Cleanup' function registers Cleanup with 'testing.T` and our own interrupt handler
• After handling the interrupt you can't cleanly stop test execution - we end up calling os.Exit(1) after the cleanup finishes
• you actually need to wrap the clean up funcs in a sync.Once since there could be race between a test cleaning up and a signal being handled. You could end up with double invocations

I also noticed timeouts don't cancel tests gracefully either - #42217

[odeke-em]

Thanks everyone for the input here, @ianlancetaylor raised the cautionary tale and precedent of us previously having tried to handle SIGINT in testing before, and then rollbacked, and @bcmills showed how the caller could invoke it and handle directly. Perhaps the fuzzing package could handle it somehow or if the testing package could detect that fuzzing is being done that it could add hooks. Either way, we haven’t resolved this issue for Go1.16, and I shall kindly implore us to try again for Go1.17. Moving to Go1.17. Thank you.