testing: add (*T).Deadline

[bcmills]

Background

#16221 added a func (*T) Context() context.Context to the testing package, which was backed out due to concerns from @dsnet, @neild, @niemeyer, me, and perhaps others as well (#18199).

CL 134395 (errgroup propagation of Goexit, among other things) mitigates those concerns somewhat: it gives test authors a straightforward way to start a collection of goroutines and wait for them to finish, and cancels them if any of the goroutines calls t.FailNow or t.SkipNow (#15758 notwithstanding).

That leaves one use-case unaddressed: as @neild noted in the golang-dev thread and @dsnet noted in #18199 (comment), if a test times out (e.g. because of a deadlock in the package under test), it is often useful to shut it down cleanly and emit useful logs rather than wait for the testing package to dump all of the running goroutines.

@bradfitz suggested that test authors can add their own derived context, but a test in general has no idea how much time it has left: if we want to check the -test.timeout flag, we need to be able to subtract off the amount of time spent so far, and that implies the use of TestMain and the boilerplate that goes with it.

---

Proposal

I propose that we add back the (*testing.T).Context method, with the same signature as before but the following semantics:

• When a test binary times out:
  • First, all tests that are still running are marked as having failed due to timing out.
  • Then, the contexts provided to tests are marked as Done, with Err returning context.DeadlineExceeded.
  • Finally, the test binary waits for some “reasonable” (but unspecified) grace period elapses, or until all tests that called their Context method (and their subtests) have returned, whichever occurs first.
  • If any tests are still running at that point, the test binary writes out the logs for any completed tests and then panics as before.
• When a test is marked as Failed or Skipped:
  • The context provided to that test (if any) is marked as Done, with Err returning context.Canceled.
• When a test function exits normally:
  • The context provided to that test is not marked as Done. That discourages the use of (*T).Context for normal cleanup of “background” goroutines, which should be accomplished by some other means (sync.WaitGroup, errgroup, tomb, or whatever alternative the test author prefers.)

[neild]

Another option would be to add a t.Deadline and leave context creation up to the test.

[bcmills]

t.Deadline would be somewhat more annoying to use, but acceptable for my use-cases.

[deanveloper]

The way I see it, current testing is very simple and I like it that way. Adding a t.Deadline I think is a good idea, however. But contexts are big constructs that can make things complicated quick, especially when looking at other people's code. Probably better to just have a t.Deadline() since that is much simpler.

[rsc]

The last attempt at adding Context was trying to signal "this test is done, all you helpers can go away".
The new attempt is trying to signal - using exactly the same mechanism! - "this test is about to be killed, you might want to die with a more useful message".

I don't see why we should prefer one over the other. They both seem very indirect. The fact that there were these two very different plausible uses suggests that having plain t.Context will be confusing to people expecting one or the other (or a third equally plausible meaning).

If the goal is to expose timeout information, let's focus on an API that does that well, instead of shoehorning it into context.

[andybons]

@rsc are you OK with t.Deadline?

Per @golang/proposal-review

[Capstan]

This appears to be related, if not a resurrection of #18368 which was FrozenDueToAge. I'm adding this comment here to act as a trail of breadcrumbs, hoping that the link will show up in the older proposal.

[rsc]

Sure, let's try adding t.Deadline for Go 1.13.