proposal: testing: provide a way to more easily test test utility

[jhenstridge]

For larger Go projects, it's common to write some utility functions or types for use in it's test suite. Ideally, those utilities should themselves be tested. However, there are certain types of behaviour that are difficult to test at present:

1. Check that a function causes a test to fail.
2. Check that a function logs a particular message.
3. Check that a function spawns an expected set of subtests.

With most testing frameworks, the usual method to handle this would be to invoke the test runner with a custom set of tests, and then inspect the results. That's not really possible with Go's testing framework, since the method of constructing a test suite and invoking the runner is all private API (either inaccessible like testing/internal/testdeps, or documented to be unstable like testing.TestMain). Also, the runner depends on global state through flags that would be shared with the real test suite.

[mvdan]

A proposal is generally a very specific idea. This seems more like a vague problem statement, and I'm not understanding what solution you're proposing.

[jhenstridge]

I guess I tagged it wrong. I was trying to describe what features I wanted without locking down an implementation.

I guess a more concrete proposal would be something like:

1. Add a publicly supported version of testing.MainStart that doesn't depend on the undocumented testDeps interface, and takes some kind of config argument to use in lieu of the command line flags (i.e. we don't want the -run command line argument to restrict what tests are run in this independent test runner).

2. Add a method to testing.M to set where to send test output instead of stdout. This could be a simple io.Writer accepting the default test output format, or more usefully something accepting the -json style TestEvent structures.

[davecheney]

@jhenstridge would a better title for this issue be “provide a way to test Testmain functions” ?

[ianlancetaylor]

Perhaps related to #40984.

[jhenstridge]

@davecheney: the scope is a bit more than what a TestMain function currently does. It's the stuff that happens before TestMain is called, and the result reporting within M.Run() that I'd like access to.

The concrete use case I have for this is go-check/check#122. I was trying to port go-check to use the stdlib test runner. While I was easily able to port the code over to using stdlib subtests, porting the tests has proved more difficult. These tests took the form of tests that create a new test runner that runs a collection of tests not part of the main test suite, and then inspecting the results. That kind of thing is all private in the stdlib testing framework.

@ianlancetaylor: the suggestion in that bug report of using cleanup functions gets close to the first use case from my original post, with something like:

func TestUtility(t *testing.T) {
    t.Cleanup(func() {
        if (!t.Faled()) {
            t.Error("utility function did not fail the test")
            return
        }
        // XXX: mark test as successful somehow?
    })
    failTheTest(t)
}

This would be enough to make the test fail if failTheTest doesn't set the failed state, but it also reports a failure if things work correctly. Perhaps some function doing the opposite of T.Fail could help in this specific case.

I'm not sure it would help out with the other two use cases I mentioned, where it is not clear how you'd check they worked directly from a test calling them.

[bcmills]

@jhenstridge, note that a test can invoke its own executable as a subprocess using the os/exec package. In many cases that suffices for testing test helpers.

(See also #41980 (comment), #39903 (comment).)

[jhenstridge]

@bcmills: Thanks for the suggestion. I'll see if I can use that to adapt the tests I'm working on. Presumably something like this:

1. Define a test that contains the logic under test that calls t.SkipNow() unless a particular environment variable is set.
2. Run os.Args[0] as a subprocess with the environment variable set and -run to select just the test in question. Maybe also -json if the I need the structured output.

Hopefully the tests I actually want to keep can fit into environment variables + log inspection as the only communication needed.

It would still be nice to have a more direct method of testing functionality like this.

[rsc]

This is not a concrete proposal, and we just spent a lot of time discussing the topic on #40984. In the absence of a concrete proposal and to avoid duplicated effort, closing.

[dnephin]

@jhenstridge I've had success testing test helpers in https://github.com/gotestyourself/gotest.tools so I thought I would share in case it helps you.

For the first two cases in the description I think the common practice is to have test helpers use a smaller interface instead of using testing.T. Once they accept their own interface in place of testing.T a fakeT can pretty easily confirm that Fatal, Error, or Log are called correctly.

"Check that a function spawns an expected set of subtests" is still difficult because T.Run always passes a *testing.T to the subtest. I believe that problem was discussed in #39903.

As long as go-check doesn't expose t.Run directly, it seems like it would be possible to replace it with a testing shim and call the subtest with a fakeT that records the hierarchy.