testing: support explicitly running non-hermetic (or non-deterministic) examples

[mvdan]

We can have examples which show up in the godoc but aren't runnable, and runnable examples which are run as part of go test.

What about examples which are too slow or unreliable for go test, for example those needing access to the internet? I can make them non-runnable, but what if a user wants to give one a try? go test -run ExampleSlow doesn't work, and the next best option is copy-pasting into a main func, which is awkward.

The only other option, which is what I tend to do, is to stick examples as main packages elsewhere. But that's also icky; if the example packages aren't hidden, go install ./... will include them, and if they're hidden in another repo or in _examples, they are hard to find.

Another disadvantage of hiding the examples in a separate dir/repo is that go test ./... won't check if they compile fine; one has to remember to check that dir/repo too.

For some context, I'm trying to collapse the https://github.com/chromedp/examples into the main repo, starting with an example_test.go file. The second and third examples are quick, but the first accesses the internet and is slow. I don't want to run it with go test, but I want it to show up in godoc and allow people to run it easily to try it out.

I'm not labelling this as a proposal, because I don't have a specific solution in mind; this is just a description of a problem I have when trying to document packages with non-trivial examples.

/cc @acln0 @mpvl @bcmills @ianlancetaylor

[mvdan]

One change that comes to mind is adding a testing flag, so that go test doesn't run a slow example, but an explicit command like go test -slowexamples -run SlowExample does. That's probably a terrible solution, but it helps illustrate the two alternatives I'd like to have available somehow.

Edit: a slightly nicer flag would be go test -allexamples -run ExampleWithInternet.

[beoran]

You could probably about add a //build:slowexamples tag to those slow examples and then run them with go test -tag slowexamples -run "ExampleSlow.*". Would that solve your problem?

[mvdan]

No; hiding the examples behind a build tag has the same issues as hiding them in an _examples package. They wouldn't show up in the godoc, and go test ./... wouldn't error if they can't build properly.

[bcmills]

What about examples which are too slow or unreliable for go test, for example those needing access to the internet?

See previously the discussion on #30595, particularly #30595 (comment).

Unfortunately, unlike tests, runnable examples do not have a Short method to interrogate, nor a Skip method they can use to abort the example if it cannot be run. (They either have an Output line and always run, or they lack an Output line and always merely build.)

[mvdan]

Thanks. I did see that thread, and I agree with Ian's response there. I don't think the same applies here; tests and examples are essentially different things. Generally, the users will be interested in the examples and will want to see and run them, regardless of whether they qualify as "slow" or "integration" examples.

[bcmills]

I think there is a big difference between “see” and “run”.

I would argue that:

• go doc should show all examples in the documentation, regardless of their speed or requirements.
• go test should (by default) test only the subset of examples that are hermetic, leaving it up to users to explicitly enable the non-hermetic tests (by setting an environment variable or build tag).
• go test -short should test only the subset of examples that are both hermetic and fast.

The connection to Short and Skip is that we currently have no way to mark an example as “slow”, and no way to check only compilation for examples that should be runnable but not run by default.

[mvdan]

What you said makes sense to me, for the most part. The only note I'd add is that I don't care about the speed of examples per se; I've rarely encountered an example which took a lot of resources or time. I should have probably titled this issue with "non-hermetic examples" instead of "slow examples". Or perhaps even "non-deterministic examples".

[thepudds]

I should have probably titled this issue with "non-hermetic examples" instead of "slow examples". Or perhaps even "non-deterministic examples".

If that is the focus, then there might be some relationship to this comment from Russ in #23799 (comment)

It would be OK to have a 'mark this as depending on external things' as long as that result was still cached by default, and then we could also add a 'rerun all the tests that depend on external things' as an explicit flag. Not sure what that would look like exactly.

Leaving for Go 1.13.

[mvdan]

Ok, I've retitled the issue to what I think is a better summary after the initial discussion.