Test: Add fail-fast mechanism

[IanButterworth]

Sometimes it's helpful to get a testset to return from a failure/error early.
This adds the option to fail-fast in two ways:

Via a @testset kwarg failfast

julia> @testset "Foo" failfast=true begin
           @test false
           @test error()
           @testset "Bar" begin
               @test false
               @test true
           end
       end
Foo: Test Failed at REPL[8]:2
  Expression: false
Stacktrace:
 [1] macro expansion
   @ ~/Documents/GitHub/julia/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:466 [inlined]
 [2] macro expansion
   @ REPL[8]:2 [inlined]
 [3] macro expansion
   @ ~/Documents/GitHub/julia/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:1377 [inlined]
 [4] top-level scope
   @ REPL[8]:2

Fail-fast enabled: Fail or Error occured

Test Summary: | Fail  Total  Time
Foo           |    1      1  0.0s
ERROR: Some tests did not pass: 0 passed, 1 failed, 0 errored, 0 broken.

Via an env var JULIA_TEST_FAILFAST, so that CI can be easily switched to fail-fast

julia> ENV["JULIA_TEST_FAILFAST"] = true
true

julia> @testset "Foo" begin
           @test false
           @test error()
           @testset "Bar" begin
               @test false
               @test true
           end
       end
Foo: Test Failed at REPL[10]:2
  Expression: false
Stacktrace:
 [1] macro expansion
   @ ~/Documents/GitHub/julia/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:466 [inlined]
 [2] macro expansion
   @ REPL[10]:2 [inlined]
 [3] macro expansion
   @ ~/Documents/GitHub/julia/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:1377 [inlined]
 [4] top-level scope
   @ REPL[10]:2

Fail-fast enabled: Fail or Error occured

Test Summary: | Fail  Total  Time
Foo           |    1      1  0.0s
ERROR: Some tests did not pass: 0 passed, 1 failed, 0 errored, 0 broken.

Without fail-fast the tests are run as per normal

julia> @testset "Foo" begin
           @test false
           @test error()
           @testset "Bar" begin
               @test false
               @test true
           end
       end
Foo: Test Failed at REPL[12]:2
  Expression: false
Stacktrace:
 [1] macro expansion
   @ ~/Documents/GitHub/julia/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:466 [inlined]
 [2] macro expansion
   @ REPL[12]:2 [inlined]
 [3] macro expansion
   @ ~/Documents/GitHub/julia/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:1377 [inlined]
 [4] top-level scope
   @ REPL[12]:2
Foo: Error During Test at REPL[12]:3
  Test threw exception
  Expression: error()
  
  Stacktrace:
   [1] error()
     @ Base ./error.jl:44
   [2] macro expansion
     @ ~/Documents/GitHub/julia/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:466 [inlined]
   [3] macro expansion
     @ REPL[12]:3 [inlined]
   [4] macro expansion
     @ ~/Documents/GitHub/julia/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:1377 [inlined]
   [5] top-level scope
     @ REPL[12]:2
Bar: Test Failed at REPL[12]:5
  Expression: false
Stacktrace:
 [1] macro expansion
   @ ~/Documents/GitHub/julia/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:466 [inlined]
 [2] macro expansion
   @ REPL[12]:5 [inlined]
 [3] macro expansion
   @ ~/Documents/GitHub/julia/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:1377 [inlined]
 [4] macro expansion
   @ REPL[12]:5 [inlined]
 [5] macro expansion
   @ ~/Documents/GitHub/julia/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:1377 [inlined]
 [6] top-level scope
   @ REPL[12]:2
Test Summary: | Pass  Fail  Error  Total  Time
Foo           |    1     2      1      4  0.0s
  Bar         |    1     1             2  0.0s
ERROR: Some tests did not pass: 1 passed, 2 failed, 1 errored, 0 broken.

[IanButterworth]

One use case of this is to help with this..

It's common for packages to do testsets sequentially like this

@testset "1" begin
   ...
end
@testset "2" begin
   ...
end

but that reports each testset individually, meaning it's always verbose and the test reports aren't nicely collated & formatted together, which can be especially awkward in noisy test suites.

So the logical thing to do is to wrap all the testsets into a parent package testset, so that everything is pulled into a single report at the end that is formatted and timed together.

@testset "PackageName" begin
  @testset "1" begin
    ...
  end
  @testset "2" begin
    ...
  end
end

And if all passes, you get a small tidy success report (or the full thing if verbose=true)

Test Summary: | Pass  Total  Time
PackageName   |    2      2  0.0s

However, while the first setup fails fast as soon as one of the testsets fails, this new setup won't and will run all testsets.

So you're stuck either with:

1. fail fast, but no collated report
2. a collated report, but no fail fast

This option allows you to do the following, which gives you a tidy report at the end, but fails fast.

@testset "PackageName" failfast=true begin
  @testset "1" begin
    ...
  end
  @testset "2" begin
    ...
  end
end

[jakobnissen]

This is great! A big headache when running tests in a loop (e.g. to cover combinations of options) is that if the test fails, it may report hundreds of nearly identical errors.

[IanButterworth]

@aviatesk thanks for the review. Do you think this is good to go? My finger's been hovering over merge but a second opinion would be appreciated

[aviatesk]

Yes, looks good to go!