Functions behaves strangely different inside and outside @testset

[KristofferC]

MWE from PkgEval error in ExactPredicates.jl (https://github.com/maleadt/BasePkgEvalReports/blob/0bb12607c245153c8e7215788e580d43d7d2d51a/pkgeval-9bd498d_vs_2e6715c/logs/ExactPredicates/1.4.0-DEV-082f446f52.log#L64-L81)

Running

using Test
using ExactPredicates
p2(x, y=0.0) = (float(x), float(y))
p3(x, y=0, z=0) = (float(x), float(y), float(z))

and then running the code below twice(!!)

@testset "pertubations" begin
    a = p2(2.0)
    b = p2(1.0, 1.0)
    c = p2(1.0, - 1.0)
    q = p2(prevfloat(0.0))
    @test incircle(a, b, c, q) == -1
end

makes it fail with

pertubations: Test Failed at REPL[23]:6
  Expression: incircle(a, b, c, q) == -1
   Evaluated: 0 == -1

If we move the definitions of a, b, c, q outside the @testset, the @test never fails even after executing it many times.

The package does some pretty heavy metaprogramming https://github.com/lairez/ExactPredicates.jl/blob/216010eb619e71b51f24020e71902382411d0d81/src/Codegen.jl#L371-L478 but not sure if that is relevant.

This doesn't happen on 1.3.1.

[JeffBezanson]

Interestingly, this is argument value sensitive:

julia> @testset "pertubations" begin
           a = p2(2.0)
           b = p2(1.0, 1.0)
           c = p2(1.0, - 1.0)
           q = (-eps(Float64)/2, 0.0)
           x = incircle(a,b,c,q)
           @show x
        end
x = 0
Test Summary: |
pertubations  | No tests
Test.DefaultTestSet("pertubations", Any[], 0, false)

julia> @testset "pertubations" begin
           a = p2(2.0)
           b = p2(1.0, 1.0)
           c = p2(1.0, - 1.0)
           q = (-nextfloat(eps(Float64)/2), 0.0)
           x = incircle(a,b,c,q)
           @show x
        end
x = -1
Test Summary: |
pertubations  | No tests
Test.DefaultTestSet("pertubations", Any[], 0, false)

It fails for anything with magnitude eps/2 or smaller.

[JeffBezanson]

One global state change: this case needs to call setrounding(Interval, ...), so after the first run IntervalArithmetic is in a different rounding mode. Setting the rounding mode involves eval'ing a bunch of method definitions (as well as calling mktemp() ... ?!). I still don't know exactly why that causes this failure, but there is definitely a lot going on.

[JeffBezanson]

This seems likely to be an instance of #24569, in which case we might not be able to do anything about it.