Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split leaky testset into separate group #1024

Merged
merged 1 commit into from
Feb 29, 2024

Conversation

ArnoStrouwen
Copy link
Member

No description provided.

@ArnoStrouwen
Copy link
Member Author

The error in @time @safetestset "Fully Out of Place adjoint sensitivity" include("adjoint_oop.jl"), does not appear when the other safetestsets are commented out. First, I thought this could be attributed to the error only appearing randomly. But this behavior seems consistent over many tries on my machine. The safe test sets might thus not be totally safe?
Another very strange bug @ChrisRackauckas.

@ChrisRackauckas
Copy link
Member

Given the timing, I think it may be JuliaLang/julia#52635. Make it a separate test set.

@ArnoStrouwen
Copy link
Member Author

reproducible steps:

julia> using TestEnv; using Pkg; Pkg.activate("."); TestEnv.activate()
  Activating project at `~/SciML/SciMLSensitivity.jl`
┌ Warning: Could not use exact versions of packages in manifest, re-resolving
└ @ TestEnv ~/.julia/packages/TestEnv/shkbW/src/julia-1.9/activate_set.jl:63
"/tmp/jl_xDgmEZ/Project.toml"

julia> cd("./test")

julia> using Test

julia> using SafeTestsets

julia> include("./adjoint_oop.jl")
Test Passed
julia> using TestEnv; using Pkg; Pkg.activate("."); TestEnv.activate()
  Activating project at `~/SciML/SciMLSensitivity.jl`
┌ Warning: Could not use exact versions of packages in manifest, re-resolving
└ @ TestEnv ~/.julia/packages/TestEnv/shkbW/src/julia-1.9/activate_set.jl:63
"/tmp/jl_oHggLP/Project.toml"

julia> cd("./test")

julia> using Test

julia> using SafeTestsets

julia> include("./automatic_sensealg_choice.jl")
Precompiling Lux
  15 dependencies successfully precompiled in 17 seconds. 100 already precompiled.
Precompiling ComponentArrays
  1 dependency successfully precompiled in 1 seconds. 64 already precompiled.
Precompiling LuxComponentArraysExt
  6 dependencies successfully precompiled in 7 seconds. 123 already precompiled.
Precompiling ComponentArraysRecursiveArrayToolsExt
  5 dependencies successfully precompiled in 4 seconds. 133 already precompiled.
Precompiling ComponentArraysSciMLBaseExt
  1 dependency successfully precompiled in 1 seconds. 132 already precompiled.
Precompiling LuxComponentArraysReverseDiffExt
  1 dependency successfully precompiled in 2 seconds. 128 already precompiled.
┌ Warning: Using fallback BLAS replacements for (["ssymv_64_"]), performance may be degraded
└ @ Enzyme.Compiler ~/.julia/packages/GPUCompiler/U36Ed/src/utils.jl:59
true

julia> include("./adjoint_oop.jl")
ERROR: LoadError: Need an adjoint for constructor QuadratureAdjoint{0, true, Val{:central}, ZygoteVJP}. Gradient is of type ChainRulesCore.ZeroTangent
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] (::Zygote.Jnew{QuadratureAdjoint{0, true, Val{:central}, ZygoteVJP}, Nothing, false})(Δ::ChainRulesCore.ZeroTangent)
    @ Zygote ~/.julia/packages/Zygote/jxHJc/src/lib/lib.jl:330
  [3] (::Zygote.var"#2210#back#313"{Zygote.Jnew{QuadratureAdjoint{0, true, Val{:central}, ZygoteVJP}, Nothing, false}})(Δ::ChainRulesCore.ZeroTangent)
    @ Zygote ~/.julia/packages/ZygoteRules/M4xmc/src/adjoint.jl:72
  [4] QuadratureAdjoint
    @ ~/SciML/SciMLSensitivity.jl/src/sensitivity_algorithms.jl:475 [inlined]
  [5] (::Zygote.Pullback{Tuple{Type{QuadratureAdjoint{0, true, Val{:central}, ZygoteVJP}}, ZygoteVJP, Float64, Float64}, Any})(Δ::ChainRulesCore.ZeroTangent)
    @ Zygote ~/.julia/packages/Zygote/jxHJc/src/compiler/interface2.jl:0
  [6] #QuadratureAdjoint#6
    @ ~/SciML/SciMLSensitivity.jl/src/sensitivity_algorithms.jl:483 [inlined]
  [7] (::Zygote.Pullback{Tuple{SciMLSensitivity.var"##QuadratureAdjoint#6", Int64, Bool, Type{…}, ZygoteVJP, Float64, Float64, Type{…}}, Tuple{Zygote.Pullback{…}, Zygote.ZBack{…}}})(Δ::ChainRulesCore.ZeroTangent)
    @ Zygote ~/.julia/packages/Zygote/jxHJc/src/compiler/interface2.jl:0
  [8] QuadratureAdjoint
    @ ~/SciML/SciMLSensitivity.jl/src/sensitivity_algorithms.jl:479 [inlined]
  [9] (::Zygote.Pullback{Tuple{typeof(Core.kwcall), @NamedTuple{abstol::Float64, reltol::Float64, autojacvec::ZygoteVJP}, Type{QuadratureAdjoint}}, Any})(Δ::ChainRulesCore.ZeroTangent)
    @ Zygote ~/.julia/packages/Zygote/jxHJc/src/compiler/interface2.jl:0
 [10] #6
    @ ~/SciML/SciMLSensitivity.jl/test/adjoint_oop.jl:185 [inlined]
 [11] (::Zygote.Pullback{Tuple{…}, Tuple{…}})(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/jxHJc/src/compiler/interface2.jl:0
 [12] (::Zygote.var"#75#76"{Zygote.Pullback{Tuple{}, Tuple{}}})(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/jxHJc/src/compiler/interface.jl:91
 [13] gradient(::Function, ::SVector{2, Float64}, ::Vararg{Any})
    @ Zygote ~/.julia/packages/Zygote/jxHJc/src/compiler/interface.jl:148
 [14] top-level scope
    @ ~/SciML/SciMLSensitivity.jl/test/adjoint_oop.jl:184
 [15] include(fname::String)
    @ Base.MainInclude ./client.jl:489
 [16] top-level scope
    @ REPL[7]:1
in expression starting at /home/arno/SciML/SciMLSensitivity.jl/test/adjoint_oop.jl:184
Some type information was truncated. Use `show(err)` to see complete types.

@ArnoStrouwen
Copy link
Member Author

Reduced to adding using Lux, @avik-pal any idea?

julia> using TestEnv; using Pkg; Pkg.activate("."); TestEnv.activate()
  Activating project at `~/SciML/SciMLSensitivity.jl`
┌ Warning: Could not use exact versions of packages in manifest, re-resolving
└ @ TestEnv ~/.julia/packages/TestEnv/shkbW/src/julia-1.9/activate_set.jl:63
"/tmp/jl_uIW71Q/Project.toml"

julia> using Test

julia> using SafeTestsets

julia> using Lux

julia> cd("./test")

julia> include("adjoint_oop.jl")
ERROR: LoadError: Need an adjoint for constructor QuadratureAdjoint{0, true, Val{:central}, ZygoteVJP}. Gradient is of type ChainRulesCore.ZeroTangent
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] (::Zygote.Jnew{QuadratureAdjoint{0, true, Val{:central}, ZygoteVJP}, Nothing, false})(Δ::ChainRulesCore.ZeroTangent)
    @ Zygote ~/.julia/packages/Zygote/jxHJc/src/lib/lib.jl:330
  [3] (::Zygote.var"#2210#back#313"{Zygote.Jnew{QuadratureAdjoint{0, true, Val{:central}, ZygoteVJP}, Nothing, false}})(Δ::ChainRulesCore.ZeroTangent)
    @ Zygote ~/.julia/packages/ZygoteRules/M4xmc/src/adjoint.jl:72
  [4] QuadratureAdjoint
    @ ~/SciML/SciMLSensitivity.jl/src/sensitivity_algorithms.jl:475 [inlined]
  [5] (::Zygote.Pullback{Tuple{Type{QuadratureAdjoint{0, true, Val{:central}, ZygoteVJP}}, ZygoteVJP, Float64, Float64}, Any})(Δ::ChainRulesCore.ZeroTangent)
    @ Zygote ~/.julia/packages/Zygote/jxHJc/src/compiler/interface2.jl:0
  [6] #QuadratureAdjoint#6
    @ ~/SciML/SciMLSensitivity.jl/src/sensitivity_algorithms.jl:483 [inlined]
  [7] (::Zygote.Pullback{Tuple{SciMLSensitivity.var"##QuadratureAdjoint#6", Int64, Bool, Type{…}, ZygoteVJP, Float64, Float64, Type{…}}, Tuple{Zygote.Pullback{…}, Zygote.ZBack{…}}})(Δ::ChainRulesCore.ZeroTangent)
    @ Zygote ~/.julia/packages/Zygote/jxHJc/src/compiler/interface2.jl:0
  [8] QuadratureAdjoint
    @ ~/SciML/SciMLSensitivity.jl/src/sensitivity_algorithms.jl:479 [inlined]
  [9] (::Zygote.Pullback{Tuple{typeof(Core.kwcall), @NamedTuple{abstol::Float64, reltol::Float64, autojacvec::ZygoteVJP}, Type{QuadratureAdjoint}}, Any})(Δ::ChainRulesCore.ZeroTangent)
    @ Zygote ~/.julia/packages/Zygote/jxHJc/src/compiler/interface2.jl:0
 [10] #6
    @ ~/SciML/SciMLSensitivity.jl/test/adjoint_oop.jl:185 [inlined]
 [11] (::Zygote.Pullback{Tuple{…}, Tuple{…}})(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/jxHJc/src/compiler/interface2.jl:0
 [12] (::Zygote.var"#75#76"{Zygote.Pullback{Tuple{}, Tuple{}}})(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/jxHJc/src/compiler/interface.jl:91
 [13] gradient(::Function, ::SVector{2, Float64}, ::Vararg{Any})
    @ Zygote ~/.julia/packages/Zygote/jxHJc/src/compiler/interface.jl:148
 [14] top-level scope
    @ ~/SciML/SciMLSensitivity.jl/test/adjoint_oop.jl:184
 [15] include(fname::String)
    @ Base.MainInclude ./client.jl:489
 [16] top-level scope
    @ REPL[7]:1
in expression starting at /home/arno/SciML/SciMLSensitivity.jl/test/adjoint_oop.jl:184
Some type information was truncated. Use `show(err)` to see complete types.

@avik-pal
Copy link
Member

IJust to be sure that I understand the problem, if you do using Lux the test starts to fail?

@ArnoStrouwen ArnoStrouwen marked this pull request as ready for review February 29, 2024 13:49
@ArnoStrouwen ArnoStrouwen changed the title [DO NOT MERGE] INVESTIGATING GROUP 3 CI ERRORS Split leaky testset into separate group Feb 29, 2024
@ArnoStrouwen
Copy link
Member Author

Yes that is true, but it is worse than that since the using Lux can also be in a different safetestset.

@ArnoStrouwen
Copy link
Member Author

Copy link

codecov bot commented Feb 29, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 58.17%. Comparing base (f5b80ec) to head (ed0c3ad).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1024      +/-   ##
==========================================
+ Coverage   56.27%   58.17%   +1.90%     
==========================================
  Files          19       19              
  Lines        4519     4519              
==========================================
+ Hits         2543     2629      +86     
+ Misses       1976     1890      -86     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@ChrisRackauckas ChrisRackauckas merged commit ecc90c4 into SciML:master Feb 29, 2024
17 of 18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants