Feature request: manually run precompilation code during testing

[MilesCranmer]

It would be great if there was a way I could manually trigger the code within @setup_workload to run, outside of precompilation. The primary use-case for this would be to run the precompilation code during testing so it shows up on code coverage, and so that some precompilation errors show up as errors during testing.

My current workaround is to call a function containing all my precompilation code, with mode=:precompile -> @setup_workload, and mode=:compile -> (run as-is). https://github.com/MilesCranmer/SymbolicRegression.jl/blob/641180a561e350918e3fd6a8fd7c898d31bef11b/src/precompile.jl#L34

macro maybe_compile_workload(mode, ex)
    precompile_ex = Expr(
        :macrocall, Symbol("@compile_workload"), LineNumberNode(@__LINE__), ex
    )
    return quote
        if $(esc(mode)) == :compile
            $(esc(ex))
        elseif $(esc(mode)) == :precompile
            $(esc(precompile_ex))
        else
            error("Invalid value for mode: " * show($(esc(mode))))
        end
    end
end

with the function:

function do_precompilation(; mode=:precompile)
    @maybe_setup_workload mode begin
    ...
end

so I can have do_precompilation() in my precompilation code, and do_precompilation(; mode=:compile) in my testing code.

However this seems like it would be of common interest so it would be great if there could be a way I could force @compile_workload code to execute outside of precompilation.

---

Brought up in MilesCranmer/SymbolicRegression.jl#198 w/ @timholy

[timholy]

Another application (though not with this implementation) would be to easily test how well it "works" by re-running the same code used during precompilation.

[MilesCranmer]

I don’t know the internals, so this could be completely off base. But what if @setup_workload overloaded a function of PrecompileTools.jl with the code passed to the macro, in addition to the normal precompilation generation?

For example, what if:

@setup_workload testgen=true begin
    context = setup_my_workload()

    @compile_workload begin
        run_my_workload(context)
    end
end

Expanded to the following code?

import PrecompileTools: _test_precompilation

function _test_precompilation(::Val{:MyPackageName})
    context = setup_my_workload()
    begin
        run_my_workload(context)
    end
end

# (regular behavior of macro here)

Then, you could test this by calling:

using MyPackageName, PrecompileTools

test_precompilation(MyPackageName) # test_precompilation(...) -> _test_precompilation(Val(...)) internally

which, given that you’ve imported the package, will execute the code within the overloaded function. And if needed you could test the precompilation of any other loaded packages too.

---

Edit: added suggested testgen=true.

[MilesCranmer]

One orthogonal idea: you would want to have a keyword argument to @setup_workflow to explicitly turn this (or some other testing mechanism) on. For example:

@setup_workload testgen=true begin

end