[Preferences] Test ignores preferences that are not explicitly listed as test dependencies.

[carstenbauer]

Came up over at MPI.jl here.

Currently, ] test ignores JULIA_LOAD_PATH. On our cluster, we use the latter to expand the default load path to an extra directory (via JULIA_LOAD_PATH=:/some/global/path) that contains a Project.toml that sets some global preferences (e.g. for MPI.jl). Consequently, the global preferences aren't respected within ] test.

I see two possible ways forward:

1. Make ] test respect JULIA_LOAD_PATH. (Why doesn't it?)
2. Propagate all preferences that are set along the load path to the test environment. (This is along the lines of Pkg.test() drops Preferences #2500 and Improve handling of preferences for sandboxes used in Pkg #2920 which didn't cover the case of global preferences, IIUC.)

(Tested with Julia 1.8.5)

[carstenbauer]

Maybe I'm wrong with my point 2) above. MWE:

Minimal package with the following Project.toml and test/runtest.jl:

name = "mwe"
uuid = "4187ee2f-2d69-45e0-8479-75d21fecdd8d"
authors = ["Carsten Bauer <crstnbr@gmail.com>"]
version = "0.1.0"

[deps]
LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
Preferences = "21216c6a-2e73-6563-6e65-726566657250"

[extras]
LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
Preferences = "21216c6a-2e73-6563-6e65-726566657250"
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

[targets]
test = ["Test", "LinearAlgebra", "Preferences"]

@show LOAD_PATH
using Preferences, LinearAlgebra
@show Preferences.load_preference(LinearAlgebra, "testpref")

Now, an extra Project.toml in some global path (say /some/global/path) with the following content:

[extras]
LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"

[preferences.LinearAlgebra]
testpref = 1

Running JULIA_LOAD_PATH=:/some/global/path julia --project and then ] test I get

LOAD_PATH = ["@", "/var/folders/qk/nm34_hqn7_7g8fvxgpg24gm40000gn/T/jl_CemQ7p"]
Preferences.load_preference(LinearAlgebra, "testpref") = 1

So, despite the fact that /some/global/path isn't in LOAD_PATH, the preference seems to propagate fine!

(Why didn't it for MPIPreferences than?)

[vchuravy]

breadcrumbs #3061 and JuliaParallel/MPI.jl#561

cc: @staticfloat

[vchuravy]

What happens if you add MPIPreferences to the test dependencies?

[carstenbauer]

What happens if you add MPIPreferences to the test dependencies?

That works!

So I take it that only preferences of direct test dependencies are forwarded?

[vchuravy]

Hm still smells like a bug, we use Base.get_preferences to collect the Preferences to propagate, and while #3061 added load path, get_preferences might only collect preferences of direct dependencies.

But looking at https://github.com/JuliaLang/julia/blob/b5482c82d486aaa68939871eb1d1bc71bb421096/base/loading.jl#L2608 I don't see how or why that would happen/

[vchuravy]

@staticfloat

Pkg.jl/src/Operations.jl

Line 1911 in 63e9558

test_project_preferences = Base.get_preferences()

returns a dictonary of Name=>Prefs but no associated Name=>UUID mapping. We then copy over the preferences

Pkg.jl/src/Operations.jl

Line 1742 in 63e9558

open(tmp_preferences, "w") do io

to the test environment, but preference loading will ignore them since the tmp_project does not contain an appropriate Name=>UUID mapping.