Path failure(s) in preferences system for a monorepo

[timholy]

I have a monorepo with the following structure:

AdmissionSuite
    Admit
    AdmitConfiguration
    AdmissionTargets

AdmissionSuite has no "real" source code on its own (though to avoid Pkg errors I have to add a dummy src/AdmissionSuite.jl with an empty module), its real role is to contain Project.toml and Manifest.toml files with all 3 subdirectories devved. This is an attempt to distribute a coherent collection of unregistered mutually-dependent packages (xref https://discourse.julialang.org/t/having-local-packages-in-a-monorepo-depend-on-one-another/50454/4?u=tim.holy).

AdmitConfiguration uses Preferences.jl and there is a AdmissionSuite/AdmitConfiguration/LocalPreferences.toml file. For debugging purposes, to base/loading.jl I added the following:

diff --git a/base/loading.jl b/base/loading.jl
index 0a2d1b2ad9..68e41b6ce0 100644
--- a/base/loading.jl
+++ b/base/loading.jl
@@ -1741,6 +1741,7 @@ function get_uuid_name(project_toml::String, uuid::UUID)
 end
 
 function collect_preferences(project_toml::String, uuid::UUID)
+    @show project_toml uuid
     # We'll return a list of dicts to be merged
     dicts = Dict{String, Any}[]
 
@@ -1770,6 +1771,7 @@ function collect_preferences(project_toml::String, uuid::UUID)
         end
     end
 
+    @show dicts
     return dicts
 end

and then I get the following:

julia> Revise.track(Base)

(AdmissionSuite) pkg> st
Status `~/.julia/dev/AdmissionSuite/Project.toml`
  [9b29e739] AdmissionTargets v0.1.0 `AdmissionTargets`
  [e9cde6de] Admit v0.1.0 `Admit`
  [19a2e0a6] AdmitConfiguration v0.1.0 `AdmitConfiguration`

julia> using AdmitConfiguration
project_toml = "/home/tim/.julia/environments/v1.8/Project.toml"
uuid = UUID("19a2e0a6-566e-4c00-8888-a0edb19dbca0")
project_toml = "/home/tim/.julia/dev/AdmissionSuite/Project.toml"
uuid = UUID("19a2e0a6-566e-4c00-8888-a0edb19dbca0")
dicts = Dict{String, Any}[]
project_toml = "/home/tim/.julia/environments/v1.8/Project.toml"
uuid = UUID("ae029012-a4dd-5104-9daa-d747884805df")
dicts = Dict{String, Any}[]
project_toml = "/home/tim/.julia/dev/AdmissionSuite/Project.toml"
uuid = UUID("ae029012-a4dd-5104-9daa-d747884805df")
project_toml = "/home/tim/.julia/environments/v1.8/Project.toml"
uuid = UUID("9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a")
project_toml = "/home/tim/.julia/dev/AdmissionSuite/Project.toml"
uuid = UUID("9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a")
project_toml = "/home/tim/.julia/environments/v1.8/Project.toml"
uuid = UUID("21216c6a-2e73-6563-6e65-726566657250")
project_toml = "/home/tim/.julia/dev/AdmissionSuite/Project.toml"
uuid = UUID("21216c6a-2e73-6563-6e65-726566657250")
project_toml = "/home/tim/.julia/environments/v1.8/Project.toml"
uuid = UUID("e1d29d7a-bbdc-5cf2-9ac0-f12de2c33e28")
project_toml = "/home/tim/.julia/dev/AdmissionSuite/Project.toml"
uuid = UUID("e1d29d7a-bbdc-5cf2-9ac0-f12de2c33e28")
here
project_toml = "/home/tim/.julia/environments/v1.8/Project.toml"
uuid = UUID("19a2e0a6-566e-4c00-8888-a0edb19dbca0")
project_toml = "/home/tim/.julia/dev/AdmissionSuite/Project.toml"
uuid = UUID("19a2e0a6-566e-4c00-8888-a0edb19dbca0")
dicts = Dict{String, Any}[]
project_toml = "/home/tim/.julia/environments/v1.8/Project.toml"
uuid = UUID("19a2e0a6-566e-4c00-8888-a0edb19dbca0")
project_toml = "/home/tim/.julia/dev/AdmissionSuite/Project.toml"
uuid = UUID("19a2e0a6-566e-4c00-8888-a0edb19dbca0")
dicts = Dict{String, Any}[]
project_toml = "/home/tim/.julia/environments/v1.8/Project.toml"
uuid = UUID("19a2e0a6-566e-4c00-8888-a0edb19dbca0")
project_toml = "/home/tim/.julia/dev/AdmissionSuite/Project.toml"
uuid = UUID("19a2e0a6-566e-4c00-8888-a0edb19dbca0")
dicts = Dict{String, Any}[]
project_toml = "/home/tim/.julia/environments/v1.8/Project.toml"
uuid = UUID("19a2e0a6-566e-4c00-8888-a0edb19dbca0")
project_toml = "/home/tim/.julia/dev/AdmissionSuite/Project.toml"
uuid = UUID("19a2e0a6-566e-4c00-8888-a0edb19dbca0")
dicts = Dict{String, Any}[]

indicating that the preferences are never loaded (here is printed at the entry point of AdmitConfiguration's attempt to load its own preferences). This is because Base.collect_preferences never receives the project_toml of the subdirectory.

There appear to be two potential solutions:

• fix the path problem for subdirectories
• store the configuration preferences in AdmissionSuite rather than the subdir package

Since the configuration written by AdmitConfiguration is intended for consumption by all the subpackages, there is a fair amount of sense in the latter. To attempt this, I added a name and uuid to AdmissionSuite/Project.toml, and then to AdmitConfiguration I added using AdmissionSuite and used set_preferences!(uuid, ...) where uuid is the uuid of AdmissionSuite. This initially seemed like it might work, but LocalPreferences.jl file got stored in my v1.6/environments despite the fact that AdmissionSuite was not in the [deps] of my global environment, and the preferences never ended up getting loaded. So, instead I modified the function that created the preferences file to use

    tomlfile = joinpath(pkgdir(AdmissionSuite), "LocalPreferences.toml")
    set_preferences!(tomlfile, "AdmissionSuite", prefs...)   # this method is undocumented

and the __init__ for AdmitConfiguration to use

    var1 = load_preference(AdmissionSuite, "var1")
    ...

This worked (the preferences get loaded with using AdmitConfiguration), but it is a narrow path to success and I only got there by reading the source code.

[vchuravy]

Have you tried 1.6.5? This feels similar to #43361

[timholy]

It is similar, but not the same. Here's a MWE:

using Pkg, TOML
Pkg.generate("Suite")
suitetoml = TOML.parsefile("Suite/Project.toml")
suiteuuidstr = suitetoml["uuid"]
cd("Suite")
Pkg.activate(".")
Pkg.generate("Config")
Pkg.generate("Use")
Pkg.develop(path="Config")
Pkg.develop(path="Use")
Pkg.activate("./Config")
Pkg.add("Preferences")
Pkg.add("UUIDs")
open("Config/src/Config.jl", "w") do io
    write(io, """
module Config
using Preferences
using UUIDs

function setpreferences()
    @set_preferences!("prefA" => "SuitePrefs")
    # Alternatives:
    # set_preferences!(UUID($suiteuuidstr), "prefA" => "SuitePrefs")
end

const prefA = Ref{String}()

function __init__()
    p = @load_preference("prefA")
    # Alternatives:
    # p = load_preference(UUID($suiteuuidstr), "prefA")
    p !== nothing && (prefA[] = p)
end
end  # module Config
""")
end
Pkg.activate("./Use")
cd("Use")
Pkg.develop(path="../Config")
Pkg.add("Test")
open("src/Use.jl", "w") do io
    write(io, """
module Use
using Config

getpref() = Config.prefA[]

end  # module Use
""")
end
mkdir("test")
open("test/runtests.jl", "w") do io
    write(io,
"""
using Use
using Test

@testset "Use" begin
    @test Use.Config.prefA[] == "SuitePrefs"
end
""")
end
cd("..") # back to Suite/
Pkg.activate(".")
Pkg.resolve()
using Config
Config.setpreferences()
Pkg.test("Use")

Test fails despite the fact that

tim@diva:/tmp/prefbug/Suite$ cat LocalPreferences.toml 
[Config]
prefA = "SuitePrefs"

[fredrikekre]

Is this JuliaLang/Pkg.jl#2500 then?

[timholy]

Seems like it could be.

[timholy]

I can work around it with push!(LOAD_PATH, suitedir) before @load_preference. EDIT: but only on Julia 1.6.

[timholy]

This seems to be working now on 1.8, so it was probably a duplicate of JuliaLang/Pkg.jl#2500