--project=@script does not activate script project if outside of the Project Directory

[awadell1]

Launching a script with julia --project=@script foo.jl does not activate the script project if ran outside the containing folder.

In test_at_script.jL

#!/usr/bin/env -S julia --project=@script
@info "Check Active Env" Base.active_project() PROGRAM_FILE LOAD_PATH Base.load_path_expand("@script")

# Check that ../Project.toml got activated
active_project = realpath(Base.active_project())
expected = realpath(joinpath(@__DIR__, "..", "Project.toml"))
@assert active_project == expected  "$active_project != $expected"

Expected Directory Structure:

Documents/
    julia_at_script_issue/
        Project.toml
        bin/
            test_at_script.jl

Alternatively, run script.sh in the attached achieve: julia_at_script_issue.zip

Running from Documents/julia_at_script_issue/ with julia --startup-file=no --project=@script bin/test_at_script.jl or ./bin/test_at_script.jl gives the following (expected) output:

❯ julia --startup-file=no --project=@script bin/test_at_script.jl
┌ Info: Check Active Env
│   Base.active_project() = "/Users/alexwadell/Documents/julia_at_script_issue/Project.toml"
│   PROGRAM_FILE = "bin/test_at_script.jl"
│   LOAD_PATH =
│    3-element Vector{String}:
│     "@"
│     "@v#.#"
│     "@stdlib"
└   Base.load_path_expand("@script") = "Project.toml"

However, running from outside of Documents/julia_at_script_issue activates the default environment instead:

❯ julia --project=@script --startup-file=no julia_at_script_issue/bin/test_at_script.jl
┌ Info: Check Active Env
│   Base.active_project() = "/Users/alexwadell/.julia/environments/v1.11/Project.toml"
│   PROGRAM_FILE = "julia_at_script_issue/bin/test_at_script.jl"
│   LOAD_PATH =
│    3-element Vector{String}:
│     "@"
│     "@v#.#"
│     "@stdlib"
└   Base.load_path_expand("@script") = "julia_at_script_issue/Project.toml"
ERROR: LoadError: AssertionError: /Users/alexwadell/.julia/environments/v1.11/Project.tom
l != /Users/alexwadell/Documents/julia_at_script_issue/Project.toml
Stacktrace:
 [1] top-level scope
   @ ~/Documents/julia_at_script_issue/bin/test_at_script.jl:5
in expression starting at /Users/alexwadell/Documents/julia_at_script_issue/bin/test_at_scr
ipt.jl:5

Oddly, enough Base.load_path_expand("@cript") resolves the correct project file.

Setting JULIA_LOAD_PATH directly does result in the expect behavior outside the project directory:

❯ JULIA_LOAD_PATH=@script julia --startup-file=no julia_at_script_issue/bin/test_at_script.jl
┌ Info: Check Active Env
│   Base.active_project() = "julia_at_script_issue/Project.toml"
│   PROGRAM_FILE = "julia_at_script_issue/bin/test_at_script.jl"
│   LOAD_PATH =
│    1-element Vector{String}:
│     "@script"
└   Base.load_path_expand("@script") = "julia_at_script_issue/Project.toml"

Julia Info

Julia Version 1.11.0-rc1
Commit 3a35aec36d1 (2024-06-25 10:23 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: macOS (arm64-apple-darwin22.4.0)
  CPU: 8 × Apple M1
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, apple-m1)
Threads: 1 default, 0 interactive, 1 GC (on 4 virtual cores)
Environment:
  JULIA_PKG_USE_CLI_GIT = true

Related PRs

#50864
#53356

[awadell1]

The root cause seems to be the use of PROGRAM_FILE to locate the project file for @script;
however PROGRAM_FILE get set in exec_options which happens after Base's __init__() method sets the active project.

Building julia with @info around the relevant parts gives the following:

+ /Users/alexwadell/Documents/julia_at_script_issue/julia/usr/bin/julia --project=@script --startup-file=no ./julia_at_script_issue/bin/test_at_script.jl
┌ Info: PROGRAM_FILE
│   PROGRAM_FILE = 
└ @ Base Base.jl:625
┌ Info: init_load_path
└ @ Base initdefs.jl:245
┌ Info: load_path_expand
│   env = @script
└ @ Base initdefs.jl:278
┌ Info: load_path_expand - script - program file
│   PROGRAM_FILE = 
└ @ Base initdefs.jl:288
┌ Info: load_path_expand - script
│   env = @script
│   dir = 
└ @ Base initdefs.jl:299
┌ Info: load_path_expand - script - found
│   out = nothing
└ @ Base initdefs.jl:302
┌ Info: init_active_project
│   project = @script
│   load_path_expand(project) = nothing
└ @ Base initdefs.jl:265
┌ Info: load_path_expand
│   env = @script
└ @ Base initdefs.jl:278
┌ Info: load_path_expand - script - program file
│   PROGRAM_FILE = 
└ @ Base initdefs.jl:288
┌ Info: load_path_expand - script
│   env = @script
│   dir = 
└ @ Base initdefs.jl:299
┌ Info: load_path_expand - script - found
│   out = nothing
└ @ Base initdefs.jl:302
[...]
┌ Info: Check Active Env
│   Base.active_project() = "/Users/alexwadell/.julia/environments/v1.12/Project.toml"
│   PROGRAM_FILE = "./julia_at_script_issue/bin/test_at_script.jl"
│   Base.load_path_expand("@script") = "./julia_at_script_issue/Project.toml"
└   Base.current_project(dirname(PROGRAM_FILE)) = "./julia_at_script_issue/Project.toml"
[...]

Doing JULIA_LOAD_PATH=@script julia ... also suffers from an unset PROGRAM_FILE, however I think it's getting a second chance in Base.active_project() due to LOAD_PATH getting re-evaluated here:

julia/base/initdefs.jl

Line 352 in d7609d8

project = load_path_expand(project)

Unfortunately, ARGS is also unset until later in julia's initialization (reply_entrypoint per Initialization of the Julia runtime), so setting PROGRAM_FILE = ARGS[1] in load_path_expand isn't feasible.

I think the fix is to add a PROGRAM_FILE entry to JLOptions and set it in jl_parse_opts(), instead of exec_options. Does that track? Or is there (hopefully) a cleaner fix?

[awadell1]

This appears to still be present in the released version of 1.11, using the example in julia_at_script_issue.zip, I'm getting the following:

❯ bash ./script.sh
+ echo 'This works'
This works
+ julia +1.11 --project=@script --startup-file=no bin/test_at_script.jl
┌ Info: Check Active Env
│   Base.active_project() = "/Users/alexwadell/Downloads/julia_at_script_issue/Project.toml"
│   PROGRAM_FILE = "bin/test_at_script.jl"
│   Base.load_path_expand("@script") = "Project.toml"
└   Base.current_project(dirname(PROGRAM_FILE)) = "Project.toml"
+ ./julia_at_script_bug/bin/test_at_script.jl
./script.sh: line 9: ./julia_at_script_bug/bin/test_at_script.jl: No such file or directory
+ cd ../
+ JULIA_LOAD_PATH=@script
+ julia --startup-file=no julia_at_script_issue/bin/test_at_script.jl
┌ Info: Check Active Env
│   Base.active_project() = "julia_at_script_issue/Project.toml"
│   PROGRAM_FILE = "julia_at_script_issue/bin/test_at_script.jl"
│   Base.load_path_expand("@script") = "julia_at_script_issue/Project.toml"
└   Base.current_project(dirname(PROGRAM_FILE)) = "julia_at_script_issue/Project.toml"
+ echo 'This does not'
This does not
+ julia +1.11 --project=@script --startup-file=no ./julia_at_script_issue/bin/test_at_script.jl
┌ Info: Check Active Env
│   Base.active_project() = "/Users/alexwadell/.julia/environments/v1.11/Project.toml"
│   PROGRAM_FILE = "./julia_at_script_issue/bin/test_at_script.jl"
│   Base.load_path_expand("@script") = "./julia_at_script_issue/Project.toml"
└   Base.current_project(dirname(PROGRAM_FILE)) = "./julia_at_script_issue/Project.toml"
ERROR: LoadError: AssertionError: /Users/alexwadell/.julia/environments/v1.11/Project.toml != /Users/alexwadell/Downloads/julia_at_script_issue/Project.toml
Stacktrace:
 [1] top-level scope
   @ ~/Downloads/julia_at_script_issue/bin/test_at_script.jl:7
in expression starting at /Users/alexwadell/Downloads/julia_at_script_issue/bin/test_at_script.jl:7
+ julia +1.11 --startup-file=no -e 'using InteractiveUtils; versioninfo()'
Julia Version 1.11.0
Commit 501a4f25c2b (2024-10-07 11:40 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: macOS (arm64-apple-darwin22.4.0)
  CPU: 8 × Apple M1
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, apple-m1)
Threads: 1 default, 0 interactive, 1 GC (on 4 virtual cores)
Environment:
  JULIA_PKG_USE_CLI_GIT = true

To summarize:

• julia --project=@script path/to/proj.jl does activate the correct environment if run outside of the project directory
• JULIA_LOAD_PATH=@script julia path/to/prog.jl does activate the correct environment, even when outside of the project directory

[tecosaur]

Thanks for reporting and investigating this @awadell1! Since you seem to have worked out what's going on here, would you be interested in spinning up a PR modifying JLOptions the way you describe?

cc: @Keno, who originally introduced this functionality.

[awadell1]

Yes, I can take a stab at it this weekend.

…

On Oct 21, 2024, at 1:29 PM, Timothy ***@***.***> wrote: Thanks for reporting and investigating this @awadell1 <https://github.com/awadell1>! Since you seem to have worked out what's going on here, would you be interested in spinning up a PR modifying JLOptions the way you describe? cc: @Keno <https://github.com/Keno>, who originally introduced this functionality. — Reply to this email directly, view it on GitHub <#55069 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABMWAEXSUO6UTLYTE3S7JMDZ4U2YVAVCNFSM6AAAAABKQ5JTF2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRXGMYTQMZQGA>. You are receiving this because you were mentioned.