Feature request: Support instantiation of multiple julia environments in parallel #3495

schlichtanders · 2023-05-19T16:38:58Z

With Julia 1.9 it gets more important to do proper precompilation ahead of time. I am using Pluto and like to instantiate all notebook environments with high speed.

Currently, when just spawning multiple julia processes and instantiating several environments, I get a GitError as soon as two environments have the same dependency (there is probably some randomness involved, but it failes constantly for me - there are several overlaps)

GitError(Code:ERROR, Class:OS, could not open '/home/myhome/.julia/packages/MyPackage/1opWe/Project.toml' for writing: Permission denied)

It would be really great to have a performant way of instantiating multiple environments at once.

~~EDIT: Just realized, that I only tested this Julia 1.8.5, I am going to test on Julia 1.9 immediately~~ Applies to 1.9 as well

vchuravy · 2023-05-19T17:40:37Z

What's the full backtrace for the GitError?

schlichtanders · 2023-05-20T07:22:50Z

Meanwhile I upgraded the Manifests (before they still were 1.8.5 manifests).

So far rerunning again produced an Error with the above mentioned MyPackage, however the error was not printed to the console. I will try rerunning and hope I get the stacktrace as well.

In addition the following error occured:

IOError: readdir("/home/jolin_user/.julia/packages/ColorSchemes/lTeyu/data"): no such file or directory (ENOENT)
Stacktrace:
  [1] try_yieldto(undo::typeof(Base.ensure_rescheduled))
    @ Base ./task.jl:920
  [2] wait()
    @ Base ./task.jl:984
  [3] wait(c::Base.GenericCondition{ReentrantLock}; first::Bool)
    @ Base ./condition.jl:130
  [4] wait
    @ ./condition.jl:125 [inlined]
  [5] take_unbuffered(c::Channel{Tuple{String, Vector{String}, Vector{String}}})
    @ Base ./channels.jl:473
  [6] take!
    @ ./channels.jl:450 [inlined]
  [7] iterate(c::Channel{Tuple{String, Vector{String}, Vector{String}}}, state::Nothing)
    @ Base ./channels.jl:592
  [8] set_readonly(path::String)
    @ Pkg /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/utils.jl:54
  [9] macro expansion
    @ /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:854 [inlined]
 [10] (::Pkg.Operations.var"#47#52"{Bool, Pkg.Types.Context, Tuple{SubString{String}, Dict{Base.UUID, Base.SHA1}}, Channel{Any}, Channel{NamedTuple{(:pkg, :urls, :path), Tuple{Pkg.Types.PackageEntry, Set{String}, String}}}})()
    @ Pkg.Operations ./task.jl:514
Stacktrace:
  [1] pkgerror(::String, ::Vararg{String})
    @ Pkg.Types /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/Types.jl:69
  [2] macro expansion
    @ /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:874 [inlined]
  [3] macro expansion
    @ ./task.jl:476 [inlined]
  [4] download_source(ctx::Pkg.Types.Context; readonly::Bool)
    @ Pkg.Operations /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:815
  [5] download_source
    @ /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:791 [inlined]
  [6] instantiate(ctx::Pkg.Types.Context; manifest::Nothing, update_registry::Bool, verbose::Bool, platform::Base.BinaryPlatforms.Platform, allow_build::Bool, allow_autoprecomp::Bool, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:1738
  [7] instantiate
    @ /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:1640 [inlined]
  [8] instantiate(; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:1639
  [9] instantiate()
    @ Pkg.API /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:1639
 [10] top-level scope

Everything is hard to reproduce - running my setup again sometimes does not show any error at all.

schlichtanders · 2023-05-20T07:30:38Z

another stacktrace

Error when installing package DataFrames:
IOError: readdir("/home/jolin_user/.julia/packages/DataFrames/LteEl/docs/src"): no such file or directory (ENOENT)
Stacktrace:
  [1] try_yieldto(undo::typeof(Base.ensure_rescheduled))
    @ Base ./task.jl:920
  [2] wait()
    @ Base ./task.jl:984
  [3] wait(c::Base.GenericCondition{ReentrantLock}; first::Bool)
    @ Base ./condition.jl:130
  [4] wait
    @ ./condition.jl:125 [inlined]
  [5] take_unbuffered(c::Channel{Tuple{String, Vector{String}, Vector{String}}})
    @ Base ./channels.jl:473
  [6] take!
    @ ./channels.jl:450 [inlined]
  [7] iterate(c::Channel{Tuple{String, Vector{String}, Vector{String}}}, state::Nothing)
    @ Base ./channels.jl:592
  [8] set_readonly(path::String)
    @ Pkg /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/utils.jl:54
  [9] macro expansion
    @ /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:854 [inlined]
 [10] (::Pkg.Operations.var"#47#52"{Bool, Pkg.Types.Context, Tuple{SubString{String}, Dict{Base.UUID, Base.SHA1}}, Channel{Any}, Channel{NamedTuple{(:pkg, :urls, :path), Tuple{Pkg.Types.PackageEntry, Set{String}, String}}}})()
    @ Pkg.Operations ./task.jl:514
Stacktrace:
  [1] pkgerror(::String, ::Vararg{String})
    @ Pkg.Types /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/Types.jl:69
  [2] macro expansion
    @ /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:874 [inlined]
  [3] macro expansion
    @ ./task.jl:476 [inlined]
  [4] download_source(ctx::Pkg.Types.Context; readonly::Bool)
    @ Pkg.Operations /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:815
  [5] download_source
    @ /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:791 [inlined]
  [6] instantiate(ctx::Pkg.Types.Context; manifest::Nothing, update_registry::Bool, verbose::Bool, platform::Base.BinaryPlatforms.Platform, allow_build::Bool, allow_autoprecomp::Bool, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:1738
  [7] instantiate
    @ /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:1640 [inlined]
  [8] instantiate(; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:1639
  [9] instantiate()
    @ Pkg.API /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:1639
 [10] top-level scope

unfortunately, the original GitError I could not reproduce so far (maybe it was depending on the old manifest)

schlichtanders · 2023-05-20T07:37:34Z

I now reverted my setup to use the old manifest and was able to replicate the GitError

ERROR: GitError(Code:ERROR, Class:OS, could not open '/home/jolin_user/.julia/packages/MyPackage/1opWe/Project.toml' for writing: Permission denied)
Stacktrace:
  [1] macro expansion
    @ /usr/local/julia/share/julia/stdlib/v1.9/LibGit2/src/error.jl:111 [inlined]
  [2] checkout_tree(repo::LibGit2.GitRepo, obj::LibGit2.GitTree; options::LibGit2.CheckoutOptions)
    @ LibGit2 /usr/local/julia/share/julia/stdlib/v1.9/LibGit2/src/repository.jl:358
  [3] checkout_tree
    @ /usr/local/julia/share/julia/stdlib/v1.9/LibGit2/src/repository.jl:354 [inlined]
  [4] checkout_tree_to_path(repo::LibGit2.GitRepo, tree::LibGit2.GitTree, path::String)
    @ Pkg.GitTools /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/GitTools.jl:84
  [5] install_git(io::Base.TTY, uuid::Base.UUID, name::String, hash::Base.SHA1, urls::Set{String}, version_path::String)
    @ Pkg.Operations /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:699
  [6] download_source(ctx::Pkg.Types.Context; readonly::Bool)
    @ Pkg.Operations /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:905
    @ /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:791 [inlined]
  [8] instantiate(ctx::Pkg.Types.Context; manifest::Nothing, update_registry::Bool, verbose::Bool, platform::Base.BinaryPlatforms.Platform, allow_build::Bool, allow_autoprecomp::Bool, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:1738
  [9] instantiate
    @ /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:1640 [inlined]
 [10] instantiate(; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:1639
 [11] instantiate()
    @ Pkg.API /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:1639
   Installed libaom_jll ─────────────────── v3.4.0+0
  [1] macro expansion
    @ /usr/local/julia/share/julia/stdlib/v1.9/LibGit2/src/error.jl:111 [inlined]
  [2] checkout_tree(repo::LibGit2.GitRepo, obj::LibGit2.GitTree; options::LibGit2.CheckoutOptions)
    @ LibGit2 /usr/local/julia/share/julia/stdlib/v1.9/LibGit2/src/repository.jl:358
  [3] checkout_tree
    @ /usr/local/julia/share/julia/stdlib/v1.9/LibGit2/src/repository.jl:354 [inlined]
  [4] checkout_tree_to_path(repo::LibGit2.GitRepo, tree::LibGit2.GitTree, path::String)
    @ Pkg.GitTools /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/GitTools.jl:84
  [5] install_git(io::Base.TTY, uuid::Base.UUID, name::String, hash::Base.SHA1, urls::Set{String}, version_path::String)
    @ Pkg.Operations /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:699
    @ Pkg.API /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:1639

vchuravy · 2023-05-20T15:42:25Z

So some of these things make sense. The checkouts of the packages are read-only.

So GitError(Code:ERROR, Class:OS, could not open '/home/jolin_user/.julia/packages/MyPackage/1opWe/Project.toml' for writing: Permission denied) is sensible, but the real question is: Who is calling instantiate on that copy?

What operations are you doing? How could one reproduce these errors?

schlichtanders · 2023-05-20T15:51:26Z

I have a couple of directories with Project.toml and Manifest.toml (actually there is really only these two files, which have been extracted from Pluto notebooks and written to their separate directories)

then I am doing just

@sync for environment_dir in environment_directories
    @async run(`julia --project $environment_dir -e 'import Pkg; Pkg.instantiate()'`)
end

I am sorry, but creating the environment_directories is a bit tough right now, I hope that the error stacks above are enough to spot which additions would be needed to make several Pkg.instantiate() safely work in parallel.

vchuravy · 2023-05-21T13:39:52Z

I am sorry but what this points to is that how ever you generate environment_directories is buggy.

You simply can't call instantiate on .julia/packages/** those directories are marked intentionally read-only.

So the error you are seeing is correct, trying to write to that directory (either to modify Project or to create Manifest) is not allowed

schlichtanders · 2023-05-21T14:06:24Z

I bet that the environment_directories are fully fine - they were generated with Pluto.jl automatically. And everything works without problems when done iteratively.

MyPackage is a bit special in the sense that it comes from a LocalRegistry and not general.

vchuravy · 2023-05-21T17:04:52Z

Without a MWE it won't be possible to determine why you are seeing this error. As a I said the last error in particular hints at someone calling instantiate on a .julia/packages/** directory. That is not supported, these are not environments, but local immutable copies of the package.

The only thing you should see races on are cache files in .julia/compiled but generally Julia supports parallel compilation (modulo the fact that in case of a race the last attempt wins).

schlichtanders · 2023-05-24T15:04:03Z

Okay, here a reproducible example. It is not minimal (I gave it an attempt to minifiy it further, but without success.), but at least it is reproducible. Concretely it mirrors my own experiments above.

start docker with docker run -it --rm julia:1.9 bash
execute the following

cd root
apt-get update && apt-get install -y git
git clone https://github.com/jolin-io/JolinWorkspaceTemplate/
cd JolinWorkspaceTemplate

# This will lead to the GitError
git reset --hard 3266327

# This will lead to the other Errors
# git reset --hard cd3614b

# julia
julia -e '
    import Pkg
    Pkg.Registry.add("General")
    Pkg.Registry.add(Pkg.RegistrySpec(url="https://github.com/jolin-io/JolinRegistry.jl"))
    Pkg.add("JolinPlutoCICD")

    using JolinPlutoCICD
    workflows = JolinPlutoCICD.get_all_workflow_paths("./workflows")
    envs = Dict(w => JolinPlutoCICD.create_pluto_env(w) for w in workflows)
    
    @sync for (_, envdir) in envs
        @show envdir
        @async Base.run(`julia --project=$envdir -e "import Pkg; Pkg.instantiate()"`)
    end
'

please mind that there are two different git reset --hard which will point to two different tests.

vchuravy · 2023-05-24T16:45:48Z

Thank you for the MWE it helped me understand what you want to achieve.

Julia itself doesn't instantiate, it defers that to Pkg.jl

I suspect that Pkg would need to either implement PIDLocks (similar to in JuliaLang/julia#49052)
or try use atomic move (fraught with pitfalls) to make sure that other Julia processes don't see "partially instantiated" or race upon the ".julia/packages/..." directories.

schlichtanders · 2023-06-06T12:57:58Z

@IanButterworth would this also be solved by the mentioned issue

charleskawczynski · 2023-07-10T17:44:05Z

I'm seeing this issue too, almost reproducibly so, in buildkite CI for https://github.com/CliMA/ClimaComms.jl

charleskawczynski · 2023-07-10T18:00:15Z

Is this a dup of #1219?

vchuravy · 2023-07-11T07:09:24Z

Not quite. Julia 1.10 will be better around races on cache-files, but there is still an open question around two separate processes calling instantiate concurrently, on the same project or a project that contains the same dependency.

IanButterworth · 2024-12-10T15:41:29Z

We should fix this. Specifically the race around

Pkg.jl/src/Operations.jl

Lines 1080 to 1083 in 116ba91

    
           success = install_archive(archive_urls, pkg.tree_hash, path, io=ctx.io) 
        
           if success && readonly 
        
               set_readonly(path) # In add mode, files should be read-only 
        
           end

vchuravy transferred this issue from JuliaLang/julia May 24, 2023

simonbyrne added the race-condition label Jul 10, 2023

IanButterworth linked a pull request Feb 23, 2025 that will close this issue

Make concurrent-process package source installation safe #4168

Open

davidanthoff mentioned this issue Feb 23, 2025

Be more resilient to corrupted package install scenarios #4169

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature request: Support instantiation of multiple julia environments in parallel #3495

Feature request: Support instantiation of multiple julia environments in parallel #3495

schlichtanders commented May 19, 2023 •

edited

Loading

vchuravy commented May 19, 2023

schlichtanders commented May 20, 2023

schlichtanders commented May 20, 2023

schlichtanders commented May 20, 2023

vchuravy commented May 20, 2023

schlichtanders commented May 20, 2023

vchuravy commented May 21, 2023

schlichtanders commented May 21, 2023

vchuravy commented May 21, 2023

schlichtanders commented May 24, 2023

vchuravy commented May 24, 2023

schlichtanders commented Jun 6, 2023

charleskawczynski commented Jul 10, 2023

charleskawczynski commented Jul 10, 2023

vchuravy commented Jul 11, 2023

IanButterworth commented Dec 10, 2024

Feature request: Support instantiation of multiple julia environments in parallel #3495

Feature request: Support instantiation of multiple julia environments in parallel #3495

Comments

schlichtanders commented May 19, 2023 • edited Loading

vchuravy commented May 19, 2023

schlichtanders commented May 20, 2023

schlichtanders commented May 20, 2023

schlichtanders commented May 20, 2023

vchuravy commented May 20, 2023

schlichtanders commented May 20, 2023

vchuravy commented May 21, 2023

schlichtanders commented May 21, 2023

vchuravy commented May 21, 2023

schlichtanders commented May 24, 2023

vchuravy commented May 24, 2023

schlichtanders commented Jun 6, 2023

charleskawczynski commented Jul 10, 2023

charleskawczynski commented Jul 10, 2023

vchuravy commented Jul 11, 2023

IanButterworth commented Dec 10, 2024

schlichtanders commented May 19, 2023 •

edited

Loading