MPI.jl fails to precompile on Julia nightly in some configurations #48039

giordano · 2022-12-29T16:47:06Z

Package seems to be somewhat broken on nightly in some, but not all, configurations, for example https://github.com/JuliaParallel/MPI.jl/actions/runs/3800486725/jobs/6464015891

 Warning: Module MPIPreferences with build ID fafbfcfd-8255-0b55-0000-006331197460 is missing from the cache.
│ This may mean MPIPreferences [3da0fdf6-3ccc-4f1b-acd9-58baa6c99267] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1648
ERROR: LoadError: ArgumentError: NULL library handle
Stacktrace:
  [1] #dlsym#1
    @ ./libdl.jl:57 [inlined]
  [2] kwcall(::NamedTuple{(:throw_error,), Tuple{Bool}}, ::typeof(Base.Libc.Libdl.dlsym), hnd::Ptr{Nothing}, s::Symbol)
    @ Base.Libc.Libdl ./libdl.jl:56
  [3] var"@mpichk"(__source__::LineNumberNode, __module__::Module, expr::Any, min_version::Any)
    @ MPI.API ~/work/MPI.jl/MPI.jl/src/api/api.jl:123
  [4] #macroexpand#65
    @ ./expr.jl:119 [inlined]
  [5] macroexpand
    @ ./expr.jl:117 [inlined]
  [6] docm(source::LineNumberNode, mod::Module, meta::Any, ex::Any, define::Bool)
    @ Base.Docs ./docs/Docs.jl:538
  [7] docm(source::LineNumberNode, mod::Module, meta::Any, ex::Any)
    @ Base.Docs ./docs/Docs.jl:538
  [8] (::DocStringExtensions.var"#35#36"{typeof(DocStringExtensions.template_hook)})(::LineNumberNode, ::Vararg{Any})
    @ DocStringExtensions ~/.julia/packages/DocStringExtensions/JVu77/src/templates.jl:11
  [9] var"@doc"(__source__::LineNumberNode, __module__::Module, x::Vararg{Any})
    @ Core ./boot.jl:539
 [10] include(mod::Module, _path::String)
    @ Base ./Base.jl:452
 [11] include(x::String)
    @ MPI.API ~/work/MPI.jl/MPI.jl/src/api/api.jl:1
 [12] top-level scope
    @ ~/work/MPI.jl/MPI.jl/src/api/api.jl:136
 [13] include(mod::Module, _path::String)
    @ Base ./Base.jl:452
 [14] include(x::String)
    @ MPI ~/work/MPI.jl/MPI.jl/src/MPI.jl:1
 [15] top-level scope
    @ ~/work/MPI.jl/MPI.jl/src/MPI.jl:43
 [16] include
    @ ./Base.jl:452 [inlined]
 [17] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
    @ Base ./loading.jl:1900
 [18] top-level scope
    @ stdin:2
in expression starting at /home/runner/work/MPI.jl/MPI.jl/src/api/generated_api.jl:642
in expression starting at /home/runner/work/MPI.jl/MPI.jl/src/api/generated_api.jl:642
in expression starting at /home/runner/work/MPI.jl/MPI.jl/src/api/api.jl:1
in expression starting at /home/runner/work/MPI.jl/MPI.jl/src/MPI.jl:1
in expression starting at stdin:2
ERROR: LoadError: Failed to precompile MPI [da04e1cc-30fd-572f-bb4f-1f8673147195] to "/home/runner/.julia/compiled/v1.10/MPI/jl_1rTeBS".
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
   @ Base ./loading.jl:2124
 [3] compilecache
   @ ./loading.jl:2016 [inlined]
 [4] _require(pkg::Base.PkgId, env::String)
   @ Base ./loading.jl:1660
 [5] _require_prelocked(uuidkey::Base.PkgId, env::String)
   @ Base ./loading.jl:1515
 [6] macro expansion
   @ ./loading.jl:1503 [inlined]
 [7] macro expansion
   @ ./lock.jl:267 [inlined]
 [8] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1466
in expression starting at /home/runner/work/MPI.jl/MPI.jl/docs/examples/01-hello.jl:1

CC @vchuravy as I suspect this is related to #47184

Originally posted by @giordano in JuliaParallel/MPI.jl#698 (comment)

Some pointers appear to be NULL, @gbaraldi mentioned he's seen something similar elsewhere.

The text was updated successfully, but these errors were encountered:

giordano · 2023-01-13T11:09:04Z

I managed to find the steps to reproduce the crash, following what's done in MPI.jl CI:

delete all compiled cache, e.g. ~/.julia/compiled/v1.10
]dev MPI
activate the environment where MPI.jl is dev-ed, e.g. ]activate ~/.julia/dev/MPI
]dev ~/.julia/dev/MPI/lib/MPIPreferences
using MPIPreferences
MPIPreferences.use_jll_binary("OpenMPI_jll")
]test

Running the tests in this environment should fail as reported above

Full stacktrace I got locally

(MPI) pkg> test
     Testing MPI
      Status `/tmp/jl_KiyUCN/Project.toml`
  [21141c5a] AMDGPU v0.4.4
  [052768ef] CUDA v3.12.1
  [497a8b3b] DoubleFloats v1.2.2
  [da04e1cc] MPI v0.20.7 `~/.julia/dev/MPI`
  [3da0fdf6] MPIPreferences v0.1.7 `~/.julia/dev/MPI/lib/MPIPreferences`
  [44cfe95a] Pkg v1.9.0 `@stdlib/Pkg`
  [9a3f8284] Random `@stdlib/Random`
  [fa267f1f] TOML v1.0.3 `@stdlib/TOML`
  [8dfed614] Test `@stdlib/Test`
      Status `/tmp/jl_KiyUCN/Manifest.toml`
  [21141c5a] AMDGPU v0.4.4
  [621f4979] AbstractFFTs v1.2.1
  [79e6a3ab] Adapt v3.4.0
⌅ [ab4f0b2a] BFloat16s v0.2.0
  [b99e7846] BinaryProvider v0.5.10
  [fa961155] CEnum v0.4.2
  [052768ef] CUDA v3.12.1
  [d360d2e6] ChainRulesCore v1.15.6
  [9e997f8a] ChangesOfVariables v0.1.4
  [34da2185] Compat v4.5.0
  [187b0558] ConstructionBase v1.4.1
⌃ [ffbed154] DocStringExtensions v0.9.1
  [497a8b3b] DoubleFloats v1.2.2
  [e2ba6199] ExprTools v0.1.8
  [0c68f7d7] GPUArrays v8.5.0
  [46192b85] GPUArraysCore v0.1.2
⌅ [61eb1bfa] GPUCompiler v0.16.7
  [14197337] GenericLinearAlgebra v0.3.5
  [3587e190] InverseFunctions v0.1.8
  [92d709cd] IrrationalConstants v0.1.1
  [692b3bcd] JLLWrappers v1.4.1
  [929cbde3] LLVM v4.14.1
  [2ab3a3ac] LogExpFunctions v0.3.19
  [da04e1cc] MPI v0.20.7 `~/.julia/dev/MPI`
  [3da0fdf6] MPIPreferences v0.1.7 `~/.julia/dev/MPI/lib/MPIPreferences`
  [1914dd2f] MacroTools v0.5.10
  [f27b6e38] Polynomials v3.2.2
  [21216c6a] Preferences v1.3.0
  [be4d8f0f] Quadmath v0.5.6
  [74087812] Random123 v1.6.0
  [e6cf234a] RandomNumbers v1.5.3
  [3cdcf5f2] RecipesBase v1.3.3
  [189a3867] Reexport v1.2.2
  [ae029012] Requires v1.3.0
  [efcf1570] Setfield v1.1.1
  [66db9d55] SnoopPrecompile v1.0.3
  [276daf66] SpecialFunctions v2.1.7
  [1e83bf80] StaticArraysCore v1.4.0
  [a759f4b9] TimerOutputs v0.5.22
  [6e34b625] Bzip2_jll v1.0.8+0
  [ab5a07f8] Elfutils_jll v0.182.0+1
  [2696aab5] HIP_jll v5.2.3+2
  [dad2f222] LLVMExtra_jll v0.0.16+0
  [86de99a1] LLVM_jll v14.0.6+1
  [d4300ac3] Libgcrypt_jll v1.8.7+0
  [7e76a0d4] Libglvnd_jll v1.6.0+0
  [7add5ba3] Libgpg_error_jll v1.42.0+0
  [94ce4f54] Libiconv_jll v1.16.1+2
  [7cb0a576] MPICH_jll v4.0.2+5
  [f1f71cc9] MPItrampoline_jll v5.0.2+1
  [9237b28f] MicrosoftMPI_jll v10.1.3+2
  [7f51dc2b] NUMA_jll v2.0.14+0
  [fe0851c0] OpenMPI_jll v4.1.3+3
  [efe28fd5] OpenSpecFun_jll v0.5.5+0
  [8fbdd1d2] ROCmCompilerSupport_jll v5.2.3+1
  [873c0968] ROCmDeviceLibs_jll v5.2.3+0
  [10ae2a08] ROCmOpenCLRuntime_jll v5.2.3+1
  [02c8fc9c] XML2_jll v2.10.3+0
  [aed1982a] XSLT_jll v1.1.34+0
  [ffd25f8a] XZ_jll v5.2.9+0
  [4f6342f7] Xorg_libX11_jll v1.6.9+4
  [0c0b7dd1] Xorg_libXau_jll v1.0.9+4
  [a3789734] Xorg_libXdmcp_jll v1.1.3+4
  [1082639a] Xorg_libXext_jll v1.3.4+4
  [a65dc6b1] Xorg_libpciaccess_jll v0.16.0+1
  [14d82f49] Xorg_libpthread_stubs_jll v0.1.0+3
  [c7cfdc94] Xorg_libxcb_jll v1.13.0+3
  [c4d99508] Xorg_xorgproto_jll v2019.2.0+2
  [c5fb5394] Xorg_xtrans_jll v1.4.0+3
  [c53206cc] argp_standalone_jll v1.3.1+0
  [d65627f6] fts_jll v1.2.8+0
  [dd59ff1a] hsa_rocr_jll v5.2.3+0
  [1cecccd7] hsakmt_roct_jll v5.2.3+0
  [8e53e030] libdrm_jll v2.4.110+0
  [43dd8cde] msgpack_jll v3.0.1+0
  [c88a4935] obstack_jll v1.2.2+0
  [1ef8cab2] rocBLAS_jll v5.2.3+2
  [52935e6f] rocPRIM_jll v5.2.3+1
  [a6151927] rocRAND_jll v5.2.3+1
  [8c6ce2ba] rocSPARSE_jll v5.2.3+1
  [5a766526] rocminfo_jll v5.2.3+0
  [0dad84c5] ArgTools v1.1.1 `@stdlib/ArgTools`
  [56f22d72] Artifacts `@stdlib/Artifacts`
  [2a0f44e3] Base64 `@stdlib/Base64`
  [ade2ca70] Dates `@stdlib/Dates`
  [8ba89e20] Distributed `@stdlib/Distributed`
  [f43a241f] Downloads v1.6.0 `@stdlib/Downloads`
  [7b1f6079] FileWatching `@stdlib/FileWatching`
  [9fa8497b] Future `@stdlib/Future`
  [b77e0a4c] InteractiveUtils `@stdlib/InteractiveUtils`
  [4af54fe1] LazyArtifacts `@stdlib/LazyArtifacts`
  [b27032c2] LibCURL v0.6.3 `@stdlib/LibCURL`
  [76f85450] LibGit2 `@stdlib/LibGit2`
  [8f399da3] Libdl `@stdlib/Libdl`
  [37e2e46d] LinearAlgebra `@stdlib/LinearAlgebra`
  [56ddb016] Logging `@stdlib/Logging`
  [d6f4376e] Markdown `@stdlib/Markdown`
  [ca575930] NetworkOptions v1.2.0 `@stdlib/NetworkOptions`
  [44cfe95a] Pkg v1.9.0 `@stdlib/Pkg`
  [de0858da] Printf `@stdlib/Printf`
  [3fa0cd96] REPL `@stdlib/REPL`
  [9a3f8284] Random `@stdlib/Random`
  [ea8e919c] SHA v0.7.0 `@stdlib/SHA`
  [9e88b42a] Serialization `@stdlib/Serialization`
  [6462fe0b] Sockets `@stdlib/Sockets`
  [2f01184e] SparseArrays `@stdlib/SparseArrays`
  [10745b16] Statistics v1.9.0 `@stdlib/Statistics`
  [fa267f1f] TOML v1.0.3 `@stdlib/TOML`
  [a4e569a6] Tar v1.10.0 `@stdlib/Tar`
  [8dfed614] Test `@stdlib/Test`
  [cf7118a7] UUIDs `@stdlib/UUIDs`
  [4ec0a83e] Unicode `@stdlib/Unicode`
  [e66e0078] CompilerSupportLibraries_jll v1.0.2+0 `@stdlib/CompilerSupportLibraries_jll`
  [d55e3150] LLD_jll v14.0.5+3 `@stdlib/LLD_jll`
  [deac9b47] LibCURL_jll v7.84.0+0 `@stdlib/LibCURL_jll`
  [29816b5a] LibSSH2_jll v1.10.2+0 `@stdlib/LibSSH2_jll`
  [c8ffd9c3] MbedTLS_jll v2.28.0+0 `@stdlib/MbedTLS_jll`
  [14a3606d] MozillaCACerts_jll v2022.10.11 `@stdlib/MozillaCACerts_jll`
  [4536629a] OpenBLAS_jll v0.3.21+0 `@stdlib/OpenBLAS_jll`
  [05823500] OpenLibm_jll v0.8.1+0 `@stdlib/OpenLibm_jll`
  [bea87d4a] SuiteSparse_jll v5.10.1+0 `@stdlib/SuiteSparse_jll`
  [83775a58] Zlib_jll v1.2.13+0 `@stdlib/Zlib_jll`
  [8f36deef] libLLVM_jll v14.0.6+0 `@stdlib/libLLVM_jll`
  [8e850b90] libblastrampoline_jll v5.2.0+0 `@stdlib/libblastrampoline_jll`
  [8e850ede] nghttp2_jll v1.48.0+0 `@stdlib/nghttp2_jll`
  [3f19e933] p7zip_jll v17.4.0+0 `@stdlib/p7zip_jll`
        Info Packages marked with ⌃ and ⌅ have new versions available, but those with ⌅ are restricted by compatibility constraints from upgrading.
Precompiling environment...
  89 dependencies successfully precompiled in 121 seconds
     Testing Running tests...
┌ Info: Running MPI tests
│   ArrayType = Array
│   nprocs = 4
│   MPIPreferences.abi = "OpenMPI"
└   MPIPreferences.binary = "OpenMPI_jll"
┌ Warning: Module MPIPreferences with build ID fafbfcfd-1776-88b9-0000-04d712353446 is missing from the cache.
│ This may mean MPIPreferences [3da0fdf6-3ccc-4f1b-acd9-58baa6c99267] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1650
┌ Warning: Module MPIPreferences with build ID fafbfcfd-1836-79ce-0000-04d7284f0aba is missing from the cache.
│ This may mean MPIPreferences [3da0fdf6-3ccc-4f1b-acd9-58baa6c99267] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1650
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for OpenMPI_jll [fe0851c0-eecd-5654-98d4-656369965a5c]
│   exception = Required dependency JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] failed to load from a cache file.
└ @ Base loading.jl:1675
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for OpenMPI_jll [fe0851c0-eecd-5654-98d4-656369965a5c]
│   exception = Required dependency JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] failed to load from a cache file.
└ @ Base loading.jl:1675
ERROR: ERROR: LoadError: LoadError: ERROR: LoadError: ERROR: LoadError: ArgumentError: NULL library handle
Stacktrace:
  [1] #dlsym#1
    @ ./libdl.jl:57 [inlined]
  [2] kwcall(::NamedTuple{(:throw_error,), Tuple{Bool}}, ::typeof(Base.Libc.Libdl.dlsym), hnd::Ptr{Nothing}, s::Symbol)
    @ Base.Libc.Libdl ./libdl.jl:56
  [3] var"@mpichk"(__source__::LineNumberNode, __module__::Module, expr::Any, min_version::Any)
    @ MPI.API ~/.julia/dev/MPI/src/api/api.jl:123
  [4] #macroexpand#64
    @ ./expr.jl:119 [inlined]
  [5] macroexpand
    @ ./expr.jl:117 [inlined]
  [6] docm(source::LineNumberNode, mod::Module, meta::Any, ex::Any, define::Bool)
    @ Base.Docs ./docs/Docs.jl:538
  [7] docm(source::LineNumberNode, mod::Module, meta::Any, ex::Any)
    @ Base.Docs ./docs/Docs.jl:538
  [8] (::DocStringExtensions.var"#35#36"{typeof(DocStringExtensions.template_hook)})(::LineNumberNode, ::ArgumentError: NULL library handle
Stacktrace:
  [1] #dlsym#1
    @ ./libdl.jl:57 [inlined]
  [2] kwcall(::NamedTuple{(:throw_error,), Tuple{Bool}}, ::typeof(Base.Libc.Libdl.dlsym), hnd::Ptr{Nothing}, s::Symbol)
    @ Base.Libc.Libdl ./libdl.jl:56
  [3] var"@mpichk"(__source__::LineNumberNode, __module__::Module, expr::Any, min_version::Any)
    @ MPI.API ~/.julia/dev/MPI/src/api/api.jl:123
  [4] #macroexpand#64
    @ ./expr.jl:119 [inlined]
  [5] macroexpand
    @ ./expr.jl:117 [inlined]
  [6] docm(source::LineNumberNode, mod::Module, meta::Any, ex::Any, define::Bool)
    @ Base.Docs ./docs/Docs.jl:538
  [7] docm(source::LineNumberNode, mod::Module, meta::Any, ex::Any)
    @ Base.Docs ./docs/Docs.jl:538
  [8] Vararg{Any})
    @ DocStringExtensions ~/.julia/packages/DocStringExtensions/JVu77/src/templates.jl:11
  [9] var"@doc"(__source__::LineNumberNode, __module__::Module, x::Vararg{Any})
    @ Core ./boot.jl:539
 [10] include(mod::Module, _path::String)
    @ Base ./Base.jl:456
 [11] include(x::String)
    @ MPI.API ~/.julia/dev/MPI/src/api/api.jl:1
 [12] top-level scope
    @ ~/.julia/dev/MPI/src/api/api.jl:136
 [13] include(mod::Module, _path::String)
    @ Base ./Base.jl:456
 [14] include(x::String)
    @ MPI ~/.julia/dev/MPI/src/MPI.jl:1
 [15] top-level scope
    @ ~/.julia/dev/MPI/src/MPI.jl:43
 [16] include
    @ ./Base.jl:456 [inlined]
 [17] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
    @ Base ./loading.jl:1902
 [18] top-level scope
    @ stdin:2
in expression starting at /home/mose/.julia/dev/MPI/src/api/generated_api.jl:642
in expression starting at /home/mose/.julia/dev/MPI/src/api/generated_api.jl:642
in expression starting at /home/mose/.julia/dev/MPI/src/api/api.jl:1
in expression starting at /home/mose/.julia/dev/MPI/src/MPI.jl:1
in expression starting at stdin:2
(::DocStringExtensions.var"#35#36"{typeof(DocStringExtensions.template_hook)})(::LineNumberNode, ::Vararg{Any})
    @ DocStringExtensions ~/.julia/packages/DocStringExtensions/JVu77/src/templates.jl:11
  [9] var"@doc"(__source__::LineNumberNode, __module__::Module, x::Vararg{Any})
    @ Core ./boot.jl:539
 [10] include(mod::Module, _path::String)
    @ Base ./Base.jl:456
 [11] include(x::String)
    @ MPI.API ~/.julia/dev/MPI/src/api/api.jl:1
 [12] top-level scope
    @ ~/.julia/dev/MPI/src/api/api.jl:136
 [13] include(mod::Module, _path::String)
    @ Base ./Base.jl:456
 [14] include(x::String)
    @ MPI ~/.julia/dev/MPI/src/MPI.jl:1
 [15] top-level scope
    @ ~/.julia/dev/MPI/src/MPI.jl:43
 [16] include
    @ ./Base.jl:456 [inlined]
 [17] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
    @ Base ./loading.jl:1902
 [18] top-level scope
    @ stdin:2
in expression starting at /home/mose/.julia/dev/MPI/src/api/generated_api.jl:642
in expression starting at /home/mose/.julia/dev/MPI/src/api/generated_api.jl:642
in expression starting at /home/mose/.julia/dev/MPI/src/api/api.jl:1
in expression starting at /home/mose/.julia/dev/MPI/src/MPI.jl:1
in expression starting at stdin:2
ArgumentError: NULL library handle
Stacktrace:
  [1] #dlsym#1
    @ ./libdl.jl:57 [inlined]
  [2] kwcall(::NamedTuple{(:throw_error,), Tuple{Bool}}, ::typeof(Base.Libc.Libdl.dlsym), hnd::Ptr{Nothing}, s::Symbol)
    @ Base.Libc.Libdl ./libdl.jl:56
  [3] var"@mpichk"(__source__::LineNumberNode, __module__::Module, expr::Any, min_version::Any)
    @ MPI.API ~/.julia/dev/MPI/src/api/api.jl:123
  [4] #macroexpand#64
    @ ./expr.jl:119 [inlined]
  [5] macroexpand
    @ ./expr.jl:117 [inlined]
  [6] docm(source::LineNumberNode, mod::Module, meta::Any, ex::Any, define::Bool)
    @ Base.Docs ./docs/Docs.jl:538
  [7] docm(source::LineNumberNode, mod::Module, meta::Any, ex::Any)
    @ Base.Docs ./docs/Docs.jl:538
  [8] (::DocStringExtensions.var"#35#36"{typeof(DocStringExtensions.template_hook)})(::LineNumberNode, ::Vararg{Any})
    @ DocStringExtensions ~/.julia/packages/DocStringExtensions/JVu77/src/templates.jl:11
  [9] var"@doc"(__source__::LineNumberNode, __module__::Module, x::Vararg{Any})
    @ Core ./boot.jl:539
 [10] include(mod::Module, _path::String)
    @ Base ./Base.jl:456
 [11] include(x::String)
    @ MPI.API ~/.julia/dev/MPI/src/api/api.jl:1
 [12] top-level scope
    @ ~/.julia/dev/MPI/src/api/api.jl:136
 [13] include(mod::Module, _path::String)
    @ Base ./Base.jl:456
 [14] include(x::String)
    @ MPI ~/.julia/dev/MPI/src/MPI.jl:1
 [15] top-level scope
    @ ~/.julia/dev/MPI/src/MPI.jl:43
 [16] include
    @ ./Base.jl:456 [inlined]
 [17] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
    @ Base ./loading.jl:1902
 [18] top-level scope
    @ stdin:2
in expression starting at /home/mose/.julia/dev/MPI/src/api/generated_api.jl:642
in expression starting at /home/mose/.julia/dev/MPI/src/api/generated_api.jl:642
in expression starting at /home/mose/.julia/dev/MPI/src/api/api.jl:1
in expression starting at /home/mose/.julia/dev/MPI/src/MPI.jl:1
in expression starting at stdin:2
ERROR: LoadError: ERROR: ArgumentError: NULL library handle
Stacktrace:
  [1] #dlsym#1
    @ ./libdl.jl:57 [inlined]
  [2] kwcall(::NamedTuple{(:throw_error,), Tuple{Bool}}, ::typeof(Base.Libc.Libdl.dlsym), hnd::Ptr{Nothing}, s::Symbol)
    @ Base.Libc.Libdl ./libdl.jl:56
  [3] var"@mpichk"(__source__::LineNumberNode, __module__::Module, expr::Any, min_version::Any)
    @ MPI.API ~/.julia/dev/MPI/src/api/api.jl:123
  [4] #macroexpand#64
    @ ./expr.jl:119 [inlined]
  [5] macroexpand
    @ ./expr.jl:117 [inlined]
  [6] docm(source::LineNumberNode, mod::Module, meta::Any, ex::Any, define::Bool)
    @ Base.Docs ./docs/Docs.jl:538
  [7] docm(source::LineNumberNode, mod::Module, meta::Any, ex::Any)
    @ Base.Docs ./docs/Docs.jl:538
  [8] LoadError: (::DocStringExtensions.var"#35#36"{typeof(DocStringExtensions.template_hook)})(::LineNumberNode, ::Vararg{Any})
    @ DocStringExtensions ~/.julia/packages/DocStringExtensions/JVu77/src/templates.jl:11
  [9] var"@doc"(__source__::LineNumberNode, __module__::Module, x::Vararg{Any})
    @ Core ./boot.jl:539
 [10] include(mod::Module, _path::String)
    @ Base ./Base.jl:456
 [11] include(x::String)
    @ MPI.API ~/.julia/dev/MPI/src/api/api.jl:1
 [12] top-level scope
    @ ~/.julia/dev/MPI/src/api/api.jl:136
 [13] include(mod::Module, _path::String)
    @ Base ./Base.jl:456
 [14] include(x::String)
    @ MPI ~/.julia/dev/MPI/src/MPI.jl:1
 [15] top-level scope
    @ ~/.julia/dev/MPI/src/MPI.jl:43
 [16] include
    @ ./Base.jl:456 [inlined]
 [17] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
    @ Base ./loading.jl:1902
 [18] top-level scope
    @ stdin:2
in expression starting at /home/mose/.julia/dev/MPI/src/api/generated_api.jl:642
in expression starting at /home/mose/.julia/dev/MPI/src/api/generated_api.jl:642
in expression starting at /home/mose/.julia/dev/MPI/src/api/api.jl:1
in expression starting at /home/mose/.julia/dev/MPI/src/MPI.jl:1
in expression starting at stdin:2
ERROR: LoadError: ERROR: LoadError: Failed to precompile MPI [da04e1cc-30fd-572f-bb4f-1f8673147195] to "/home/mose/.julia/compiled/v1.10/MPI/jl_TF4GXN".
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
   @ Base ./loading.jl:2145
 [3] compilecache
   @ ./loading.jl:2018 [inlined]
 [4] _require(pkg::Base.PkgId, env::String)
   @ Base ./loading.jl:1662
 [5] _require_prelocked(uuidkey::Base.PkgId, env::String)
   @ Base ./loading.jl:1517
 [6] macro expansion
   @ ./loading.jl:1505 [inlined]
 [7] macro expansion
   @ ./lock.jl:267 [inlined]
 [8] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1468
in expression starting at /home/mose/.julia/dev/MPI/docs/examples/01-hello.jl:1
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------

[15742] signal (15): Terminated
in expression starting at none:0

[15743] signal (15): Terminated
in expression starting at none:0
_ZNK4llvm12MachineInstr27isCandidateForCallSiteEntryENS0_9QueryTypeE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZN4llvm18ScheduleDAGSDNodes12EmitScheduleERNS_26MachineInstrBundleIteratorINS_12MachineInstrELb0EEE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZN4llvm16SelectionDAGISel17CodeGenAndEmitDAGEv at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZN4llvm16SelectionDAGISel20SelectAllBasicBlocksERKNS_8FunctionE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZN4llvm16SelectionDAGISel20runOnMachineFunctionERNS_15MachineFunctionE.part.902 at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZN12_GLOBAL__N_115X86DAGToDAGISel20runOnMachineFunctionERN4llvm15MachineFunctionE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZN4llvm19MachineFunctionPass13runOnFunctionERNS_8FunctionE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZN4llvm13FPPassManager13runOnFunctionERNS_8FunctionE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZN4llvm13FPPassManager11runOnModuleERNS_6ModuleE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc14SimpleCompilerclERNS_6ModuleE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
operator() at /home/mose/repo/julia/src/jitlayers.cpp:1198
_ZN4llvm3orc14IRCompileLayer4emitESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EENS0_16ThreadSafeModuleE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16IRTransformLayer4emitESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EENS0_16ThreadSafeModuleE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
emit at /home/mose/repo/julia/src/jitlayers.cpp:619
_ZN4llvm3orc31BasicIRLayerMaterializationUnit11materializeESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc19MaterializationTask3runEv at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZN4llvm6detail18UniqueFunctionBaseIvJSt10unique_ptrINS_3orc4TaskESt14default_deleteIS4_EEEE8CallImplIPFvS7_EEEvPvRS7_ at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession22dispatchOutstandingMUsEv at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession17OL_completeLookupESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EESt10shared_ptrINS0_23AsynchronousSymbolQueryEESt8functionIFvRKNS_8DenseMapIPNS0_8JITDylibENS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISF_vEEEENSG_ISD_vEENS_6detail12DenseMapPairISD_SI_EEEEEE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc25InProgressFullLookupState8completeESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession19OL_applyQueryPhase1ESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EENS_5ErrorE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS0_10LookupKindERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS8_EENS0_15SymbolLookupSetENS0_11SymbolStateENS_15unique_functionIFvNS_8ExpectedINS_8DenseMapINS0_15SymbolStringPtrENS_18JITEvaluatedSymbolENS_12DenseMapInfoISI_vEENS_6detail12DenseMapPairISI_SJ_EEEEEEEEESt8functionIFvRKNSH_IS6_NS_8DenseSetISI_SL_EENSK_IS6_vEENSN_IS6_SV_EEEEEE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS7_EERKNS0_15SymbolLookupSetENS0_10LookFailed to precompile MPI [da04e1cc-30fd-572f-bb4f-1f8673147195] to "/home/mose/.julia/compiled/v1.10/MPI/jl_JDDt85".
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
   @ Base ./loading.jl:2145
 [3] compilecache
   @ ./loading.jl:2018 [inlined]
 [4] _require(pkg::Base.PkgId, env::String)
   @ Base ./loading.jl:1662
 [5] _require_prelocked(uuidkey::Base.PkgId, env::String)
   @ Base ./loading.jl:1517
 [6] macro expansion
   @ ./loading.jl:1505 [inlined]
 [7] macro expansion
   @ ./lock.jl:267 [inlined]
 [8] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1468
in expression starting at /home/mose/.julia/dev/MPI/docs/examples/01-hello.jl:1
upKindENS0_11SymbolStateESt8functionIFvRKNS_8DenseMapIS5_NS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISK_vEEEENSL_IS5_vEENS_6detail12DenseMapPairIS5_SN_EEEEEE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
addModule at /home/mose/repo/julia/src/jitlayers.cpp:1401
jl_add_to_ee at /home/mose/repo/julia/src/jitlayers.cpp:1798
jl_add_to_ee at /home/mose/repo/julia/src/jitlayers.cpp:1777
_jl_compile_codeinst at /home/mose/repo/julia/src/jitlayers.cpp:241
jl_generate_fptr_impl at /home/mose/repo/julia/src/jitlayers.cpp:447
jl_compile_method_internal at /home/mose/repo/julia/src/gf.c:2280 [inlined]
jl_compile_method_internal at /home/mose/repo/julia/src/gf.c:2198
_jl_invoke at /home/mose/repo/julia/src/gf.c:2675 [inlined]
ijl_apply_generic at /home/mose/repo/julia/src/gf.c:2865
#showerror#902 at ./errorshow.jl:96
kwcall at ./errorshow.jl:94
unknown function (ip: 0x7fa294703ce2)
show_exception_stack at ./errorshow.jl:894
display_error at ./client.jl:111
unknown function (ip: 0x7fa2947037e6)
display_error at ./client.jl:114
jfptr_display_error_38802 at /home/mose/repo/julia/usr/lib/julia/sys.so (unknown line)
jl_apply at /home/mose/repo/julia/src/julia.h:1872 [inlined]
jl_f__call_latest at /home/mose/repo/julia/src/builtins.c:778
#invokelatest#2 at ./essentials.jl:818 [inlined]
invokelatest at ./essentials.jl:815 [inlined]
exec_options at ./client.jl:310
_start at ./client.jl:522
jfptr__start_40790 at /home/mose/repo/julia/usr/lib/julia/sys.so (unknown line)
jl_apply at /home/mose/repo/julia/src/julia.h:1872 [inlined]
true_main at /home/mose/repo/julia/src/jlapi.c:573
jl_repl_entrypoint at /home/mose/repo/julia/src/jlapi.c:717
main at /home/mose/repo/julia/cli/loader_exe.c:58
unknown function (ip: 0x7fa2aa99828f)
__libc_start_main at /usr/lib/libc.so.6 (unknown line)
_start at /build/glibc/src/glibc/csu/../sysdeps/x86_64/start.S:115
unknown function (ip: (nil))
Allocations: 604992 (Pool: 604182; Big: 810); GC: 1
--------------------------------------------------------------------------
mpiexec detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[47321,1],3]
  Exit code:    1
--------------------------------------------------------------------------
ERROR: failed process: Process(`/home/mose/.julia/artifacts/65763fcda97bdafc42a83deacb687015551f72d1/bin/mpiexec -n 4 /home/mose/repo/julia/usr/bin/julia --startup-file=no -q /home/mose/.julia/dev/MPI/test/../docs/examples/01-hello.jl`, ProcessExited(1)) [1]

Stacktrace:
  [1] pipeline_error
    @ ./process.jl:565 [inlined]
  [2] run(::Cmd; wait::Bool)
    @ Base ./process.jl:480
  [3] run(::Cmd)
    @ Base process.jl:477
  [4] (::var"#3#4")(exe::String)
    @ Main none:4
  [5] (::JLLWrappers.var"#2#3"{var"#3#4", String})()
    @ JLLWrappers ~/.julia/packages/JLLWrappers/QpMQW/src/runtime.jl:49
  [6] withenv(::JLLWrappers.var"#2#3"{var"#3#4", String}, ::Pair{String, String}, ::Vararg{Pair{String, String}})
    @ Base env.jl:235
  [7] withenv_executable_wrapper(f::Function, executable_path::String, PATH::String, LIBPATH::String, adjust_PATH::Bool, adjust_LIBPATH::Bool)
    @ JLLWrappers ~/.julia/packages/JLLWrappers/QpMQW/src/runtime.jl:48
  [8] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Base ./essentials.jl:818
  [9] invokelatest(::Any, ::Any, ::Vararg{Any})
    @ Base ./essentials.jl:815
 [10] mpiexec(f::Function; adjust_PATH::Bool, adjust_LIBPATH::Bool)
    @ OpenMPI_jll ~/.julia/packages/JLLWrappers/QpMQW/src/products/executable_generators.jl:21
 [11] mpiexec(f::Function)
    @ OpenMPI_jll ~/.julia/packages/JLLWrappers/QpMQW/src/products/executable_generators.jl:19
 [12] top-level scope
    @ none:4
mpiexecjl: Error During Test at /home/mose/.julia/dev/MPI/test/mpiexecjl.jl:4
  Got exception outside of a @test
  failed process: Process(`/tmp/jl_yInzWd/mpiexecjl -n 4 --project=/tmp/jl_yInzWd /home/mose/repo/julia/usr/bin/julia --startup-file=no -q /home/mose/.julia/dev/MPI/test/../docs/examples/01-hello.jl`, ProcessExited(1)) [1]
  
  Stacktrace:
    [1] pipeline_error
      @ ./process.jl:565 [inlined]
    [2] run(::Cmd; wait::Bool)
      @ Base ./process.jl:480
    [3] run
      @ ./process.jl:477 [inlined]
    [4] #5
      @ ~/.julia/dev/MPI/test/mpiexecjl.jl:20 [inlined]
    [5] withenv(f::var"#5#11"{String, String, String, String, Int64}, keyvals::Pair{String, String})
      @ Base ./env.jl:235
    [6] (::var"#3#9")(dir::String)
      @ Main ~/.julia/dev/MPI/test/mpiexecjl.jl:19
    [7] mktempdir(fn::var"#3#9", parent::String; prefix::String)
      @ Base.Filesystem ./file.jl:760
    [8] mktempdir(fn::Function, parent::String)
      @ Base.Filesystem ./file.jl:756
    [9] mktempdir(fn::Function)
      @ Base.Filesystem ./file.jl:756
   [10] macro expansion
      @ ~/.julia/dev/MPI/test/mpiexecjl.jl:5 [inlined]
   [11] macro expansion
      @ ~/repo/julia/usr/share/julia/stdlib/v1.10/Test/src/Test.jl:1502 [inlined]
   [12] top-level scope
      @ ~/.julia/dev/MPI/test/mpiexecjl.jl:5
   [13] include(fname::String)
      @ Base.MainInclude ./client.jl:478
   [14] top-level scope
      @ ~/.julia/dev/MPI/test/runtests.jl:36
   [15] include(fname::String)
      @ Base.MainInclude ./client.jl:478
   [16] top-level scope
      @ none:6
   [17] eval
      @ ./boot.jl:370 [inlined]
   [18] exec_options(opts::Base.JLOptions)
      @ Base ./client.jl:280
   [19] _start()
      @ Base ./client.jl:522
Test Summary: | Pass  Error  Total   Time
mpiexecjl     |    1      1      2  25.8s
ERROR: LoadError: Some tests did not pass: 1 passed, 0 failed, 1 errored, 0 broken.
in expression starting at /home/mose/.julia/dev/MPI/test/mpiexecjl.jl:4
in expression starting at /home/mose/.julia/dev/MPI/test/runtests.jl:33
ERROR: Package MPI errored during testing

vchuravy · 2023-01-13T11:22:42Z

Hm, can you run with JULIA_DEBUG=loading this looks like a cache race. Similar to #48217

giordano · 2023-01-13T12:02:07Z

Details

% JULIA_DEBUG=loading julia-master --project --startup-file=no -q
(MPI) pkg> test
     Testing MPI
      Status `/tmp/jl_Le86MK/Project.toml`
  [21141c5a] AMDGPU v0.4.4
  [052768ef] CUDA v3.12.1
  [497a8b3b] DoubleFloats v1.2.2
  [da04e1cc] MPI v0.20.7 `~/.julia/dev/MPI`
  [3da0fdf6] MPIPreferences v0.1.7 `~/.julia/dev/MPI/lib/MPIPreferences`
  [44cfe95a] Pkg v1.9.0 `@stdlib/Pkg`
  [9a3f8284] Random `@stdlib/Random`
  [fa267f1f] TOML v1.0.3 `@stdlib/TOML`
  [8dfed614] Test `@stdlib/Test`
      Status `/tmp/jl_Le86MK/Manifest.toml`
  [21141c5a] AMDGPU v0.4.4
  [621f4979] AbstractFFTs v1.2.1
  [79e6a3ab] Adapt v3.4.0
⌅ [ab4f0b2a] BFloat16s v0.2.0
  [b99e7846] BinaryProvider v0.5.10
  [fa961155] CEnum v0.4.2
  [052768ef] CUDA v3.12.1
  [d360d2e6] ChainRulesCore v1.15.6
  [9e997f8a] ChangesOfVariables v0.1.4
  [34da2185] Compat v4.5.0
  [187b0558] ConstructionBase v1.4.1
⌃ [ffbed154] DocStringExtensions v0.9.1
  [497a8b3b] DoubleFloats v1.2.2
  [e2ba6199] ExprTools v0.1.8
  [0c68f7d7] GPUArrays v8.5.0
  [46192b85] GPUArraysCore v0.1.2
⌅ [61eb1bfa] GPUCompiler v0.16.7
  [14197337] GenericLinearAlgebra v0.3.5
  [3587e190] InverseFunctions v0.1.8
  [92d709cd] IrrationalConstants v0.1.1
  [692b3bcd] JLLWrappers v1.4.1
  [929cbde3] LLVM v4.14.1
  [2ab3a3ac] LogExpFunctions v0.3.19
  [da04e1cc] MPI v0.20.7 `~/.julia/dev/MPI`
  [3da0fdf6] MPIPreferences v0.1.7 `~/.julia/dev/MPI/lib/MPIPreferences`
  [1914dd2f] MacroTools v0.5.10
  [f27b6e38] Polynomials v3.2.2
  [21216c6a] Preferences v1.3.0
  [be4d8f0f] Quadmath v0.5.6
  [74087812] Random123 v1.6.0
  [e6cf234a] RandomNumbers v1.5.3
  [3cdcf5f2] RecipesBase v1.3.3
  [189a3867] Reexport v1.2.2
  [ae029012] Requires v1.3.0
  [efcf1570] Setfield v1.1.1
  [66db9d55] SnoopPrecompile v1.0.3
  [276daf66] SpecialFunctions v2.1.7
  [1e83bf80] StaticArraysCore v1.4.0
  [a759f4b9] TimerOutputs v0.5.22
  [6e34b625] Bzip2_jll v1.0.8+0
  [ab5a07f8] Elfutils_jll v0.182.0+1
  [2696aab5] HIP_jll v5.2.3+2
  [dad2f222] LLVMExtra_jll v0.0.16+0
  [86de99a1] LLVM_jll v14.0.6+1
  [d4300ac3] Libgcrypt_jll v1.8.7+0
  [7e76a0d4] Libglvnd_jll v1.6.0+0
  [7add5ba3] Libgpg_error_jll v1.42.0+0
  [94ce4f54] Libiconv_jll v1.16.1+2
  [7cb0a576] MPICH_jll v4.0.2+5
  [f1f71cc9] MPItrampoline_jll v5.0.2+1
  [9237b28f] MicrosoftMPI_jll v10.1.3+2
  [7f51dc2b] NUMA_jll v2.0.14+0
  [fe0851c0] OpenMPI_jll v4.1.3+3
  [efe28fd5] OpenSpecFun_jll v0.5.5+0
  [8fbdd1d2] ROCmCompilerSupport_jll v5.2.3+1
  [873c0968] ROCmDeviceLibs_jll v5.2.3+0
  [10ae2a08] ROCmOpenCLRuntime_jll v5.2.3+1
  [02c8fc9c] XML2_jll v2.10.3+0
  [aed1982a] XSLT_jll v1.1.34+0
  [ffd25f8a] XZ_jll v5.2.9+0
  [4f6342f7] Xorg_libX11_jll v1.6.9+4
  [0c0b7dd1] Xorg_libXau_jll v1.0.9+4
  [a3789734] Xorg_libXdmcp_jll v1.1.3+4
  [1082639a] Xorg_libXext_jll v1.3.4+4
  [a65dc6b1] Xorg_libpciaccess_jll v0.16.0+1
  [14d82f49] Xorg_libpthread_stubs_jll v0.1.0+3
  [c7cfdc94] Xorg_libxcb_jll v1.13.0+3
  [c4d99508] Xorg_xorgproto_jll v2019.2.0+2
  [c5fb5394] Xorg_xtrans_jll v1.4.0+3
  [c53206cc] argp_standalone_jll v1.3.1+0
  [d65627f6] fts_jll v1.2.8+0
  [dd59ff1a] hsa_rocr_jll v5.2.3+0
  [1cecccd7] hsakmt_roct_jll v5.2.3+0
  [8e53e030] libdrm_jll v2.4.110+0
  [43dd8cde] msgpack_jll v3.0.1+0
  [c88a4935] obstack_jll v1.2.2+0
  [1ef8cab2] rocBLAS_jll v5.2.3+2
  [52935e6f] rocPRIM_jll v5.2.3+1
  [a6151927] rocRAND_jll v5.2.3+1
  [8c6ce2ba] rocSPARSE_jll v5.2.3+1
  [5a766526] rocminfo_jll v5.2.3+0
  [0dad84c5] ArgTools v1.1.1 `@stdlib/ArgTools`
  [56f22d72] Artifacts `@stdlib/Artifacts`
  [2a0f44e3] Base64 `@stdlib/Base64`
  [ade2ca70] Dates `@stdlib/Dates`
  [8ba89e20] Distributed `@stdlib/Distributed`
  [f43a241f] Downloads v1.6.0 `@stdlib/Downloads`
  [7b1f6079] FileWatching `@stdlib/FileWatching`
  [9fa8497b] Future `@stdlib/Future`
  [b77e0a4c] InteractiveUtils `@stdlib/InteractiveUtils`
  [4af54fe1] LazyArtifacts `@stdlib/LazyArtifacts`
  [b27032c2] LibCURL v0.6.3 `@stdlib/LibCURL`
  [76f85450] LibGit2 `@stdlib/LibGit2`
  [8f399da3] Libdl `@stdlib/Libdl`
  [37e2e46d] LinearAlgebra `@stdlib/LinearAlgebra`
  [56ddb016] Logging `@stdlib/Logging`
  [d6f4376e] Markdown `@stdlib/Markdown`
  [ca575930] NetworkOptions v1.2.0 `@stdlib/NetworkOptions`
  [44cfe95a] Pkg v1.9.0 `@stdlib/Pkg`
  [de0858da] Printf `@stdlib/Printf`
  [3fa0cd96] REPL `@stdlib/REPL`
  [9a3f8284] Random `@stdlib/Random`
  [ea8e919c] SHA v0.7.0 `@stdlib/SHA`
  [9e88b42a] Serialization `@stdlib/Serialization`
  [6462fe0b] Sockets `@stdlib/Sockets`
  [2f01184e] SparseArrays `@stdlib/SparseArrays`
  [10745b16] Statistics v1.9.0 `@stdlib/Statistics`
  [fa267f1f] TOML v1.0.3 `@stdlib/TOML`
  [a4e569a6] Tar v1.10.0 `@stdlib/Tar`
  [8dfed614] Test `@stdlib/Test`
  [cf7118a7] UUIDs `@stdlib/UUIDs`
  [4ec0a83e] Unicode `@stdlib/Unicode`
  [e66e0078] CompilerSupportLibraries_jll v1.0.2+0 `@stdlib/CompilerSupportLibraries_jll`
  [d55e3150] LLD_jll v14.0.5+3 `@stdlib/LLD_jll`
  [deac9b47] LibCURL_jll v7.84.0+0 `@stdlib/LibCURL_jll`
  [29816b5a] LibSSH2_jll v1.10.2+0 `@stdlib/LibSSH2_jll`
  [c8ffd9c3] MbedTLS_jll v2.28.0+0 `@stdlib/MbedTLS_jll`
  [14a3606d] MozillaCACerts_jll v2022.10.11 `@stdlib/MozillaCACerts_jll`
  [4536629a] OpenBLAS_jll v0.3.21+0 `@stdlib/OpenBLAS_jll`
  [05823500] OpenLibm_jll v0.8.1+0 `@stdlib/OpenLibm_jll`
  [bea87d4a] SuiteSparse_jll v5.10.1+0 `@stdlib/SuiteSparse_jll`
  [83775a58] Zlib_jll v1.2.13+0 `@stdlib/Zlib_jll`
  [8f36deef] libLLVM_jll v14.0.6+0 `@stdlib/libLLVM_jll`
  [8e850b90] libblastrampoline_jll v5.2.0+0 `@stdlib/libblastrampoline_jll`
  [8e850ede] nghttp2_jll v1.48.0+0 `@stdlib/nghttp2_jll`
  [3f19e933] p7zip_jll v17.4.0+0 `@stdlib/p7zip_jll`
        Info Packages marked with ⌃ and ⌅ have new versions available, but those with ⌅ are restricted by compatibility constraints from upgrading.
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/Preferences/pWSk8_flz9e.ji for  [top-level] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 3, check_bounds = true, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
└ @ Base loading.jl:2570
Precompiling environment...
  89 dependencies successfully precompiled in 114 seconds
     Testing Running tests...
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/Requires/IyxeS_tEWw4.so for Requires [ae029012-a4dd-5104-9daa-d747884805df]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/DocStringExtensions/KRdZs_tEWw4.so for DocStringExtensions [ffbed154-4ef7-542d-bbb7-c09d3a79fcae]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/Preferences/pWSk8_tEWw4.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/MPIPreferences/rGk9Z_0mbTD.so for MPIPreferences [3da0fdf6-3ccc-4f1b-acd9-58baa6c99267]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/JLLWrappers/7Zgw7_tEWw4.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/OpenMPI_jll/UdERG_tEWw4.so for OpenMPI_jll [fe0851c0-eecd-5654-98d4-656369965a5c]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/MPI/nO0XF_tEWw4.so for MPI [da04e1cc-30fd-572f-bb4f-1f8673147195]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/SnoopPrecompile/TcfDT_tEWw4.so for SnoopPrecompile [66db9d55-30c0-4569-8b51-7e840670fc0c]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/RecipesBase/8e2Mm_tEWw4.so for RecipesBase [3cdcf5f2-1ef4-517c-9805-6587b60abb01]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/Polynomials/OaK78_tEWw4.so for Polynomials [f27b6e38-b328-58d1-80ce-0feddd5e7a45]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/GenericLinearAlgebra/Tm5A3_tEWw4.so for GenericLinearAlgebra [14197337-ba66-59df-a3e3-ca00e7dcff7a]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/Quadmath/q52id_tEWw4.so for Quadmath [be4d8f0f-7fa4-5f49-b795-2f01399ab2dd]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/IrrationalConstants/ukdUG_tEWw4.so for IrrationalConstants [92d709cd-6900-40b7-9082-c6be49f344b6]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/Compat/GSFWK_tEWw4.so for Compat [34da2185-b29b-5c13-b0c7-acf172513d20]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/ChainRulesCore/G6ax7_tEWw4.so for ChainRulesCore [d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/ChangesOfVariables/wqbay_tEWw4.so for ChangesOfVariables [9e997f8a-9a97-42d5-a9f1-ce6bfc15e2c0]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/InverseFunctions/PkVmn_tEWw4.so for InverseFunctions [3587e190-3f89-42d0-90ee-14403ec27112]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/LogExpFunctions/cmCYR_tEWw4.so for LogExpFunctions [2ab3a3ac-af41-5b50-aa03-7779005ae688]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/OpenLibm_jll/ToVO1_tEWw4.so for OpenLibm_jll [05823500-19ac-5b8b-9628-191a04bc5112]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/OpenSpecFun_jll/TDl1L_tEWw4.so for OpenSpecFun_jll [efe28fd5-8261-553b-a9e1-b2916fc3738e]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/SpecialFunctions/78gOt_tEWw4.so for SpecialFunctions [276daf66-3868-5448-9aa4-cd146d93841b]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/DoubleFloats/KzTCm_tEWw4.so for DoubleFloats [497a8b3b-efae-58df-a0af-a86822472b78]
└ @ Base loading.jl:950
┌ Info: Running MPI tests
│   ArrayType = Array
│   nprocs = 4
│   MPIPreferences.abi = "OpenMPI"
└   MPIPreferences.binary = "OpenMPI_jll"
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/MPI/nO0XF_tEWw4.ji for MPI [da04e1cc-30fd-572f-bb4f-1f8673147195] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 0
│   cache file:      use_pkgimages = true, debug_level = 3, check_bounds = true, opt_level = 2
└ @ Base loading.jl:2570
┌ Debug: Precompiling MPI [da04e1cc-30fd-572f-bb4f-1f8673147195]
└ @ Base loading.jl:2031
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/Requires/IyxeS_tEWw4.ji for Requires [ae029012-a4dd-5104-9daa-d747884805df] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 0
│   cache file:      use_pkgimages = true, debug_level = 3, check_bounds = true, opt_level = 2
└ @ Base loading.jl:2570
┌ Debug: Precompiling Requires [ae029012-a4dd-5104-9daa-d747884805df]
└ @ Base loading.jl:2031
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/Requires/IyxeS_ozwjF.so for Requires [ae029012-a4dd-5104-9daa-d747884805df]
└ @ Base loading.jl:950
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/DocStringExtensions/KRdZs_tEWw4.ji for DocStringExtensions [ffbed154-4ef7-542d-bbb7-c09d3a79fcae] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 0
│   cache file:      use_pkgimages = true, debug_level = 3, check_bounds = true, opt_level = 2
└ @ Base loading.jl:2570
┌ Debug: Precompiling DocStringExtensions [ffbed154-4ef7-542d-bbb7-c09d3a79fcae]
└ @ Base loading.jl:2031
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/DocStringExtensions/KRdZs_ozwjF.so for DocStringExtensions [ffbed154-4ef7-542d-bbb7-c09d3a79fcae]
└ @ Base loading.jl:950
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/MPIPreferences/rGk9Z_0mbTD.ji for MPIPreferences [3da0fdf6-3ccc-4f1b-acd9-58baa6c99267] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 0
│   cache file:      use_pkgimages = true, debug_level = 3, check_bounds = true, opt_level = 2
└ @ Base loading.jl:2570
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/Preferences/pWSk8_tEWw4.ji for Preferences [21216c6a-2e73-6563-6e65-726566657250] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 0
│   cache file:      use_pkgimages = true, debug_level = 3, check_bounds = true, opt_level = 2
└ @ Base loading.jl:2570
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/Preferences/pWSk8_flz9e.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/MPIPreferences/rGk9Z_yde3t.so for MPIPreferences [3da0fdf6-3ccc-4f1b-acd9-58baa6c99267]
└ @ Base loading.jl:950
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/OpenMPI_jll/UdERG_tEWw4.ji for OpenMPI_jll [fe0851c0-eecd-5654-98d4-656369965a5c] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 0
│   cache file:      use_pkgimages = true, debug_level = 3, check_bounds = true, opt_level = 2
└ @ Base loading.jl:2570
┌ Debug: Precompiling OpenMPI_jll [fe0851c0-eecd-5654-98d4-656369965a5c]
└ @ Base loading.jl:2031
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/Preferences/pWSk8_tEWw4.ji for Preferences [21216c6a-2e73-6563-6e65-726566657250] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 0
│   cache file:      use_pkgimages = true, debug_level = 3, check_bounds = true, opt_level = 2
└ @ Base loading.jl:2570
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/Preferences/pWSk8_flz9e.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/MPIPreferences/rGk9Z_yde3t.so for MPIPreferences [3da0fdf6-3ccc-4f1b-acd9-58baa6c99267]
└ @ Base loading.jl:950
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/JLLWrappers/7Zgw7_tEWw4.ji for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 0
│   cache file:      use_pkgimages = true, debug_level = 3, check_bounds = true, opt_level = 2
└ @ Base loading.jl:2570
┌ Debug: Precompiling JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:2031
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/Preferences/pWSk8_tEWw4.ji for Preferences [21216c6a-2e73-6563-6e65-726566657250] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 0
│   cache file:      use_pkgimages = true, debug_level = 3, check_bounds = true, opt_level = 2
└ @ Base loading.jl:2570
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/Preferences/pWSk8_flz9e.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/JLLWrappers/7Zgw7_ozwjF.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/JLLWrappers/7Zgw7_ozwjF.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/OpenMPI_jll/UdERG_ozwjF.so for OpenMPI_jll [fe0851c0-eecd-5654-98d4-656369965a5c]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/Requires/IyxeS_ozwjF.so for Requires [ae029012-a4dd-5104-9daa-d747884805df]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/DocStringExtensions/KRdZs_ozwjF.so for DocStringExtensions [ffbed154-4ef7-542d-bbb7-c09d3a79fcae]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/Preferences/pWSk8_flz9e.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/MPIPreferences/rGk9Z_yde3t.so for MPIPreferences [3da0fdf6-3ccc-4f1b-acd9-58baa6c99267]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/JLLWrappers/7Zgw7_ozwjF.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/OpenMPI_jll/UdERG_ozwjF.so for OpenMPI_jll [fe0851c0-eecd-5654-98d4-656369965a5c]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/MPI/nO0XF_ozwjF.so for MPI [da04e1cc-30fd-572f-bb4f-1f8673147195]
└ @ Base loading.jl:950
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/MPI/nO0XF_ozwjF.ji for MPI [da04e1cc-30fd-572f-bb4f-1f8673147195] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 0
└ @ Base loading.jl:2570
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/MPI/nO0XF_tEWw4.ji for MPI [da04e1cc-30fd-572f-bb4f-1f8673147195] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 3, check_bounds = true, opt_level = 2
└ @ Base loading.jl:2570
┌ Debug: Precompiling MPI [da04e1cc-30fd-572f-bb4f-1f8673147195]
└ @ Base loading.jl:2031
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/MPI/nO0XF_ozwjF.ji for MPI [da04e1cc-30fd-572f-bb4f-1f8673147195] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 0
└ @ Base loading.jl:2570
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/MPI/nO0XF_tEWw4.ji for MPI [da04e1cc-30fd-572f-bb4f-1f8673147195] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 3, check_bounds = true, opt_level = 2
└ @ Base loading.jl:2570
┌ Debug: Precompiling MPI [da04e1cc-30fd-572f-bb4f-1f8673147195]
└ @ Base loading.jl:2031
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/MPI/nO0XF_ozwjF.ji for MPI [da04e1cc-30fd-572f-bb4f-1f8673147195] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 0
└ @ Base loading.jl:2570
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/MPI/nO0XF_tEWw4.ji for MPI [da04e1cc-30fd-572f-bb4f-1f8673147195] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 3, check_bounds = true, opt_level = 2
└ @ Base loading.jl:2570
┌ Debug: Precompiling MPI [da04e1cc-30fd-572f-bb4f-1f8673147195]
└ @ Base loading.jl:2031
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/MPI/nO0XF_ozwjF.ji for MPI [da04e1cc-30fd-572f-bb4f-1f8673147195] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 0
└ @ Base loading.jl:2570
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/MPI/nO0XF_tEWw4.ji for MPI [da04e1cc-30fd-572f-bb4f-1f8673147195] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 3, check_bounds = true, opt_level = 2
└ @ Base loading.jl:2570
┌ Debug: Precompiling MPI [da04e1cc-30fd-572f-bb4f-1f8673147195]
└ @ Base loading.jl:2031
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/Requires/IyxeS_ozwjF.ji for Requires [ae029012-a4dd-5104-9daa-d747884805df] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 0
└ @ Base loading.jl:2570
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/Requires/IyxeS_tEWw4.ji for Requires [ae029012-a4dd-5104-9daa-d747884805df] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 3, check_bounds = true, opt_level = 2
└ @ Base loading.jl:2570
┌ Debug: Precompiling Requires [ae029012-a4dd-5104-9daa-d747884805df]
└ @ Base loading.jl:2031
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/Requires/IyxeS_ozwjF.ji for Requires [ae029012-a4dd-5104-9daa-d747884805df] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 0
└ @ Base loading.jl:2570
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/Requires/IyxeS_tEWw4.ji for Requires [ae029012-a4dd-5104-9daa-d747884805df] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 3, check_bounds = true, opt_level = 2
└ @ Base loading.jl:2570
┌ Debug: Precompiling Requires [ae029012-a4dd-5104-9daa-d747884805df]
└ @ Base loading.jl:2031
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/Requires/IyxeS_ozwjF.ji for Requires [ae029012-a4dd-5104-9daa-d747884805df] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 0
└ @ Base loading.jl:2570
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/Requires/IyxeS_ozwjF.ji for Requires [ae029012-a4dd-5104-9daa-d747884805df] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 0
└ @ Base loading.jl:2570
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/Requires/IyxeS_tEWw4.ji for Requires [ae029012-a4dd-5104-9daa-d747884805df] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 3, check_bounds = true, opt_level = 2
└ @ Base loading.jl:2570
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/Requires/IyxeS_tEWw4.ji for Requires [ae029012-a4dd-5104-9daa-d747884805df] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 3, check_bounds = true, opt_level = 2
└ @ Base loading.jl:2570
┌ Debug: Precompiling Requires [ae029012-a4dd-5104-9daa-d747884805df]
└ @ Base loading.jl:2031
┌ Debug: Precompiling Requires [ae029012-a4dd-5104-9daa-d747884805df]
└ @ Base loading.jl:2031
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/Requires/IyxeS_6Oy6w.so for Requires [ae029012-a4dd-5104-9daa-d747884805df]
└ @ Base loading.jl:950
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/DocStringExtensions/KRdZs_ozwjF.ji for DocStringExtensions [ffbed154-4ef7-542d-bbb7-c09d3a79fcae] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 0
└ @ Base loading.jl:2570
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/DocStringExtensions/KRdZs_tEWw4.ji for DocStringExtensions [ffbed154-4ef7-542d-bbb7-c09d3a79fcae] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 3, check_bounds = true, opt_level = 2
└ @ Base loading.jl:2570
┌ Debug: Precompiling DocStringExtensions [ffbed154-4ef7-542d-bbb7-c09d3a79fcae]
└ @ Base loading.jl:2031
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/Requires/IyxeS_6Oy6w.so for Requires [ae029012-a4dd-5104-9daa-d747884805df]
└ @ Base loading.jl:950
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/DocStringExtensions/KRdZs_ozwjF.ji for DocStringExtensions [ffbed154-4ef7-542d-bbb7-c09d3a79fcae] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 0
└ @ Base loading.jl:2570
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/DocStringExtensions/KRdZs_tEWw4.ji for DocStringExtensions [ffbed154-4ef7-542d-bbb7-c09d3a79fcae] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 3, check_bounds = true, opt_level = 2
└ @ Base loading.jl:2570
┌ Debug: Precompiling DocStringExtensions [ffbed154-4ef7-542d-bbb7-c09d3a79fcae]
└ @ Base loading.jl:2031
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/Requires/IyxeS_6Oy6w.so for Requires [ae029012-a4dd-5104-9daa-d747884805df]
└ @ Base loading.jl:950
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/DocStringExtensions/KRdZs_ozwjF.ji for DocStringExtensions [ffbed154-4ef7-542d-bbb7-c09d3a79fcae] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 0
└ @ Base loading.jl:2570
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/DocStringExtensions/KRdZs_tEWw4.ji for DocStringExtensions [ffbed154-4ef7-542d-bbb7-c09d3a79fcae] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 3, check_bounds = true, opt_level = 2
└ @ Base loading.jl:2570
┌ Debug: Precompiling DocStringExtensions [ffbed154-4ef7-542d-bbb7-c09d3a79fcae]
└ @ Base loading.jl:2031
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/Requires/IyxeS_6Oy6w.so for Requires [ae029012-a4dd-5104-9daa-d747884805df]
└ @ Base loading.jl:950
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/DocStringExtensions/KRdZs_ozwjF.ji for DocStringExtensions [ffbed154-4ef7-542d-bbb7-c09d3a79fcae] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 0
└ @ Base loading.jl:2570
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/DocStringExtensions/KRdZs_tEWw4.ji for DocStringExtensions [ffbed154-4ef7-542d-bbb7-c09d3a79fcae] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 3, check_bounds = true, opt_level = 2
└ @ Base loading.jl:2570
┌ Debug: Precompiling DocStringExtensions [ffbed154-4ef7-542d-bbb7-c09d3a79fcae]
└ @ Base loading.jl:2031
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/DocStringExtensions/KRdZs_6Oy6w.so for DocStringExtensions [ffbed154-4ef7-542d-bbb7-c09d3a79fcae]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/Preferences/pWSk8_flz9e.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/DocStringExtensions/KRdZs_6Oy6w.so for DocStringExtensions [ffbed154-4ef7-542d-bbb7-c09d3a79fcae]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/MPIPreferences/rGk9Z_yde3t.so for MPIPreferences [3da0fdf6-3ccc-4f1b-acd9-58baa6c99267]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/DocStringExtensions/KRdZs_6Oy6w.so for DocStringExtensions [ffbed154-4ef7-542d-bbb7-c09d3a79fcae]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/DocStringExtensions/KRdZs_6Oy6w.so for DocStringExtensions [ffbed154-4ef7-542d-bbb7-c09d3a79fcae]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/Preferences/pWSk8_flz9e.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:950
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/OpenMPI_jll/UdERG_ozwjF.ji for OpenMPI_jll [fe0851c0-eecd-5654-98d4-656369965a5c] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 0
└ @ Base loading.jl:2570
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/OpenMPI_jll/UdERG_tEWw4.ji for OpenMPI_jll [fe0851c0-eecd-5654-98d4-656369965a5c] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 3, check_bounds = true, opt_level = 2
└ @ Base loading.jl:2570
┌ Debug: Precompiling OpenMPI_jll [fe0851c0-eecd-5654-98d4-656369965a5c]
└ @ Base loading.jl:2031
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/MPIPreferences/rGk9Z_yde3t.so for MPIPreferences [3da0fdf6-3ccc-4f1b-acd9-58baa6c99267]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/Preferences/pWSk8_flz9e.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/Preferences/pWSk8_flz9e.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/MPIPreferences/rGk9Z_yde3t.so for MPIPreferences [3da0fdf6-3ccc-4f1b-acd9-58baa6c99267]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/MPIPreferences/rGk9Z_yde3t.so for MPIPreferences [3da0fdf6-3ccc-4f1b-acd9-58baa6c99267]
└ @ Base loading.jl:950
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/OpenMPI_jll/UdERG_ozwjF.ji for OpenMPI_jll [fe0851c0-eecd-5654-98d4-656369965a5c] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 0
└ @ Base loading.jl:2570
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/OpenMPI_jll/UdERG_tEWw4.ji for OpenMPI_jll [fe0851c0-eecd-5654-98d4-656369965a5c] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 3, check_bounds = true, opt_level = 2
└ @ Base loading.jl:2570
┌ Debug: Precompiling OpenMPI_jll [fe0851c0-eecd-5654-98d4-656369965a5c]
└ @ Base loading.jl:2031
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/OpenMPI_jll/UdERG_ozwjF.ji for OpenMPI_jll [fe0851c0-eecd-5654-98d4-656369965a5c] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 0
└ @ Base loading.jl:2570
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/OpenMPI_jll/UdERG_tEWw4.ji for OpenMPI_jll [fe0851c0-eecd-5654-98d4-656369965a5c] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 3, check_bounds = true, opt_level = 2
└ @ Base loading.jl:2570
┌ Debug: Precompiling OpenMPI_jll [fe0851c0-eecd-5654-98d4-656369965a5c]
└ @ Base loading.jl:2031
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/OpenMPI_jll/UdERG_ozwjF.ji for OpenMPI_jll [fe0851c0-eecd-5654-98d4-656369965a5c] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 0
└ @ Base loading.jl:2570
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/OpenMPI_jll/UdERG_tEWw4.ji for OpenMPI_jll [fe0851c0-eecd-5654-98d4-656369965a5c] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 3, check_bounds = true, opt_level = 2
└ @ Base loading.jl:2570
┌ Debug: Precompiling OpenMPI_jll [fe0851c0-eecd-5654-98d4-656369965a5c]
└ @ Base loading.jl:2031
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/Preferences/pWSk8_flz9e.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/MPIPreferences/rGk9Z_yde3t.so for MPIPreferences [3da0fdf6-3ccc-4f1b-acd9-58baa6c99267]
└ @ Base loading.jl:950
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/JLLWrappers/7Zgw7_ozwjF.ji for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 0
└ @ Base loading.jl:2570
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/JLLWrappers/7Zgw7_tEWw4.ji for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 3, check_bounds = true, opt_level = 2
└ @ Base loading.jl:2570
┌ Debug: Precompiling JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:2031
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/Preferences/pWSk8_flz9e.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/MPIPreferences/rGk9Z_yde3t.so for MPIPreferences [3da0fdf6-3ccc-4f1b-acd9-58baa6c99267]
└ @ Base loading.jl:950
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/JLLWrappers/7Zgw7_ozwjF.ji for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 0
└ @ Base loading.jl:2570
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/JLLWrappers/7Zgw7_tEWw4.ji for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 3, check_bounds = true, opt_level = 2
└ @ Base loading.jl:2570
┌ Debug: Precompiling JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:2031
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/Preferences/pWSk8_flz9e.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/Preferences/pWSk8_flz9e.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/MPIPreferences/rGk9Z_yde3t.so for MPIPreferences [3da0fdf6-3ccc-4f1b-acd9-58baa6c99267]
└ @ Base loading.jl:950
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/JLLWrappers/7Zgw7_ozwjF.ji for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 0
└ @ Base loading.jl:2570
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/JLLWrappers/7Zgw7_tEWw4.ji for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 3, check_bounds = true, opt_level = 2
└ @ Base loading.jl:2570
┌ Debug: Precompiling JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:2031
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/MPIPreferences/rGk9Z_yde3t.so for MPIPreferences [3da0fdf6-3ccc-4f1b-acd9-58baa6c99267]
└ @ Base loading.jl:950
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/JLLWrappers/7Zgw7_ozwjF.ji for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 0
└ @ Base loading.jl:2570
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/JLLWrappers/7Zgw7_tEWw4.ji for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 3, check_bounds = true, opt_level = 2
└ @ Base loading.jl:2570
┌ Debug: Precompiling JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:2031
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/Preferences/pWSk8_flz9e.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/Preferences/pWSk8_flz9e.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/Preferences/pWSk8_flz9e.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/Preferences/pWSk8_flz9e.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/JLLWrappers/7Zgw7_6Oy6w.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/JLLWrappers/7Zgw7_6Oy6w.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/JLLWrappers/7Zgw7_6Oy6w.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/JLLWrappers/7Zgw7_6Oy6w.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:950
┌ Debug: Ignoring cache file /home/mose/.julia/compiled/v1.10/JLLWrappers/7Zgw7_6Oy6w.ji for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] (fafbfcfd-fda9-a69c-0000-08197d4c2056) since it is does not provide desired build_id (fafbfcfd-95bd-ae9c-0000-081977e86929)
└ @ Base loading.jl:2604
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/JLLWrappers/7Zgw7_ozwjF.ji for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 0
└ @ Base loading.jl:2570
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/JLLWrappers/7Zgw7_tEWw4.ji for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 3, check_bounds = true, opt_level = 2
└ @ Base loading.jl:2570
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for OpenMPI_jll [fe0851c0-eecd-5654-98d4-656369965a5c]
│   exception = Required dependency JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] failed to load from a cache file.
└ @ Base loading.jl:1675
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/JLLWrappers/7Zgw7_6Oy6w.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:950
┌ Debug: Ignoring cache file /home/mose/.julia/compiled/v1.10/JLLWrappers/7Zgw7_6Oy6w.ji for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] (fafbfcfd-fda9-a69c-0000-08197d4c2056) since it is does not provide desired build_id (fafbfcfd-a1d5-0303-0000-08197ab9ce18)
└ @ Base loading.jl:2604
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/JLLWrappers/7Zgw7_ozwjF.ji for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 0
└ @ Base loading.jl:2570
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/JLLWrappers/7Zgw7_tEWw4.ji for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 3, check_bounds = true, opt_level = 2
└ @ Base loading.jl:2570
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for OpenMPI_jll [fe0851c0-eecd-5654-98d4-656369965a5c]
│   exception = Required dependency JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] failed to load from a cache file.
└ @ Base loading.jl:1675
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/JLLWrappers/7Zgw7_6Oy6w.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:950
┌ Debug: Ignoring cache file /home/mose/.julia/compiled/v1.10/JLLWrappers/7Zgw7_6Oy6w.ji for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] (fafbfcfd-fda9-a69c-0000-08197d4c2056) since it is does not provide desired build_id (fafbfcfd-726b-0ca9-0000-08197d0e3724)
└ @ Base loading.jl:2604
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/JLLWrappers/7Zgw7_ozwjF.ji for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 0
└ @ Base loading.jl:2570
┌ Debug: Rejecting cache file /home/mose/.julia/compiled/v1.10/JLLWrappers/7Zgw7_tEWw4.ji for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = false, opt_level = 2
│   cache file:      use_pkgimages = true, debug_level = 3, check_bounds = true, opt_level = 2
└ @ Base loading.jl:2570
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for OpenMPI_jll [fe0851c0-eecd-5654-98d4-656369965a5c]
│   exception = Required dependency JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] failed to load from a cache file.
└ @ Base loading.jl:1675
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/JLLWrappers/7Zgw7_6Oy6w.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/JLLWrappers/7Zgw7_6Oy6w.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:950
┌ Debug: Loading object cache file /home/mose/.julia/compiled/v1.10/OpenMPI_jll/UdERG_6Oy6w.so for OpenMPI_jll [fe0851c0-eecd-5654-98d4-656369965a5c]
└ @ Base loading.jl:950
ERROR: LoadError: ERROR: LoadError: ERROR: LoadError: ArgumentError: NULL library handle
Stacktrace:
  [1] #dlsym#1
    @ ./libdl.jl:57 [inlined]
  [2] kwcall(::NamedTuple{(:throw_error,), Tuple{Bool}}, ::typeof(Base.Libc.Libdl.dlsym), hnd::Ptr{Nothing}, s::Symbol)
    @ Base.Libc.Libdl ./libdl.jl:56
  [3] var"@mpichk"(__source__::LineNumberNode, __module__::Module, expr::Any, min_version::Any)
    @ MPI.API ~/.julia/dev/MPI/src/api/api.jl:123
  [4] #macroexpand#64
    @ ./expr.jl:119 [inlined]
  [5] macroexpand
    @ ./expr.jl:117 [inlined]
  [6] docm(source::LineNumberNode, mod::Module, meta::Any, ex::Any, define::Bool)
    @ Base.Docs ./docs/Docs.jl:538
  [7] docm(source::LineNumberNode, mod::Module, meta::Any, ex::Any)
    @ Base.Docs ./docs/Docs.jl:538
  [8] ArgumentError: NULL library handle
Stacktrace:
  [1] #dlsym#1
    @ ./libdl.jl:57 [inlined]
  [2] kwcall(::(::DocStringExtensions.var"#35#36"{typeof(DocStringExtensions.template_hook)})(::LineNumberNode, ::NamedTuple{(:throw_error,), Tuple{Bool}}, ::typeof(Base.Libc.Libdl.dlsym), hnd::Ptr{Nothing}, s::Symbol)
    @ Base.Libc.Libdl ./libdl.jl:56
  [3] var"@mpichk"(__source__::LineNumberNode, __module__::Module, expr::Any, min_version::Any)
    @ MPI.API ~/.julia/dev/MPI/src/api/api.jl:123
  [4] #macroexpand#64
    @ ./expr.jl:119 [inlined]
  [5] macroexpand
    @ ./expr.jl:117 [inlined]
  [6] docm(source::LineNumberNode, mod::Module, meta::Any, ex::Any, define::Bool)
    @ Base.Docs ./docs/Docs.jl:538
  [7] docm(source::LineNumberNode, mod::Module, meta::Any, ex::Any)
    @ Base.Docs ./docs/Docs.jl:538
  [8] Vararg{Any})
    @ DocStringExtensions ~/.julia/packages/DocStringExtensions/JVu77/src/templates.jl:11
  [9] var"@doc"(__source__::LineNumberNode, __module__::Module, x::Vararg{Any})
    @ Core ./boot.jl:539
 [10] include(mod::Module, _path::String)
    @ Base ./Base.jl:456
 [11] include(x::String)
    @ MPI.API ~/.julia/dev/MPI/src/api/api.jl:1
 [12] top-level scope
    @ ~/.julia/dev/MPI/src/api/api.jl:136
 [13] include(mod::Module, _path::String)
    @ Base ./Base.jl:456
 [14] include(x::String)
    @ MPI ~/.julia/dev/MPI/src/MPI.jl:1
 [15] top-level scope
    @ ~/.julia/dev/MPI/src/MPI.jl:43
 [16] include
    @ ./Base.jl:456 [inlined]
 [17] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
    @ Base ./loading.jl:1902
 [18] top-level scope
    @ stdin:2
in expression starting at /home/mose/.julia/dev/MPI/src/api/generated_api.jl:642
in expression starting at /home/mose/.julia/dev/MPI/src/api/generated_api.jl:642
in expression starting at /home/mose/.julia/dev/MPI/src/api/api.jl:1
in expression starting at /home/mose/.julia/dev/MPI/src/MPI.jl:1
in expression starting at stdin:2
(::DocStringExtensions.var"#35#36"{typeof(DocStringExtensions.template_hook)})(::LineNumberNode, ::ArgumentError: NULL library handle
Stacktrace:
  [1] #dlsym#1
    @ ./libdl.jl:57 [inlined]
  [2] kwcall(::NamedTuple{(:throw_error,), Tuple{Bool}}, ::typeof(Base.Libc.Libdl.dlsym), hnd::Ptr{Nothing}, s::Symbol)
    @ Base.Libc.Libdl ./libdl.jl:56
  [3] var"@mpichk"(__source__::LineNumberNode, __module__::Module, expr::Any, min_version::Any)
    @ MPI.API ~/.julia/dev/MPI/src/api/api.jl:123
  [4] #macroexpand#64
    @ ./expr.jl:119 [inlined]
  [5] macroexpand
    @ ./expr.jl:117 [inlined]
  [6] docm(source::LineNumberNode, mod::Module, meta::Any, ex::Any, define::Bool)
    @ Base.Docs ./docs/Docs.jl:538
  [7] docm(source::LineNumberNode, mod::Module, meta::Any, ex::Any)
    @ Base.Docs ./docs/Docs.jl:538
  [8] Vararg{Any})
    @ DocStringExtensions ~/.julia/packages/DocStringExtensions/JVu77/src/templates.jl:11
  [9] var"@doc"(__source__::LineNumberNode, __module__::Module, x::Vararg{Any})
    @ Core ./boot.jl:539
 [10] include(mod::Module, _path::String)
    @ Base ./Base.jl:456
 [11] include(x::String)
    @ MPI.API ~/.julia/dev/MPI/src/api/api.jl:1
 [12] top-level scope
    @ ~/.julia/dev/MPI/src/api/api.jl:136
 [13] include(mod::Module, _path::String)
    @ Base ./Base.jl:456
 [14] include(x::String)
    @ MPI ~/.julia/dev/MPI/src/MPI.jl:1
 [15] top-level scope
    @ ~/.julia/dev/MPI/src/MPI.jl:43
 [16] include
    @ ./Base.jl:456 [inlined]
 [17] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
    @ Base ./loading.jl:1902
 [18] top-level scope
    @ stdin:2
in expression starting at /home/mose/.julia/dev/MPI/src/api/generated_api.jl:642
in expression starting at /home/mose/.julia/dev/MPI/src/api/generated_api.jl:642
in expression starting at /home/mose/.julia/dev/MPI/src/api/api.jl:1
in expression starting at /home/mose/.julia/dev/MPI/src/MPI.jl:1
in expression starting at stdin:2
(::DocStringExtensions.var"#35#36"{typeof(DocStringExtensions.template_hook)})(::LineNumberNode, ::Vararg{Any})
    @ DocStringExtensions ~/.julia/packages/DocStringExtensions/JVu77/src/templates.jl:11
  [9] var"@doc"(__source__::LineNumberNode, __module__::Module, x::Vararg{Any})
    @ Core ./boot.jl:539
 [10] include(mod::Module, _path::String)
    @ Base ./Base.jl:456
 [11] include(x::String)
    @ MPI.API ~/.julia/dev/MPI/src/api/api.jl:1
 [12] top-level scope
    @ ~/.julia/dev/MPI/src/api/api.jl:136
 [13] include(mod::Module, _path::String)
    @ Base ./Base.jl:456
 [14] include(x::String)
    @ MPI ~/.julia/dev/MPI/src/MPI.jl:1
 [15] top-level scope
    @ ~/.julia/dev/MPI/src/MPI.jl:43
 [16] include
    @ ./Base.jl:456 [inlined]
 [17] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
    @ Base ./loading.jl:1902
 [18] top-level scope
    @ stdin:2
in expression starting at /home/mose/.julia/dev/MPI/src/api/generated_api.jl:642
in expression starting at /home/mose/.julia/dev/MPI/src/api/generated_api.jl:642
in expression starting at /home/mose/.julia/dev/MPI/src/api/api.jl:1
in expression starting at /home/mose/.julia/dev/MPI/src/MPI.jl:1
in expression starting at stdin:2
ERROR: LoadError: ERROR: LoadError: ERROR: LoadError: Failed to precompile MPI [da04e1cc-30fd-572f-bb4f-1f8673147195] to "/home/mose/.julia/compiled/v1.10/MPI/jl_uFQl1h".
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
   @ Base ./loading.jl:2145
 [3] compilecache
   @ ./loading.jl:2018 [inlined]
 [4] _require(pkg::Base.PkgId, env::String)
   @ Base ./loading.jl:1662
 [5] _require_prelocked(uuidkey::Base.PkgId, env::String)
   @ Base ./loading.jl:1517
 [6] macro expansion
   @ ./loading.jl:1505 [inlined]
 [7] macro expansion
   @ ./lock.jl:267 [inlined]
 [8] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1468
in expression starting at /home/mose/.julia/dev/MPI/docs/examples/01-hello.jl:1
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
Failed to precompile MPI [da04e1cc-30fd-572f-bb4f-1f8673147195] to "/home/mose/.julia/compiled/v1.10/MPI/jl_78TCMt".
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
   @ Base ./loading.jl:2145
 [3] compilecache
   @ ./loading.jl:2018 [inlined]
 [4] _require(pkg::Base.PkgId, env::String)
   @ Base ./loading.jl:1662
 [5] _require_prelocked(uuidkey::Base.PkgId, env::String)
   @ Base ./loading.jl:1517
 [6] macro expansion
   @ ./loading.jl:1505 [inlined]
 [7] macro expansion
   @ ./lock.jl:267 [inlined]
 [8] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1468
in expression starting at /home/mose/.julia/dev/MPI/docs/examples/01-hello.jl:1

[26305] signal (15): Terminated
in expression starting at none:0
_ZN4llvm7isGuardEPKNS_4UserE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZL14isKnownNonZeroPKN4llvm5ValueERKNS_5APIntEjRKN12_GLOBAL__N_15QueryE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZL14isKnownNonZeroPKN4llvm5ValueEjRKN12_GLOBAL__N_15QueryE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZN4llvm14isKnownNonZeroEPKNS_5ValueERKNS_10DataLayoutEjPNS_15AssumptionCacheEPKNS_11InstructionEPKNS_13DominatorTreeEb at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZL16SimplifyICmpInstjPN4llvm5ValueES1_RKNS_13SimplifyQueryEj at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZN4llvm16InstCombinerImpl13visitICmpInstERNS_8ICmpInstE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZN4llvm16InstCombinerImpl3runEv at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZL31combineInstructionsOverFunctionRN4llvm8FunctionERNS_19InstructionWorklistEPNS_9AAResultsERNS_15AssumptionCacheERNS_17TargetLibraryInfoERNS_19TargetTransformInfoERNS_13DominatorTreeERNS_25OptimizationRemarkEmitterEPNS_18BlockFrequencyInfoEPNS_18ProfileSummaryInfoEjPNS_8LoopInfoE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZN4llvm24InstructionCombiningPass13runOnFunctionERNS_8FunctionE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZN4llvm13FPPassManager13runOnFunctionERNS_8FunctionE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZN12_GLOBAL__N_113CGPassManager11runOnModuleERN4llvm6ModuleE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
operator() at /home/mose/repo/julia/src/jitlayers.cpp:1144
withModuleDo<(anonymous namespace)::OptimizerT::operator()(llvm::orc::ThreadSafeModule, llvm::orc::MaterializationResponsibility&)::<lambda(llvm::Module&)> > at /home/mose/repo/julia/usr/include/llvm/ExecutionEngine/Orc/ThreadSafeModule.h:136 [inlined]
operator() at /home/mose/repo/julia/src/jitlayers.cpp:1109 [inlined]
CallImpl<(anonymous namespace)::OptimizerT> at /home/mose/repo/julia/usr/include/llvm/ADT/FunctionExtras.h:222
_ZN4llvm3orc16IRTransformLayer4emitESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EENS0_16ThreadSafeModuleE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
emit at /home/mose/repo/julia/src/jitlayers.cpp:619
_ZN4llvm3orc31BasicIRLayerMaterializationUnit11materializeESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc19MaterializationTask3runEv at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)

[26341] signal (15): Terminated
in expression starting at none:0

[26299] signal (15): Terminated
in expression starting at /home/mose/.julia/dev/MPI/docs/examples/01-hello.jl:1
epoll_wait at /usr/lib/libc.so.6 (unknown line)
_ZN4llvm23MemoryDependenceResults13getDependencyEPNS_11InstructionE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZN4llvm7GVNPass11processLoadEPNS_8LoadInstE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZN4llvm7GVNPass18processInstructionEPNS_11InstructionE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZN4llvm7GVNPass12processBlockEPNS_10BasicBlockE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZN4llvm7GVNPass7runImplERNS_8FunctionERNS_15AssumptionCacheERNS_13DominatorTreeERKNS_17TargetLibraryInfoERNS_9AAResultsEPNS_23MemoryDependenceResultsEPNS_8LoopInfoEPNS_25OptimizationRemarkEmitterEPNS_9MemorySSAE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
uv__io_poll at /workspace/srcdir/libuv/src/unix/epoll.c:236
uv_run at /workspace/srcdir/libuv/src/unix/core.c:400
ijl_task_get_next at /home/mose/repo/julia/src/partr.c:381
_ZN4llvm3gvn13GVNLegacyPass13runOnFunctionERNS_8FunctionE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZN4llvm13FPPassManager13runOnFunctionERNS_8FunctionE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZN4llvm13FPPassManager11runOnModuleERNS_6ModuleE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
_ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE at /home/mose/repo/julia/usr/bin/../lib/libLLVM-14jl.so (unknown line)
poptask at ./task.jl:963
wait at ./task.jl:972
#wait#624 at ./condition.jl:130
operator() at /home/mose/repo/julia/src/aotcompile.cpp:651
jl_dump_native_impl at /home/mose/repo/julia/src/aotcompile.cpp:710
ijl_write_compiler_output at /home/mose/repo/julia/src/precompile.c:126
ijl_atexit_hook at /home/mose/repo/julia/src/init.c:258
jl_repl_entrypoint at /home/mose/repo/julia/src/jlapi.c:718
main at /home/mose/repo/julia/cli/loader_exe.c:58
unknown function (ip: 0x7fe6369d628f)
__libc_start_main at /usr/lib/libc.so.6 (unknown line)
_start at /build/glibc/src/glibc/csu/../sysdeps/x86_64/start.S:115
unknown function (ip: (nil))
Allocations: 1887200 (Pool: 1885710; Big: 1490); GC: 4
wait at ./condition.jl:125 [inlined]
wait at ./process.jl:661
success at ./process.jl:523
jfptr_success_32685 at /home/mose/repo/julia/usr/lib/julia/sys.so (unknown line)
compilecache at ./loading.jl:2054
compilecache at ./loading.jl:2018 [inlined]
_require at ./loading.jl:1662
_require_prelocked at ./loading.jl:1517
macro expansion at ./loading.jl:1505 [inlined]
macro expansion at ./lock.jl:267 [inlined]
require at ./loading.jl:1468
jfptr_require_33838 at /home/mose/repo/julia/usr/lib/julia/sys.so (unknown line)
jl_apply at /home/mose/repo/julia/src/julia.h:1872 [inlined]
call_require at /home/mose/repo/julia/src/toplevel.c:466 [inlined]
eval_import_path at /home/mose/repo/julia/src/toplevel.c:503
jl_toplevel_eval_flex at /home/mose/repo/julia/src/toplevel.c:728
jl_toplevel_eval_flex at /home/mose/repo/julia/src/toplevel.c:853
ijl_toplevel_eval_in at /home/mose/repo/julia/src/toplevel.c:968
eval at ./boot.jl:370 [inlined]
include_string at ./loading.jl:1756
_include at ./loading.jl:1816
include at ./Base.jl:456
jfptr_include_49891 at /home/mose/repo/julia/usr/lib/julia/sys.so (unknown line)
exec_options at ./client.jl:307
_start at ./client.jl:522
jfptr__start_40790 at /home/mose/repo/julia/usr/lib/julia/sys.so (unknown line)
jl_apply at /home/mose/repo/julia/src/julia.h:1872 [inlined]
true_main at /home/mose/repo/julia/src/jlapi.c:573
jl_repl_entrypoint at /home/mose/repo/julia/src/jlapi.c:717
main at /home/mose/repo/julia/cli/loader_exe.c:58
unknown function (ip: 0x7f573b70028f)
__libc_start_main at /usr/lib/libc.so.6 (unknown line)
_start at /build/glibc/src/glibc/csu/../sysdeps/x86_64/start.S:115
unknown function (ip: (nil))
Allocations: 3006 (Pool: 2996; Big: 10); GC: 0
--------------------------------------------------------------------------
mpiexec detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[58132,1],1]
  Exit code:    1
--------------------------------------------------------------------------
ERROR: failed process: Process(`/home/mose/.julia/artifacts/65763fcda97bdafc42a83deacb687015551f72d1/bin/mpiexec -n 4 /home/mose/repo/julia/usr/bin/julia --startup-file=no -q /home/mose/.julia/dev/MPI/test/../docs/examples/01-hello.jl`, ProcessExited(1)) [1]

Stacktrace:
  [1] pipeline_error
    @ ./process.jl:565 [inlined]
  [2] run(::Cmd; wait::Bool)
    @ Base ./process.jl:480
  [3] run(::Cmd)
    @ Base process.jl:477
  [4] (::var"#3#4")(exe::String)
    @ Main none:4
  [5] (::JLLWrappers.var"#2#3"{var"#3#4", String})()
    @ JLLWrappers ~/.julia/packages/JLLWrappers/QpMQW/src/runtime.jl:49
  [6] withenv(::JLLWrappers.var"#2#3"{var"#3#4", String}, ::Pair{String, String}, ::Vararg{Pair{String, String}})
    @ Base env.jl:235
  [7] withenv_executable_wrapper(f::Function, executable_path::String, PATH::String, LIBPATH::String, adjust_PATH::Bool, adjust_LIBPATH::Bool)
    @ JLLWrappers ~/.julia/packages/JLLWrappers/QpMQW/src/runtime.jl:48
  [8] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Base ./essentials.jl:818
  [9] invokelatest(::Any, ::Any, ::Vararg{Any})
    @ Base ./essentials.jl:815
 [10] mpiexec(f::Function; adjust_PATH::Bool, adjust_LIBPATH::Bool)
    @ OpenMPI_jll ~/.julia/packages/JLLWrappers/QpMQW/src/products/executable_generators.jl:21
 [11] mpiexec(f::Function)
    @ OpenMPI_jll ~/.julia/packages/JLLWrappers/QpMQW/src/products/executable_generators.jl:19
 [12] top-level scope
    @ none:4
mpiexecjl: Error During Test at /home/mose/.julia/dev/MPI/test/mpiexecjl.jl:4
  Got exception outside of a @test
  failed process: Process(`/tmp/jl_TA9A0o/mpiexecjl -n 4 --project=/tmp/jl_TA9A0o /home/mose/repo/julia/usr/bin/julia --startup-file=no -q /home/mose/.julia/dev/MPI/test/../docs/examples/01-hello.jl`, ProcessExited(1)) [1]
  
  Stacktrace:
    [1] pipeline_error
      @ ./process.jl:565 [inlined]
    [2] run(::Cmd; wait::Bool)
      @ Base ./process.jl:480
    [3] run
      @ ./process.jl:477 [inlined]
    [4] #5
      @ ~/.julia/dev/MPI/test/mpiexecjl.jl:20 [inlined]
    [5] withenv(f::var"#5#11"{String, String, String, String, Int64}, keyvals::Pair{String, String})
      @ Base ./env.jl:235
    [6] (::var"#3#9")(dir::String)
      @ Main ~/.julia/dev/MPI/test/mpiexecjl.jl:19
    [7] mktempdir(fn::var"#3#9", parent::String; prefix::String)
      @ Base.Filesystem ./file.jl:760
    [8] mktempdir(fn::Function, parent::String)
      @ Base.Filesystem ./file.jl:756
    [9] mktempdir(fn::Function)
      @ Base.Filesystem ./file.jl:756
   [10] macro expansion
      @ ~/.julia/dev/MPI/test/mpiexecjl.jl:5 [inlined]
   [11] macro expansion
      @ ~/repo/julia/usr/share/julia/stdlib/v1.10/Test/src/Test.jl:1502 [inlined]
   [12] top-level scope
      @ ~/.julia/dev/MPI/test/mpiexecjl.jl:5
   [13] include(fname::String)
      @ Base.MainInclude ./client.jl:478
   [14] top-level scope
      @ ~/.julia/dev/MPI/test/runtests.jl:36
   [15] include(fname::String)
      @ Base.MainInclude ./client.jl:478
   [16] top-level scope
      @ none:6
   [17] eval
      @ ./boot.jl:370 [inlined]
   [18] exec_options(opts::Base.JLOptions)
      @ Base ./client.jl:280
   [19] _start()
      @ Base ./client.jl:522
Test Summary: | Pass  Error  Total   Time
mpiexecjl     |    1      1      2  23.0s
ERROR: LoadError: Some tests did not pass: 1 passed, 0 failed, 1 errored, 0 broken.
in expression starting at /home/mose/.julia/dev/MPI/test/mpiexecjl.jl:4
in expression starting at /home/mose/.julia/dev/MPI/test/runtests.jl:33
ERROR: Package MPI errored during testing

Note that this happens in the mpiexecjl test, so a race condition is likely. But this didn't seem to happen before.

vchuravy · 2023-01-13T12:12:33Z

I wonder if the increased likelihood of races is due to us producing two files and one process one on one file and the other on the other.

giordano · 2023-01-13T14:47:07Z

For the record, it appears the main problem is indeed related to race conditions (in MPI.jl we run tests using multiple parallel Julia processes), and the fact that now there are more flags which invalidate the cache, so it's even more important to pass consistent flags everywhere to julia processes.

giordano · 2023-01-14T19:13:51Z

Resolved in JuliaParallel/MPI.jl#706 by consistently using Base.julia_cmd() in tests to ensure we pass the same flags to julia subprocesses.

Again, for future reference the take-away lesson from this is that using same flags everywhere is more important than ever because more flags can invalidate the compiler cache, and spawning multiple subprocesses at the same time recompiling the cache (as we do in MPI.jl) is very likely to cause bad race conditions.

giordano added the regression Regression in behavior compared to a previous version label Dec 29, 2022

giordano mentioned this issue Jan 3, 2023

test on mvapich JuliaParallel/MPI.jl#700

Merged

Moelf mentioned this issue Jan 10, 2023

compilecache failed when @everywhere using from remote machines #48217

Closed

giordano mentioned this issue Jan 13, 2023

Consistently use Base.julia_cmd() in tests JuliaParallel/MPI.jl#706

Merged

giordano closed this as completed Jan 14, 2023

giordano mentioned this issue Jan 24, 2023

Make julia_exename public API? #48393

Closed

giordano mentioned this issue Mar 11, 2024

Add mpiexecjl for windows JuliaParallel/MPI.jl#783

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MPI.jl fails to precompile on Julia nightly in some configurations #48039

MPI.jl fails to precompile on Julia nightly in some configurations #48039

giordano commented Dec 29, 2022

giordano commented Jan 13, 2023

vchuravy commented Jan 13, 2023

giordano commented Jan 13, 2023 •

edited

Loading

vchuravy commented Jan 13, 2023

giordano commented Jan 13, 2023

giordano commented Jan 14, 2023

MPI.jl fails to precompile on Julia nightly in some configurations #48039

MPI.jl fails to precompile on Julia nightly in some configurations #48039

Comments

giordano commented Dec 29, 2022

giordano commented Jan 13, 2023

vchuravy commented Jan 13, 2023

giordano commented Jan 13, 2023 • edited Loading

vchuravy commented Jan 13, 2023

giordano commented Jan 13, 2023

giordano commented Jan 14, 2023

giordano commented Jan 13, 2023 •

edited

Loading