No way to install Julia signal handler when other dependency uses juliacall

[MilesCranmer]

Affects: JuliaCall

Describe the bug

Related to #219. There's currently a scenario with no possible workaround other than forcing a downstream user to set up environment variables to avoid a segfault.

Say that I import a package which uses serial Julia code. That package runs

from juliacall import Main as jl

to start Julia.

Then, I import a different package. That package calls multithreaded Julia code, and thus to avoid a segfault (https://juliapy.github.io/PythonCall.jl/stable/faq/#Is-PythonCall/JuliaCall-thread-safe?) instead performs the import instead as:

import os
os.environ["PYTHON_JULIACALL_HANDLE_SIGNALS"] = "yes"

from juliacall import Main as jl

However, at this point, the Julia runtime has already started. So now when I run a multithreaded portion of code, I will get a segfault, even though I implemented special workarounds in the multithreaded package.

How could I avoid this scenario? Indeed the user could always set up the environment variables themselves. But managing to figure this out from a random bus error will only be ~5% of users.

[MilesCranmer]

My temporary workaround is to put this code:

if "juliacall" in sys.modules:
    warnings.warn(
        "juliacall module already imported. Make sure that you have set `PYTHON_JULIACALL_HANDLE_SIGNALS=yes` to avoid segfaults."
    )

before the import statement.

[cjdoris]

Well firstly, packages shouldn't generally be setting global config such as that env var, though I understand why you're doing it.

Over in #219 I suggested adding a warning to set PYTHON_JULIACALL_HANDLE_SIGNALS=yes if Julia is started with multiple threads. Do you think this would be a sufficient warning?

Also, given Julia starts with 1 thread by default, presumably you're already telling your users to set PYTHON_JULIACALL_NUM_THREADS? In which case you can also tell them to set PYHON_JULIACALL_HANDLE_SIGNALS?

[hameerabbasi]

If it helps, I was able to get a stacktrace for this:

Stacktrace

Stacktrace:
  [1] wait
    @ ./task.jl:352 [inlined]
  [2] open_exclusive(path::String; mode::UInt16, poll_interval::Int64, wait::Bool, stale_age::Int64, refresh::Float64)
    @ FileWatching.Pidfile ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:268
  [3] open_exclusive
    @ ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:232 [inlined]
  [4] mkpidlock(at::String, pid::Int32; stale_age::Int64, refresh::Float64, kwopts::@Kwargs{})
    @ FileWatching.Pidfile ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:67
  [5] mkpidlock
    @ ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:62 [inlined]
  [6] mkpidlock(f::Pkg.Types.var"#51#54"{String, String, Dates.DateTime, String}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64})
    @ FileWatching.Pidfile ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:91
  [7] mkpidlock
    @ ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:90 [inlined]
  [8] mkpidlock
    @ ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:88 [inlined]
  [9] write_env_usage(source_file::String, usage_filepath::String)
    @ Pkg.Types ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/Pkg/src/Types.jl:539
 [10] Pkg.Types.EnvCache(env::Nothing)
    @ Pkg.Types ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/Pkg/src/Types.jl:377
 [11] EnvCache
    @ ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/Pkg/src/Types.jl:356 [inlined]
 [12] add_snapshot_to_undo(env::Nothing)
    @ Pkg.API ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/Pkg/src/API.jl:2189
 [13] add_snapshot_to_undo
    @ ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/Pkg/src/API.jl:2185 [inlined]
 [14] activate(path::String; shared::Bool, temp::Bool, io::Base.DevNull)
    @ Pkg.API ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/Pkg/src/API.jl:1973
 [15] top-level scope
    @ none:7

    nested task error: InterruptException:
    Stacktrace:
     [1] poptask(W::Base.IntrusiveLinkedListSynchronized{Task})
       @ Base ./task.jl:985
     [2] wait()
       @ Base ./task.jl:994
     [3] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool)
       @ Base ./condition.jl:130
     [4] wait
       @ ./condition.jl:125 [inlined]
     [5] _trywait(t::Timer)
       @ Base ./asyncevent.jl:142
     [6] wait
       @ ./asyncevent.jl:159 [inlined]
     [7] sleep(sec::Int64)
       @ Base ./asyncevent.jl:265
     [8] (::FileWatching.Pidfile.var"#13#14"{Int64, String})()
       @ FileWatching.Pidfile ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:263
    
    caused by: IOError: FileMonitor (start): no such file or directory (ENOENT)
    Stacktrace:
     [1] uv_error
       @ ./libuv.jl:100 [inlined]
     [2] start_watching(t::FileWatching.FileMonitor)
       @ FileWatching ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/FileWatching.jl:525
     [3] wait(m::FileWatching.FileMonitor)
       @ FileWatching ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/FileWatching.jl:659
     [4] watch_file(s::String, timeout_s::Float64)
       @ FileWatching ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/FileWatching.jl:772
     [5] watch_file
       @ ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/FileWatching.jl:778 [inlined]
     [6] (::FileWatching.Pidfile.var"#13#14"{Int64, String})()
       @ FileWatching.Pidfile ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:260

This can also be avoided by setting the environment variable PYTHONFAULTHANDLER to some writable file. Credit to @willow-ahrens for some of the heavy lifting on this, cc @mtsokol.