Add an option to disable file locking

[cjdoris]

Hi! Just to follow up on this, I've run into this same issue and it basically would mean we can't use Julia at all in a shared environment (read: I have to tell someone they have to port thousands of lines of Julia code into Python or C++). What's not clear to me is why the pidlock is used even when I set export PYTHON_JULIAPKG_OFFLINE=yes. I guess someone could write simultaneously while I only read, but that seems like it's basically my own fault. Anyway, would it be possible to add an env variable that would disable the pidlock check completely for those of us who don't want it? As it stands, one user installs everything on the filesystem that we need, makes the entire directly read-only, and then nobody should ever write there again. In this case (since nobody has write permissions to the entire python and julia environement directory), we have a strong guarantee that nobody is writing while we are reading, and so the pidlock is just a burden.

Having said all that, it's still not at all clear to me that Julia has any great way to share not just the core executable, but also all installed dependencies, so is trying that essentially bound to fail?

Thank you!

Edit: it also seems like the lock.pid is not being cleaned up even after I exit the python session, whereas I assume it should be cleaned up as soon as juliapkg has resolved.

Originally posted by @nilsdeppe in #46

[JamesWrigley]

I was trying to fix this and it wasn't working quite the way I expected. I added an environment variable to disable the file lock and that worked fine, but when creating a readonly shared environment I had issues when using it from different Python environments. Here's what happened:

• I start Python as an admin user in a shared environment (read-only to everyone else).
• Set PYTHON_JULIAPKG_PROJECT=/path/to/julia/env and run import juliacall to initialize the environment and create the pyjuliapkg dir.
• As a regular user under the same shared Python environment, I set PYTHON_JULIAPKG_OFFLINE=yes/PYTHON_JULIAPKG_LOCKFILE=no/PYTHON_JULIAPKG_PROJECT=/path/to/julia/env so that it shouldn't install or try to write anything. Running import juliacall works 👍
• As a regular user under a different Python environment and with the same environment variables import juliacall fails when it tries to re-resolve the environment and write a metadata file:

---------------------------------------------------------------------------
PermissionError                           Traceback (most recent call last)
Cell In[2], line 1
----> 1 import juliacall
      2 from juliacall import Main as jl

File ~/.conda/envs/scratch/lib/python3.12/site-packages/juliacall/__init__.py:288
    285     import juliacall.ipython
    286     juliacall.ipython.load_ipython_extension(ip)
--> 288 init()

File ~/.conda/envs/scratch/lib/python3.12/site-packages/juliacall/__init__.py:160, in init()
    157 import juliapkg
    159 # Find the Julia executable and project
--> 160 CONFIG['exepath'] = exepath = juliapkg.executable()
    161 CONFIG['project'] = project = juliapkg.project()
    163 # Find the Julia library

File ~/.conda/envs/scratch/lib/python3.12/site-packages/juliapkg/deps.py:578, in executable()
    577 def executable():
--> 578     resolve()
    579     return STATE["executable"]

File ~/.conda/envs/scratch/lib/python3.12/site-packages/juliapkg/deps.py:517, in resolve(force, dry_run, update)
    515     run_julia(script, executable=exe, project=project)
    516 # record that we resolved
--> 517 save_meta(
    518     {
    519         "meta_version": META_VERSION,
    520         "dev": STATE["dev"],
    521         "version": str(ver),
    522         "executable": exe,
    523         "deps_files": {
    524             filename: {
    525                 "timestamp": os.path.getmtime(filename),
    526                 "hash_sha256": _get_hash(filename),
    527             }
    528             for filename in deps_files()
    529         },
    530         "pkgs": [pkg.dict() for pkg in pkgs],
    531         "offline": bool(STATE["offline"]),
    532         "override_executable": STATE["override_executable"],
    533     }
    534 )
    535 STATE["resolved"] = True
    536 STATE["executable"] = exe

File ~/.conda/envs/scratch/lib/python3.12/site-packages/juliapkg/deps.py:45, in save_meta(meta)
     42     if meta_json == old_meta_json:
     43         # No need to write out if nothing changed
     44         return
---> 45 with open(fn, "w") as fp:
     46     json.dump(meta, fp)

PermissionError: [Errno 13] Permission denied: '/path/to/julia/env/pyjuliapkg/meta.json'

I think the issue is that the deps_files() are different depending on the Python environment, which causes juliapkg to try re-resolving. Do you know of a way around that? It'd be quite limiting if one could only use a shared Julia environment from a single Python environment.

[cjdoris]

Hmm right so in CondaPkg there is a notion of whether the environment you are using is shared or not depending on how it was specified. If an env is shared, it will only add to it and never remove packages.

JuliaPkg has no such notion - all environments are unshared so will fully re-resolve when anything changes. It could be added.

Even after supporting sharedness, a resolve might still be necessary when you change environments, if some new packages need to be installed. In which case you'd still hit the same error - which is presumably a permissions thing caused by different users trying to write to the same place.

I can't think of a good workaround sorry. What I'd suggest is making the Julia environment yourself and telling JuliaCall to use that directly but JuliaCall has no such config currently.

[JamesWrigley]

Ah you mean adding an override for these lines?
https://github.com/JuliaPy/PythonCall.jl/blob/22cbe26454802d4b56741495710154e5273a0a22/pysrc/juliacall/__init__.py#L159-L161

Yeah that sounds easier 😅 Will give it a shot.