PyJulia/PyCall.jl not working well with new compilation cache management in Julia 0.7/Pkg3.jl

[tkf]

I'm moving the discussion from JuliaPy/pyjulia#173 to gather more attention. I also describe in the last paragraphs how it may help not only PyJulia users but also other Julia users.

(@stevengj, @Keno, please correct me or add more details if I'm missing something)

Prior to Julia 0.7, PyJulia users could use multiple Python interpreters without rebuilding PyCall.jl (PyJulia depends heavily on PyCall.jl) against each Python interpreters. Considering it's a common practice to have multiple Python virtual environments (with venv/virtualenv/conda), being able to use PyJulia without recompiling PyCall.jl every time is an important feature. However, supporting this feature in Julia ≥ 0.7 seems hard.

(Some details: Just to be clear, not all PyJulia users will notice this effect. Notably, Windows or macOS users only using Python 3 (or only using Python 2...) are unaffected. Linux users only using dynamically linked Python 3 (or 2) are also fine. However, Linux users using statically linked Python, such as the ones distributed by Ubuntu and Anaconda, will not be able to use PyJulia. I think those platforms are popular enough to pay a special attention.)

PyJulia relies on Base.LOAD_CACHE_PATH in Julia 0.6 to separate the cache of PyCall.jl. It inserts a private path to Base.LOAD_CACHE_PATH[1] if the Python interpreter in which PyJulia is loaded is incompatible with the Python interpreter with which PyCall.jl is configured:

# Add a private cache directory. PyCall needs a different
# configuration and so do any packages that depend on it.
self._call(u"unshift!(Base.LOAD_CACHE_PATH, abspath(Pkg.Dir._pkgroot()," +
    "\"lib\", \"pyjulia%s-v$(VERSION.major).$(VERSION.minor)\"))" % sys.version_info[0])

--- https://github.com/JuliaPy/pyjulia/blob/18d98e5b1b616a4d663273cc36cdd835ab0b33da/julia/core.py#L357-L360

This "hack" doesn't work in Julia ≥ 0.7 as Base.LOAD_CACHE_PATH is gone now. Also, reading how cache_file_entry is used to locate the .ji files, I couldn't find a way to implement a new "hack" other than duplicating the whole DEPOT_PATH[1].

julia/base/loading.jl

Lines 594 to 608 in d038f2f

	cache_file_entry(pkg::PkgId) = joinpath(
	"compiled",
	"v$(VERSION.major).$(VERSION.minor)",
	pkg.uuid === nothing ? "$(pkg.name).ji" : joinpath(pkg.name, "$(package_slug(pkg.uuid)).ji")
	)
	function find_all_in_cache_path(pkg::PkgId)
	paths = String[]
	entry = cache_file_entry(pkg)
	for depot in DEPOT_PATH
	path = joinpath(depot, entry)
	isfile_casesensitive(path) && push!(paths, path)
	end
	return paths
	end

But creating a new DEPOT_PATH[1] for each incompatible Python interpreter type is rather overkill and ignores Pkg3 working hard to de-duplicate resources. It would be really nice to have a direct support from Julia and Pkg3, if it is feasible.

An idea: "build option" support

I think one way to directly support it is to add "build option" support in Julia and Pkg3. What I mean by "build option" is something that altering it changes the precompilation cache. Currently Julia packages do this via environment variables. Concrete examples are PYTHON of PyCall.jl and CONDA_JL_HOME of Conda.jl. If those options can be directly configured by Project.toml file, and if compilation cache paths depend on such options, PyJulia can just create a new Project.toml with appropriate options and launch a new Julia interpreter. Importantly, the compilation cache path of a package has to depend also on the build options of all its (direct and indirect) dependencies.

I think the "build option" support can help other Julia users (who do not use PyJulia) too. For example, you can change CONDA_JL_HOME for a particular Project.toml to tweak versions of external libraries provided via Conda.jl without contaminating other projects. (Maybe similar benefits can be gained via BinaryProvider.jl too?) I think supporting 32-bit and 64-bit Julia JuliaLang/Pkg.jl#93 can also be done by changing "build option" of the "root" julia package. I suppose then it makes sense for deps directories to depend on "build options" too (JuliaLang/Pkg.jl#93 (comment)).

[tkf]

I just noticed @stevengj already suggested it in JuliaLang/Juleps#38

[rh314]

I'm currently hacking away at core.py in pyjulia to see if I can't get it to work with Julia v1.0.0.

PS: @tkf, is there any workarounds to make this work in the meantime? (As a new Julia language user, getting interop up and running between Julia and Python is important for me)

[tkf]

Please submit the bug reports / questions for PyJulia to https://github.com/JuliaPy/pyjulia This issue is specifically about discussing julia core functionality that can help PyJulia developers improving PyJulia.

But the short answer is that, in some limited situations (see the forth paragraph above "Some details: ..."), PyJulia should work even with Julia 1.0. Or more precisely, can be fixed to work --- we only tested it with 0.7-rc2 at the moment https://travis-ci.org/JuliaPy/pyjulia/builds/412394521.

[tkf]

@rh314 Checkout JuliaPy/pyjulia#183

[StefanKarpinski]

Is there anything actionable here for Julia itself?

[tkf]

I found that you included package options (what I called "build options" above) in "features we need" list JuliaLang/Pkg.jl#165 (comment) so I suppose this would be solved eventually. But for that to work, the path to precompilation file has to depend on the package options for itself and all upstream packages. Do you think it's feasible to have it in Julia? I guess implementing it as a hash tree would probably be straightforward (if it does not interfere with some other existing mechanisms) so the question is more like if it is reasonable to have such complexity.

[longemen3000]

is this issue still valid?. i came here from https://pyjulia.readthedocs.io/en/latest/limitations.html

[ViralBShah]

Close?

[tkf]

It's now possible to fix it with something like JuliaPy/PyCall.jl#945