Pkg operations fail for packages in LOAD_PATH

[bjarthur]

it works fine if the package is in .julia/v0.x, but if it is somewhere else then it claims there is no test/runtests.jl even if my LOAD_PATH is set such that i can using it. i know that the recommended dev cycle is to keep packages in .julia, but would it make sense to make Pkg.test() work otherwise?

[IainNZ]

If the package isn't in .julia, then none of the other Pkg.* functionality will work on it either, right?

[bjarthur]

correct. no Pkg.* functionality works outside of the Pkg.dir() directory, and i'm not sure that makes sense.

seems to me it would be nice to have a formal way of interacting with the system wide packages in <julia>/local/share and <julia>/share and in any other directories the user puts in LOAD_PATH. for example, having Pkg.installed() list them, Pkg.add() not put another copy of a dependency in .julia/v0.x if it already exists in LOAD_PATH, Pkg.test() to test them, etc.

admittedly a low priority, but long term would be good, no?

[IainNZ]

How would Pkg.installed() work? What makes a package? What happens if you do Pkg.add and there is a name conflict? etc. Seems like a massive increase in complexity for little gain.
I could buy into the idea of JULIA_PKGDIR being able to be list of locations, instead of the single location it is now, but not so much the LOAD_PATH.

There is a chance of the whole Pkg code being overhauled - it could be addressed then.

[tkelman]

A libgit2-based Pkg rewrite is planned (#7339). Was supposed to be a GSoC, that didn't happen, would be great if someone else wants to look at it.

Also sort of ties into how we want to implement #5155. If the idea is to move some pieces of Base out into packages and promote other packages to some sort of default installation mechanism, will need to figure out a good way to do it.

[swt30]

Having convinced a number of people in our department to try Julia, I came to leave an issue about Pkg.installed() not showing system-wide packages in LOAD_PATH (<julia>/local/share etc) and found this. So I'll just note that it would be nice to have a way to list these packages so that people can know whether they need to be getting their own copies. As far as I can tell, .julia packages installed because of a user's REQUIRE take precedence, which works well. People can install their own copies if they want control over the versions, or fall back on the system defaults otherwise.

I saw #9699, which adds tab-completion for these system-wide packages, but I don't know what would be required to make them show up in Pkg.installed().

[swt30]

We had further issues (which have so far showed up only in Nettle.jl) because Pkg.dir("Nettle") returns the user's .julia/v0.x/Nettle, even though this path doesn't exist. Instead, Nettle exists in <julia>/local/share. This means that Nettle doesn't load because it effectively can't find itself. Would it make more sense for Pkg.dir("Package") to return the actual Nettle directory if it exists?

I guess I'm saying that Pkg.dir("Package") seems to have two different meanings:

1. the path where a package should be installed (used during build)
2. the path where a package actually is installed (used to locate it on the system later)

and it seems like these might be at odds. Basically, if we want to store some packages in a place that's in LOAD_PATH but not .julia, they should be able to find themselves using Pkg.dir("ThisPackage") But we also want the package manager to put new packages into .julia.

To answer @IainNZ's questions, I would expect the following:

• Pkg.add() working based on the version in .julia seems important so that you can use both system-wide Julia packages (in LOAD_PATH) and user-specific packages. You can currently install packages system-wide by temporarily changing your JULIA_PKGDIR to somewhere in <julia>/local/share/ (see LOAD_PATH). If there is a .julia version of the same package, it will take priority, which seems like nice behaviour to me.
• Ideally Pkg.installed() would list all the packages in both JULIA_PKGDIR and LOAD_PATH, and more usefully it would do so in a way that indicates the preferred version.

I'm going to have a quick look at the Pkg code to try and get a better handle on the way things work right now. Handling a scenario where a sysadmin can provide a system-wide copy of Julia and some commonly-used packages would be really nice.

[stevengj]

@swt30, while I agree that Pkg functionality should work, where possible, for packages installed in the LOAD_PATH, I think the issue with Pkg.dir is more basic. I don't think packages should be using Pkg.dir("Foo") to discover absolute pathnames of their own files.

Using Pkg.dir("Foo") inside a package seems inherently fragile. If I am in a file Foo.jl and I want the path of Foo.jl, I should ask Julia what file I am in—the documented @__FILE__ macro—not where the package system thinks the Foo directory is.

So, for example, Nettle should use joinpath(dirname(dirname(@__FILE__), "deps") to find its deps directory. (Maybe there should be a shorter command for this, something like joinpath(:.., "deps").)

[swt30]

@stevengj, this is exactly what I was looking for, thanks. I'll submit a PR to Nettle to fix that problem for now and keep an eye out for any other packages that have similar problems as we install more of them. A shorter syntax would be nice - Pkg.dir("Nettle") is very readable as it is.

[stevengj]

@swt30, here is a list of packages that currently use Pkg.dir("...") in their source files: ApproxFun, BioSeq, Blink, Blosc, BlossomV, Bokeh, CPLEX, CasaCore, CauseMap, Cbc, Compose, CoreNLP, Cubature, DCEMRI, DeclarativePackages, ECOS, EconDatasets, FLANN, FreeType, GARCH, GLFW, GLMNet, GLPK, GLPlot, GLText, GeoIP, GoogleCharts, GraphViz, Gurobi, Hadamard, Homebrew, Images, Ipopt, JellyFish, Jewel, JulieTest, KernSmooth, LMDB, Languages, LevelDB, Lexicon, LibCURL, LightXML, Logging, MNIST, MarketData, MathProgBase, MatrixDepot, MeCab, MetadataTools, Metis, MixedModels, Mosek, NetCDF, Nettle, ObjectiveC, PGFPlots, Patchwork, Phylogenetics, PicoSAT, ProfileView, PublicSuffix, PySide, Quandl, QuantEcon, RDatasets, RNGTest, RandomMatrices, RdRand, ReverseDiffSparse, SCS, SDE, ShapeModels, Stan, TermWin, TikzGraphs, TimeModels, TimeSeries, UAParser, Vega, VideoIO, WORLD, Weave, WinRPM, ZMQ

A lot of PRs, unfortunately. Probably you're right that Pkg.dir("Foo") should be more flexible — when Pkg is deciding where to install something, it should use something like Pkg.installdir("Foo") or just joinpath(Pkg.dir(), "Foo"), whereas Pkg.dir("Foo") should return where Foo actually is.

(But it still seems wrong to me to use Pkg.dir to find the deps directory. It just seems too easy to confuse, e.g. if I have more than one version of a package, in different directories, and I manually load one from an alternative location, it should find its own deps rather looking at the global Pkg directory.)

[stevengj]

cc: @StefanKarpinski

[StefanKarpinski]

I guess this could be made to work, but things that are installed elsewhere aren't necessarily packages at all, nor were they installed via Pkg, so it's a little weird to test them with Pkg.test. It feels like it should be some kind of orthogonal mechanism somehow.

[IainNZ]

Indeed, this is what I think I was getting at with my comment from October (wooah)

What makes a package?

[StefanKarpinski]

I mean, you could look for $pkg/test/runtests.jl anywhere in LOAD_PATH, but it just feels off like in that case we shouldn't be using the Pkg prefix for this.

[stevengj]

Let me propose a simple criterion: if Foo is in the load path somewhere, it has the Pkg directory structure (REQUIRES file, module in Foo/src/Foo.jl), and it is a valid git repo, then it is a package. If it is listed in METADATA (with corresponding git commit hashes), then it is a registered package.

Basically, the question is, should systemwide packages in (e.g. in /usr/share/julia) be first-class (other than the fact that they are probably read-only)? I would prefer that the answer be yes. Systemwide packages on multi-user systems (or shared filesystems in Athena-like environments) are a good thing for Julia to support, and support well.

More concretely, it is confusing for systemwide packages to be omitted in Pkg.installed, or when using Foo works but Pkg.dir("Foo") does not (or worse, when using Foo breaks if the Foo/ directory is installed somewhere else in the load path).

(I think we all agree that Pkg.add("Foo") should always add in ~/.julia, so that users can individually override systemwide package versions.)

[tkelman]

I went and made PR's or committed fixes to master for

Blosc, Cbc, ECOS, GLPK, Gurobi, Images, Ipopt, MathProgBase, Metis, Mosek, WinRPM, ZMQ

which is approximately the subset of that list that I've personally used. Jameson recommending doing things using dirname(@__FILE__) a while ago and I was sold on it, but needed to see a real use case of Pkg.dir causing problems before going and bothering to change it a bunch of places. Note that not every package in that list is using Pkg.dir to refer to itself, which I think is the main problem here. Some packages use Pkg.dir to refer to locations of other packages, where I think making additional assumptions about things being installed outside of Pkg.dir is likely to be just as brittle.

[swt30]

It's a little telling that there's duplication of the same

if isfile(deps.jl)
    include(deps.jl)
else
    error("PACKAGE not built correctly; please run Pkg.build(PACKAGE)")
end

kind of logic in a couple places, when that seems like something that could be handled better by the package loader, but these changes should help in the meantime. I'll check a few more packages that I might use: Bokeh, Lexicon, ProfileView, RDatasets, Weave.

[swt30]

@timholy, I'm submitting a PR on ProfileView to make this change, but I can't do anything immediately about this line: https://github.com/timholy/ProfileView.jl/blob/151c2b3722eaa9d222bbf6910b905c5e5101bdbc/src/ProfileView.jl#L14 which assumes ImageView is installed in Pkg.dir().

[tkelman]

kind of logic in a couple places, when that seems like something that could be handled better by the package loader

Part of the trouble is that BinDeps is an external package rather than part of the package manager in base Julia. And it's more appropriate for it to be maintainable (... sort of ...) separately from base Julia and on its own release schedules. There are lines to this extent in the auto-generated deps/deps.jl file, but the error message that this logic is trying to clarify happens when deps/deps.jl has not been created at all.

[swt30]

I was bitten by this again in Interact.jl, which includes the test

if Pkg.installed("IJulia") >= v"0.1.3-"

Unfortunately since a system-wide version of IJulia doesn't show up in Pkg.installed(), this tries to compare nothing to v"0.1.3-". Changing the way Pkg.* looks for installed packages could fix this, but is there another way I could get the version of IJulia in the meantime?

[stevengj]

@swt30, the right thing is to take a page from autoconf and to test for features, not versions; i.e. if a specific function was added to IJulia that is needed, look for the presence of that function. However, at this point @shashi can probably just remove the test for "ancient" (year+ old) IJulia versions.

[tkelman]

and add the version assumptions to REQUIRE instead

[swt30]

That makes sense - I'll look at submitting a PR to Interact. I've also been thinking that making soft links from .julia to the individual system packages might be a useful workaround for getting them to show up in Pkg.*, though it will play havoc as soon as you try to do anything that needs write access.

[stevengj]

@tkelman, Interact does not require IJulia; it is an optional dependency.

[tkelman]

It looks like there's some mention of @__FILE__ / source_path in #8745, hopefully @vtjnash can document and advertise if the common wisdom about how to handle that needs to change when loading from a precompiled package.

[richardreeve]

Given the docs for Pkg.dir() say "In particular, Pkg.dir(pkg) returns the path to the package pkg." and it is extremely desirable functionality if, for instance, you want to install a resource with a package for reference, are there any plans to fix this, because @__FILE__ only works inside a package?

[tkelman]

ref https://github.com/JuliaLang/Juleps/blob/master/Pkg3.md

[KristofferC]

Obsolete with new package manager.