WIP: Include support for cross-platform deps.jl

[rened]

Given two machines Mac and Linux with the respective OSes, the following currently does not work:

• addproc({"Linux"}) on the Mac
• using HDF5

The problem is that the auto-generated deps.jl file, which is included by HDF5/src/plain.jl on PID 1, has the path to the dynamic library hard-coded, and every worker sees exactly that string, not the one which corresponds to the worker's machine.

(HDF5 is just used as an example here, and any OS/package-manager mismatch where workers have their libraries at different paths would fail as well)

This PR includes 2 changes:

• if the generated deps.jl cannot load the hard-coded path, it looks for the local installation of the package and uses its deps.jl
• a helper function include_deps(pkgname) which can be used by HDF5 to load its library (see update for cross platform bindeps JuliaIO/HDF5.jl#221), which make sure to use the local deps.jl

Only drawback I found so far is that existing build.jl scripts need to be updated to include the package name when calling @BinDeps.install - any ideas how to get around this?

[mlubin]

@rened, could you restore the old behavior if the package name isn't provided? No need to break things.

[mlubin]

Also the include_deps pattern defeats the point of deps.jl which was to avoid needing to load the BinDeps package.

[rened]

Sure, I'll include the default behavior for install.
I was not aware that BinDeps should not be needed at runtime, I'll try to refactor that so that it still works.

[tkelman]

This should also avoid using Pkg.dir, and rather make use of @__FILE__ for packages that might get installed in non-default locations, see JuliaLang/julia#8679 (comment)

I really don't think mixed-platform addprocs is something that base Julia claims to support, and adding complication to the already-difficult-to-maintain bindeps system to accommodate this use case seems iffy to me.

[rened]

I'll try to make is as uninvasive as possible. I believe having the ability to work on your own machine (with your favorite OS / distribution), and then quickly summon some machines from a cluster (oder simply other random machines in the lab) would be really neat to have. Let's see whether there is a simple and robust way to do it.

[samoconnor]

I believe having the ability to work on your own machine (with your favorite OS / distribution), and then quickly summon some machines from a cluster (oder simply other random machines in the lab) would be really neat to have.

I think that's a pretty essential requirement.
Linux is clearly the dominant OS for server/cloud/cluster deployment, but it is not what is installed on most laptops. Even if an end-user machine is running Linux, it is likely to be a different distribution/version to the one used by the cluster/cloud provider.

[nalimilan]

Why not just include(depsfile)?

[nalimilan]

While I'm sympathetic to the goal, this doesn't sound like the right level to support this. It doesn't make much sense to have @checked_lib take a path as an argument, but load it from deps.jl in case of failure. If it's so smart, why not load deps.jl unconditionnally?

Looks like we need to rethink the system more seriously. I'm not familiar with the parallel code, but in the present case it seems that the library path (or equivalently the contents of deps.jl) shouldn't be considered as something to share between workers. The right fix is to find a way to tell Julia that it needs to load this information separately for each worker.