add support for the Nix package manager

[rened]

Nix is a package manager which can be used on various UNIXes, it is also the basis of the NixOS distribution.

One of its strengths is the fact that non-root users can install packages.

[rened]

it seems the travis error is just a config error, it is not cause by the code in this PR

[StefanKarpinski]

I would kind of love to be able to just use Nix everywhere since it would probably make a lot of interactions with installing system software drastically less brittle.

[rened]

That would be absolutely fantastic!

I am dreaming of a system where I could define a .nix expression in my project, nix-shell into it, and have julia, all the packages required and their bindeps in scope. And in a different terminal a different project with its depedencies etc.

It would allow to have (hopefully) 100% reproducible julia environments for each and every research project, readily accessible as .nix in its respective git repo.

I was planning to look into how npm2nix does this. (https://github.com/NixOS/npm2nix, http://sandervanderburg.blogspot.co.at/2014/10/deploying-npm-packages-with-nix-package.html).

But a cleaner (and more radical) approach would be to use nix as the underlying architecture for julia packages, merging the functionality of Pkg and BinDeps...

[rened]

just a gentle bump ;-)

[staticfloat]

As a general rule, I'm against providing multiple ways to get packages on a single platform as that just increases the surface area available for installation errors to occur. (For example, the recent port proposal) That being said, this seems to offer the ability to install packages as non-root, which is a huge plus. I'm interested in how they manage this in general for binary packages. Does Nix ever build from source?

[rened]

Nix does build from source, but has a binary cache at nixos.org, so in most cases you dont actually need to build things. But as the absolute path is part of the hash used to identify build products, you get this benefit only in the case when nix can store its packages in /nix. If a non-standard location like ~/nix is used everything is build from source.

[staticfloat]

I see. Very similar to Homebrew then. (Homebrew does have detection/declaration of whether a binary "bottle" is relocatable, but in practice many larger software packages are not relocatable)

Correct me if I'm wrong, but this seems to mean that the main benefit of including this change is that we have a standardized method of building from source? So with the Nix builder, we'll be able to move from providing our own build-from-source steps, and Nix will just take care of it for us?

[rened]

I do agree that the multiple providers are a bit cumbersome. We have support for apt, yum and pacman as they are the underlying managers for debian, centos and arch. Nix does this for nixos, but that is a niche distro, I agree.
On the other hand, nix does run on all (of these) linux distos, so it would allow to use only nix instead of apt/yum/pacman.

[rened]

Yes, exactly what you are saying. In theory it would even allow us to use it as sole BinDeps provider for linux distros + OSX, but OSX support, while present, is currently being reworked for 10.10.

[staticfloat]

Alright. In that case, we need to ensure that Nix doesn't get used when the likes of AptGet or Yum could have been used; we always want to use Binaries when they are available, since nobody really wants to build Cairo from scratch when all they want to do is plot the darn array. ;)

I believe if we treat Nix as a source builder it will do this automatically. Essentially, we want Nix to fall into this ordering properly. As it stands, if we were to merge this PR I'm not sure how BinDeps would resolve a choice between AptGet and Nix. I'm guessing it would just choose whichever of the two it happens to grab first.

We may need to create a new "category" of builders, e.g. SourcePackagers if we can't relabel Nix as a BuildProcess. I've never done this before, so I'm not sure how it'll work out.

How does that sound to you?

[rened]

Yes, Nix should only be used when apt/yum/pacman are not available, so BuildProcess sounds good: Depending on whether the user installed it in /nix or not it will transparently install binaries or from source.

[nalimilan]

Wait, this is really interesting, but if we are to choose one default package management system over the others, we should make a comprehensive comparison of the different systems before that. I can see at least these criteria:

• Availability of many libraries in the package format (so that we don't need packaging everything)
• Ability to install pre-compiled binaries without admin rights
• Ability to use the same system on different platforms
• Ability to install several versions of the same library in parallel (would help if Julia packages end up requiring conflicting versions, as will inevitably happen)

Regarding the second point, you said Nix needs to build packages from source when installing them into the user home directory, which is unfortunate. Though Guix, which is based on Nix, claims that user profiles can be managed with only normal privileges: http://www.gnu.org/software/guix/manual/guix.html#Features It may be that the daemon runs as root, and is allowed to install any packages that user ask for, because they don't interfere with other users. This is a solution which may still be interesting.

But there are several alternatives available. For example, 0install claims that installs with user rights only are possible, and that it runs on Windows and OS X: http://0install.net/why.html Linstaller seems to offer a similar set of features: http://0install.net/comparison.html

[rened]

Yes, once nix (similarly Guix) is installed by root, individual users can do whatever they want without interference and get the benefit of binary downloads. Just like you are saying, it is a daemon running as the user who owns /nix (http://nixos.org/nix/manual/#ssec-multi-user).

Having the installation in a non-nix location, resorting to from-source builds, is the fallback in the case that no one-time installation of nix into /nix in possible on that system.

While I would suggest to simply use Nix as one additional provider at the moment, keeping apt/yum/pacman and reevaluate a Nix-for-all strategy in the future, here are the notes for the bullet points:

• Library availability: I checked the major julia packages, their dependencies are there: http://nixos.org/nixos/packages.html. I will add the Nix provider to every Julia package once this PR is in, where possible.
• Install pre-compiled: yes, if nix got installed into /nix, one-time sudo required
• Cross platform: Linux, OSX (currently being refactored, good progress)
• Multiple versions: yes.

[kmsquire]

FWIW, brew (which is already used for OSX) has a Linux port which uses the same recipes.

[staticfloat]

I've used linuxbrew to install things on a locked-down Linux cluster, and it works okay, but I wouldn't consider it ready for primetime yet.

[staticfloat]

@rened have you used this patch against BinDeps to install a nix package yet?

[rened]

@staticfloat Yes, JuliaIO/HDF5.jl#177, but I can try with more packages (I want to add that to as many packages as possible anyway) and see how well it works? Also, I could try to disable e.g. apt on Ubuntu and make some experiments there. Any challenging packages you would be particularly interested in?

[staticfloat]

If you want challenging, try Gtk.jl or Cairo.jl. Good candidates for things that might not have Nix formulae yet are things like nlopt and Cbc.jl. I'd also be interested in AudioIO.jl. I'm not saying you HAVE to prove all those work, but it would be interesting.

Finally, if you're eager to start using this package manager for all of these, I should point out that Homebrew.jl doesn't have any custom code inside of BinDeps.jl; it's all extended from the outside. Looking at your changes here, you should be able to do the same thing, so perhaps the best thing to do is to simply create a Nix.jl, and experiment with it out there? Then once we're confident that we can provide better coverage for packages, we could either move it in, or keep it out as we have with Homebrew.jl in order to keep things in BinDeps.jl as small as possible (We were thinking about factoring out Apt and Yum, etc... for a while, but decided not to since they are so incredibly small in comparison to Homebrew.jl). If it turns out Nix.jl doesn't give much benefit, we then don't have to worry about excising it from BinDeps.jl. Plus, this makes it easy for other packages to use Nix in the meantime. What do you think?

[rened]

Sounds good, I'll try that during the next couple of days - thanks for the pointers!

[mk]

hey there! We're working on offering a hosted notebook environment for julia using non root containers and would be very interested in getting this to work. Anything we can do to help?

[tbenst]

Thought I'd bump this. Most of Julia works flawlessly on NixOS, but I just tried to install HDF5.jl which has BinDeps as a dependency, and the install failed due to the use of ldconfig -p, which is not used on NixOS. I tried naively just commenting this code out, and the code fails at the following step: /home/tyler/.julia/packages/HDF5/H0XJB/deps/src/hdf5-1.10.0-patch1/configure --prefix=/home/tyler/.julia/packages/HDF5/H0XJB/deps/usr, which I'm guessing is due to a missing dependency but it's hard to tell without seeing the subcommand output.

Edit: for anyone on NixOS that finds this, here's how I'm currently solving this: https://gist.github.com/tbenst/7c33680b5bac66a2a758827655517ea4

[StefanKarpinski]

This PR is pretty out of date at this point and it seems like the change needs to be with BinaryBuilder or BinaryProvider not with BinDeps, so I'm not sure there's anything to do here anymore, is there?

[thomasjm]

FWIW some packages still use BinDeps. For example, Cairo.jl still does (they've had a PR to migrate to BinaryProvider for almost a year).

@tbenst 's solution is very helpful. But maybe this PR should still be landed since BinDeps may not go away anytime soon.