Discuss approaches to supporting packages on JuliaBox

[tanmaykm]

Stable and pre-installed packages are one of the important things people are looking for on JuliaBox. Here are a few points that came out from what @ViralBShah and myself discussed recently. This is to initiate discussions on this topic. Please feel free to jump in add anything I might have missed.

Option 1. Reintroduce the larger JuliaBox package bundle
But this time, we should have:

• a release-cycle, probably once a quarter.
• update JuliaBox image with os patches, latest julia versions on each cycle
• all packages to be bundled will be installed together, package install script should install all required dependencies
• packages will run through Pkg.test() and failing tests will be notified
• packages passing tests will be shipped in the default package bundle

We will need help from package maintainers to solve any issues and ensure cross compatibility.

Option 2. Allow anyone to publish package sets to JuliaBox

• A package set is a collection of packages that are often used together (as a tar of .julia folder).
• It can be built and hosted external to JuliaBox
• User's can see a list of registered package sets, and switch between them
• all binary dependencies need to be built by the package build script

Since a package set will typically have a smaller number of packages, it will need less coordination to maintain. Also, multiple versions of the same package set can be active, to allowing people to deal with breaking changes.

Other concerns

• how to handle conflicting package dependencies across multiple packages?
• how to handle conflicts due to users adding packages to their package folder?
• should JuliaBox work on a frozed metadata, or allow Pkg.add Pkg.update against the default metadata?

ref some recent issues: #316 #315 #242

[tkelman]

On option 2, what about allowing users to switch to a different child docker image? If the base image is built on docker hub, then anyone can build layers on top of it to install more packages, and publish back (or auto build) to docker hub.

[tkelman]

We might want to collaborate with https://github.com/binder-project/binder as they've got a similar setup with docker hosted jupyter notebooks.

[ViralBShah]

There is also the issue of having default packages and installing your own. I would love to be able to have default pinned packages, and then users to install other packages by themselves.

[ViralBShah]

Cc @mlubin @simonbyrne

[mlubin]

Well the biggest issue at the moment is that adding basic packages on a Julia 0.4 kernel is straight out broken (JuliaLang/julia#12508 (comment)). Otherwise I'm in favor of option 1. Realistically I think creating your own package sets is a lot of work to ask for, I don't see many professors doing this if they want to use JuliaBox for a demonstration in a class. The bundles that are created will likely be one-offs that nobody maintains or updates. I'd rather see a curated set of packages that are trivially available in a working state.

should JuliaBox work on a frozed metadata, or allow Pkg.add Pkg.update against the default metadata?

I could see it being frustrating for package authors if they release a new package or new updates which aren't accessible on JuliaBox possibly for multiple months. However, if this conflicts with option 1, I think it would be okay to offer the curated packages with frozen metadata plus separate kernels with up-to-date metadata but no preinstalled packages.

[ViralBShah]

Frozen METADATA does sound like a good idea. Should we do branches on the METADATA and tag releases?

[simonbyrne]

One way to improve reliability of option 1 would be to use the same image as the basis of PkgEval?

@IainNZ any thoughts?

[tkelman]

PkgEval is based on vagrant not docker so that's not exactly trivial. I don't like the sound of frozen metadata at all. There needs to be a way to work around that to get package updates, and since different kernels are different executable within the same image rather than different images, it doesn't seem easy to accomplish either.

[tanmaykm]

@tkelman Yes, in option 2 we can also switch to a different docker image if that's built on the same base as JuliaBox.

But, since Julia releases are self contained, package requirements can be specified per Julia release, and packages are installed onto separate areas, it should not be an issue to support different kernels on the same image, right?

Having many different packages in one single bundle can bring up cross-dependency conflicts, particularly if some of them require specific versions of another. I don't think PkgEval reports this today.

[tkelman]

different minor versions are separate, but maybe via separate JULIA_PKG_DIR or whatevet that env var is...

[tanmaykm]

Yes, JULIA_PKGDIR can be used. We have had only the latest patch of each minor version till now.

[mlubin]

@tanmaykm, are there any real examples of packages having conflicting dependencies?

[tanmaykm]

I can't find it now, but I do recollect this issue where a package specified a hard version range in its dependency list. I suppose packages use it occasionally to handle breaking changes in dependencies.

This be something that's discouraged, at least in the JuliaBox set of repos, right?