default packages?

[StefanKarpinski]

It might be nice to ship Julia with some default packages. There are issue, however. This implies the creation of a .julia package directory, which we currently don't do automatically, plus the installation, at julia install time, of some packages there. While that's feasible, if someone later changes to a different .julia directory, those packages would need to be installed there as well, which seems strange. One solution would be to install them in a system-wide location and add complex layered package resolution support (see #1692). Another option would be having some packages installed inside of the julia home directory and not participating in normal version resolution, although at that point it seems a bit weird to even deal with that via the package manager.

[ViralBShah]

I think we will need some resolution of #1692. We may want to ship julia with some default packages - in fact we need julia+Winston+Gadfly by the next week. Also, over time, others will want to create custom distributions of julia with their own set of packages - say at universities, or production setup, etc. This is certainly important and useful.

[aviks]

Given these issues, what is the benefit of having these default things as packages? Why cant they simply be modules within the standard library, in addition to base.

Packages currently provide a distribution mechanism, and a dependency resolution mechanism. For things which are distributed with the compiler/stdlib itself, the distribution mechanism is a no-op, and dependency resolution will, in any case, need to manually maintained at the point of creating a julia distribution (as in, it is the julia packaging managers job to build and test a compatible set of default packages"

Also, I think it creates a cognitive overload to have some packages more privileged than the others. If everything that is default is in the standard library (currently only base) and all other packages are similar, it then creates a simpler system to reason about.

Just my $.02

[JeffBezanson]

Part of the problem is that we don't want some packages more privileged than others, and yet there needs to be a way to create easy-to-use julia installs that don't need extra setup steps.

[johnmyleswhite]

I'm torn here, because having privileged packages is troubling, but that's exactly the solution that R took in order to make sure that the default language install has enough functionality to be usable.

[tknopp]

I actually think that privileging packages is not a bad thing. It means that these packages are of high quality and seem to be of great interest for the user base. If there are different competing packages it also might be a good thing to get these consolidated into a better standard package. Of course only if its the right time for this move.

[IainNZ]

I disagree. I feel that if a package is truly important enough to be "priviledged" and auto-installed in .julia then its good enough to include as a submodule of Base.

[JeffBezanson]

To me the only distinction is "ease of removal". Most of the time most people don't care how big something like Base is, but if you have special needs that make you want to remove something, you shouldn't feel trapped. For example if lots of things in Base depend on LAPACK, not just the LinAlg module, then it starts to become impossible to use julia without LAPACK.

[StefanKarpinski]

A lot of LAPACK functionality also has generic counterparts that we use for non-BLAS types. Without BLAS and LAPACK, you should be able to get most of the functionality, just with somewhat less good performance.

[amitmurthy]

Just adding my 2 bits to the discussion here.

If you look at the relationship between C# / .NET, or Java / J2EE or Erlang / OTP, in each of the cases it is the platform that provides much of the functionality for someone to get started easily in that particular environment. Some of the "platforms" are technically specifications, but in practical terms, people use the standard / default SDK that provides the same.

I am of the the opinion that we should necessarily have "privileged" packages, call it a standard library if you will, that is spoken for by the JuliaLang community.

But over the longer term we should define the contours of a Julia Computing Platform (deployment tools, database/storage, cluster tools, web interfaces, etc) and what goes into it.

However, it is to be noted that C#/Java/Erlang platforms are all backed by enterprises (deep pockets) and I cannot think of any open source community equivalents.

[quinnj]

I've been thinking recently about various ideas for GSoC, and this eventually came to mind. I was originally thinking about the help system/documentation and eventually had the thought that it even it could be a separate package (i.e. there are cases where you don't necessarily want the extra baggage of documentation with a headless server install for example).
This also coincides with another idea that has been thrown around of a Julia-Lite; where all/most of the dependencies are stripped out and you get a bare-bones install/functionality. This stripped down julia could then be more easily molded for things like emscripten compatibility, ARM support, and most of all, custom distros through default packages.
With the recent work of #5687, it seems like bundling in packages as a part of a build is not possible and fairly painless (I haven't actually tried it out yet). I can see a setup where when you install "Julia", what you're really installing is Julia-Core + default packages, which would be the Help module, LinAlg module, FFTW module, PCRE module, etc. Over time if there were other packages that became of sufficient quality, they could be included in this "default install", but it's also less of an issue because people can easily create their own distros, tacking on a few packages to the list of defaults already specified.
I don't think the wording of "privileged packages" is the right way to see it. "Default packages" are packages that would be pretty much maintained by core Julia developers anyway (or of extremely high quality and sufficient need/interest).
I feel like the modularity achieved in this approach would be a huge win and keeps with the goals of providing a tight, clean core with add-on functionality.

[johnmyleswhite]

It's a shame that the words "Base" and "Core" are already used, since that seems like the name you'd want for this.

[JeffBezanson]

I think it would be great to experiment with this. I'd like to see the system structured this way.

[quinnj]

Thinking through this a little more, I guess one of the biggest changes would be porting the make rules for current deps to BinDeps installation steps, right? It also makes me wonder which deps are truly core vs. easily made into packages. A dep like PCRE seems fairly straightforward and isolated, but what about libuv? double-conversion?

I guess what's convenient about this idea is that it's fairly easy to break it into tiny chunks. First step would be to iron out the package install process (i.e. including arbitrary packages in Julia build process), and then subsequently creating standalone packages for various deps.

[ablewhiskey]

"Kernel"?

[pao]

It's a shame that the words "Base" and "Core" are already used, since that seems like the name you'd want for this.

"Prelude" has precedent.

[JeffBezanson]

One problem is that the dependency network is denser than you'd think. For example, lots of fairly basic things in the system use regular expressions (e.g. for processing file paths).

The dependencies do have to be dealt with one by one. double-conversion is small and simple, so it can just stay in the core. libuv is crucial and quite difficult to remove, but you'd probably need to remove it for some embedded systems.

[quinnj]

Yes, I imagine the dependency network can be quite hairy with some of these. I would imagine that would be a huge side-benefit though of going through this: the chance to parse out dependency-dependent code, though my impression is that things have been kept fairly clean.

[porterjamesj]

+1 for a stripped down distribution of Julia. I think there's a lot of people interested in the language per se without all the scientific/technical computing stuff (e.g. for going things like embedding Julia, GUI application development, emscripten as @karbarcca mentioned). Having a lightweight distribution would be great for this kind of thing.

[jiahao]

the language per se without all the scientific/technical computing stuff

I have no idea what that would look like ;)

[JeffBezanson]

For the sake of discussion I'll choose to take the question seriously :)

Stuff that is "part of the language" is listed in base/boot.jl and comprises the Core module. The problem is you really can't do anything useful with it. Then Base contains two kinds of things: stuff you really need (like integer arithmetic), and other functions that you might not need. The distinction should be obvious but really isn't. However I'd love to see any particular view of what's in the first category. Can be done just by aggressively editing down base/. Anybody who does it will be instantly rewarded with the world's fastest julia startup time ;)

[tknopp]

In #5155 I have made the proposal to shrink base and introduce a "standard library" which in sum is much larger than current base. I think this proposal goes in line with @karbarcca thoughts.

Jeff, I think its simpler to define what shrinked Base contains than whats currently in Base. Shrinked Base would minimize all shared libraries that are required in addition to libjulia. No fftw, no lapack and so on. All the arithmetic stuff should be in and IMHO also the Julia fallback implementation for Blas routines working on AbstractArray.

I would really love if this proposal could move forward. One technical question would be where the standard library should be hosted. My thought is that this should follow the Julia release cycle and not be put into .julia but into the global julia directory (i.e. as with current base).

One completely alternative approach that would not live in Julia is to make different distributions of Julia by pulling the Julia source, putting some packages to it and the distribute it. But I think that the centralized approach is better.

[tknopp]

And with stripped down Julia and REPL.jl we will have GPL-free Julia distribution (although using julia-basic and omitting fftw is not a big thing either).

[tknopp]

Maybe a good wording for stripped down Julia would be Runtime. Core and stripped down Base would be the Julia Runtime and the Standard Library consists of high quality packages that are precompiled and part of the Julia distribution.

[tknopp]

Ok I think in order to move this proposal forward the following things have to be discussed:

1. How to realize this technically. There is solution a) where all the development on the default packages happens on the julia git repo and I see solution b) where running make pulls in the newest version of all default packages and the actual development happens in different repositories. This has both pros an cons. Its more scalable but one has to track several places.
2. The default packages should be precompiled. Should they simply be put into userimg.jl or will we have soon a solution where individual packages can be precompiled.
3. What are the default packages and how to reorganize base. This is probably the hardest thing to answer. I think this is something one simply has to try out. And this is also something that will not happen in a single sweep but will be a gradual process.
4. I would be a fan of calling the whole thing "standard library"

[tkelman]

1. It sounds like some of what will become "default packages" exist currently as parts of base, and some exist currently as separate packages. I think migrating things out of the base repository is easier than migrating packages into it.
2. userimg might be good enough, but it would be great if this can be done seamlessly for any package, default or not, without needing a from-source build of Julia. I don't think going all the way to a dll on Windows is possible yet without having MinGW binutils, but maybe lld will eventually get useful enough for this? Precompiling package images just into .ji files would still speed up load time, right?
3. It's easy and uncontroversial enough to put anything that is removed from base into this category. Choosing which currently-separate packages get promoted to default could be a little more controversial. Maybe auto-collecting usage statistics (the Pkg phone home issue) would allow making this decision more impartially.
4. Guess "Base is the standard library" from https://github.com/JuliaLang/julia/blob/master/doc/manual/modules.rst would need to get tweaked for that choice. "Julia Platform" could work too.

[tknopp]

1. Yes I also think that it would be ideal if this would be possible to do without a from-source build. On the other hand I think this should not be a stopping criterion at this point. If putting things into the system image is what works now I think this is a good solution for now.
2. Minimizing base should be the first driver here. Increasing this stdlib should be done after that and probably needs a defined process.

[JeffBezanson]

I believe this is subsumed by #5155.