Automatically install all registries available from pkg server.

[GunnarFarneback]

Background discussion: JuliaPackaging/PkgServer.jl#26

This only makes a difference if you have a JULIA_PKG_SERVER that serves at least one registry that is not the General registry.

Without this PR, the only way to install such a registry is by UUID, e.g.

pkg> registry add 1e95ee64-cd1a-42e9-a29a-16f7a437e6bf

With this PR it is installed automatically, in the same way that General is installed automatically today. More precisely all registries served by the pkg server are installed automatically.

[GunnarFarneback]

The second commit implements @fredrikekre's proposal from JuliaPackaging/PkgServer.jl#26 (comment) that registry add without arguments installs all known registries (General and whatever is served by the package server) that are not already installed. This is a complement to the first commit that installs all known registries automatically when none is installed.

[KristofferC]

This seems kinda odd / non-composable to me due to that you can only have one active PkgServer at a time. It would make more make sense to me if there would be a possibility of having a list of PkgServers and each PkgServer says what registries it "tracks". Then for a PkgServer serving packages from a custom registry, you add that server to the list and when you download packages from that custom registry it will try get them from the new PkgServer.

With the current implementation, what does the workflow look like? You want to have easy access to a custom registry so you change your PkgServer to a custom one but now all your packages are forced to be downloaded through the custom PkgServer. Seems like a big side effect for the advantage of not having to manually add the registry.

Something like

[GunnarFarneback]

Being able to have a list of package servers would definitely be nice too and make some things more flexible, but it's not a requirement for this PR to be useful.

The workflow is that you set up your own package server that can serve both your own registry and General and point JULIA_PKG_SERVER to it. This indeed means that all packages will have to be downloaded through your own package server, but from the point of view of a company, for example, this is quite useful since the packages will be cached within your local network and will be blazingly fast to download, not least in CI where you may instantiate environments from scratch repeatedly.

When it comes to your local packages, serving them from a package server rather than from git repositories has the same advantages as General packages have. But in addition your local packages probably require some form of authentication when obtaining them from git, e.g. ssh keys. Having them served from a package server gives more flexibility for authentication.

From the point of view of a user of your Julia code within your organization, the setup scenarios roughly look as follows. (More specifically, this is how it would look where I am.)

With local packages served through git:

* Download Julia and install.
* Set up ssh keys.
* Start Julia and do
pkg> registry add General
pkg> registry add <somewhat complicated URL to your local registry git repository>

With local packages served through package server but without this PR:

* Download Julia and install.
* Set JULIA_PKG_SERVER to your local package server host.
* Start Julia and do
pkg> registry add General
pkg> registry add <totally arbitrary string of characters that happen to be the UUID of your registry>

With local packages served through package server and with this PR:

* Download Julia and install.
* Set JULIA_PKG_SERVER to your local package server host.

Of course these processes can be documented in copy and paste friendly instructions, but the perception of the installation and setup will be much more positive in the last case.

[GunnarFarneback]

The workflow is that you set up your own package server that can serve both your own registry and General

And here is the tool to do that: https://github.com/GunnarFarneback/LocalPackageServer.jl

[StefanKarpinski]

Great PR. This looks good to me.

[StefanKarpinski]

Indeed, the reason there's no stack of package servers is that the point of a package server is to be the single point of contact for all the things the client can install. When you instantiate a manifest, it goes through that one package server which knows how to get you all the things you have access to. Therefore, in the future, if you're talking to the same package server, you will always be able to instantiate the same manifest again. If there was a stack of package servers, then this guarantee falls apart: you instantiate a manifest using some mix of package servers and later, unless you have the same set of package servers, there's no guarantee that the manifest can be instantiated.

One of the other points of the design is that from a systems administration point of view, there should be a single server that needs to be whitelisted. If we do a stack of package servers, then your sysadmins need to make sure you can access all of those servers or things start to get wonky—and in a very unlcear way because some packages work but others don't. Instead, this design means that you can either access the package server or you can't. It's a very simple request to make of a sysadmin: they either whitelist HTTPS access to a single external address, or they run a local package server so that they can control what you can / can't install.

This is also why the Pkg Server is a cache and the Storage Protocol is a strict subprotocol of the Pkg Protocol—you can easily chain them. If you want to make General accessible to your local users, you can use pkg.julialang.org as an upstream storage server.

[StefanKarpinski]

We should get this merged today so that it goes into 1.5.

[fredrikekre]

Maybe this can just replace install_default_registries? I mean, maybe the registry add no-arg is not needed?

[StefanKarpinski]

That's a good point. I do think this should be done automatically, so why is registry add needed?

[GunnarFarneback]

The first commit only adds the served registries if none is installed (in the same way that happens for General today). This has two weaknesses:

• If you already had some other registry installed before hooking up the package server, you first need to remove it unless you want to add the served registries by UUID. registry add without arguments would solve this problem.
• If you want to install another non-served registry (e.g. at the start of an install from scratch for CI use), you need some way to trigger the automatic install of the served registries first and registry add without arguments would be a reasonable way to do that.

A different approach would be to always add missing served registries whenever doing Pkg operations but I didn't try to go that far with this PR and I have a suspicion that it would turn out to be annoying at times. But it's certainly a possibility.

[fredrikekre]

But the usual case will be that the server and Julia install is set up by som admin before the user enters the picture, right? So in the case where you have experimented with Julia before, maybe you can just reset it yourself too?

[StefanKarpinski]

I had imagined just auto-installing all registries advertised by the current pkg server. In what way do you think that might get annoying?

[GunnarFarneback]

I can imagine myself wanting to remove a registry to test something, only to have it bounce back immediately.

[GunnarFarneback]

But the usual case will be that the server and Julia install is set up by som admin before the user enters the picture, right? So in the case where you have experimented with Julia before, maybe you can just reset it yourself too?

I'd rather cover the general case than the usual case, but no, at the scale (and type) of company I'm working, there is no admin who installs anything on people's computers.

[StefanKarpinski]

Ok, I'd say let's go with this for now.