Precompile in the cloud

[davidanthoff]

One way to get rid of precompile times for most normal users would be that for every registered package/version, precompiled code is hosted somewhere in the cloud that the package manager automatically pulls when available during package installation. As long as users stay on tagged versions of registered packages, that might lead to a situation where no precompile ever happens on the user machine, which would be a much better experience from a performance point of view.

Obviously a ton of details related to that: where does this get compiled (travis? some julia specific service?), where is it hosted (github? somewhere else?) etc.

[ViralBShah]

This is certainly a good idea, and @aviks may perhaps know what is blocking it (since I have discussed it a bit with him and @vtjnash). If not already possible, we are certainly close.

If we do this, I think that there should be some automatic service to precompile, test, and publish somewhere automatically, and of course Pkg would need to know about it.

[ViralBShah]

Perhaps this is more a discourse discussion - rather than a Julia issue.

[davidanthoff]

I mainly opened the issue so that 1) it can eventually be added to a milestone, 2) I can subscribe to it and follow any progress on it (given that I won't be contributing to it, way out of my league) :)

[ViralBShah]

It is unlikely we can add such a large feature to one milestone. This and #30488 are things that will most likely get done slowly in multiple different releases. It only makes sense to add things to milestones when people are working on things that push us in the right direction. In general, we have moved away in the base repos from having such large milestones.

As far as subscribing, you can do that on discourse for relevant threads and receive updates there as well.

[jpsamaroo]

One thing to note is that currently, precompile files contain absolute paths within them (paths that are only valid on the system they were generated on), making them non-transferable. That problem would probably need to be solved before this issue can be adequately addressed.

[JeffBezanson]

Precompile files are also a function of your package environment. I suppose we'd have to provide files only for certain supported combinations of package versions.

[tkf]

Let me also add that precompilation file may depend on the configuration of each machine, e.g., by embedding the path of the shared library to be used in ccall as a constant.

Precompiling everything after add/update #16409 seems to be easier to do. Maybe add post-add hook registry so that users can do Pkg.after_add(Pkg.precompile)? I posted an issue in JuliaLang/Pkg.jl#973

[jlperla]

For what it is worth, I (1) agree that this is the biggest impediment to getting matlab users to consider Julia; (2) am not sure that this is the best step towards a solution because it could make things more fragile.

To me, the real issue is less that the compiling happens and more that it happens when you don't want it to. Nobody bats an eyelid that an installation of matlab takes an hour or conda updates take forever, because it all happens at once. It is the precompilation after the compilation that is infuriating.

I think the bigger problem, then is increasing the number of things that can be done in the precompilation step (e.g. plotting, etc.)

[ViralBShah]

What do you mean by precompilation after compilation?

[jlperla]

Sorry, switch the word ordering. It is the amount of "compilation after precompilation". Especially with plotting.

I really thing you have cart blanche to spend as much time as you want precompiling on a particular computer as long as it is all happening at predictable "installation/upgrade" event times. In jupyterhub setups those steps can be done in the generation of a particular docker container for the user images. Anything you can move to a precompilation phase makes the experience better for everyone

... Again, for what it is worth.

[ViralBShah]

We have made tremendous progress on this, and it is a topic of ongoing improvement. I think it is fine to close this issue.

[LilithHafner]

I still spend a fair bit of time waiting for precompilation to take place on my machine—I think there is still lots of room for improvement here.

Nevertheless, the points against precompiling in the cloud do seem valid.

[vtjnash]

That is mostly the fault of package authors, either depending on slow packages or making them. The new conditional packages may help with that problem somewhat, as people adopt it.