Improve packaging and distribution system

[pditommaso]

Summary

Nextflow application is a packaged as a set of JAR files (Java archives) that are uploaded to Maven central repository with the exception of the launcher app that's stored in http://nextflow.io/releases/

The nextflow "binary" is a mere Bash script that checks if the launcher app is stored locally. If it cannot be found it download it from the http://nextflow.io/releases URL and run it.

Then the launcher app, uses the Capsule package manager to download the main application JARs from Maven along with all required dependencies files.

This system worked quite nicely for many years, however there are some serious open issues:

• The Capsule system has been is not maintained anymore by the original author. Considering the new fast release cadence of the JVM this can because soon a problem (see below)
• Recent Java release has introduced a slightly different version scheme that the Capsule system fail to recognise. This creates problems with some users that need to change Java distribution to use Nextflow. See here and here and here
• The Capsule system breaks when using _JAVA_OPTIONS variable. See here
• The Capsule system fails to download and unpack the deps in some systems because it requires to create file system lock, which not all file system supports. See here
• The new nextflow plugins system makes the capsule system not needed anymore.

Goals

• Remove the dependency on Capsule package system
• Create instead a standalone nextflow distribution for the core runtime. The Nextflow plugin system will continue to download the required plugins separately
• Keep the compatibility with the existing nextflow launcher wrapper

Non-goals

• the support for JReleaser or other Linux package managers such as RPM, Debian, etc. is out of the scope of this issue, however, it's desirable that the overall design to be compatible with the packaging those system, that could be adopted in the future

[ewels]

It would be great to use this as an opportunity to move the installation files away from http://nextflow.io/releases/ if possible. It terrifies me having it on the same domain as the website, which is being changed on a regular basis. It would be pretty easy to accidentally break installations by creating the wrong web page URLs there, for example.

It would be better to use a dedicated subdomain that only serves the installation assets, so that it's kept entirely separate from the website hosting. For example, we could use https://download.nextflow.io

We could leave the existing assets at http://nextflow.io/releases/ or potentially set up redirects to the new download link.

[ewels]

Another thing to consider whilst we're here: it would be great to be able to easily install snapshots of Nextflow that are more fine-grained than edge releases, without needing to build locally.

We already build Nextflow on every push to master as part of the GitHub CI automation. Hopefully we could simply save those files as GitHub Actions artefacts and then hook them up to the installer script so that this kind of syntax would work:

NXF_VER=0a8a484 curl -s https://get.nextflow.io/ | bash

GitHub artefacts are deleted after 90 days, so we may want to mirror them to storage somewhere else if that's not too difficult. Doing so may also make it easier for the installer script to work.

[ewels]

It would be trivial to take this same approach ☝🏻 and do it with a cron GitHub Actions trigger instead, to have nightly builds if that is preferred.

Note that we could also set up some redirects to have shortcut versions such as latest or nightly etc which point to the latest bleeding edge version for convenience.

[pditommaso]

Nextflow is finally Capsule free as for version 24.07.0-edge 🎉