Ensure compatibility of nf-core download with Tower

[MatthiasZepper]

Description of feature

Although the output format of nf-core download is a self-contained version of a nf-core pipeline, the format is incompatible with sequeralabs Nextflow Tower at the moment.

Therefore, I would like to implment a --tower flag to nf-core download, which modifies the output so that can be directly added to Tower.

[MatthiasZepper]

Brief update on the non-existent progress: The actual implementation in tools is probably not too hard (I have implemented a flag and written a skeleton function in the respective class already), but I first need to know what steps need to be carried out exactly. Unfortunately, I still struggle to reliably add manually downloaded and modified pipelines to Tower.

Typically, this fails with Oops... Unable to process request - Error ID: ... error messages.

A common, but not exclusive, cause seems to be a mysterious lack of credentials: -No credentials found:

Nov-25 12:46:10.758 [io-executor-thread-48801] XXX.XXX.XXX.XXX DEBUG i.s.t.s.pipeline.PipelineServiceImpl - No credentials found for repo=file:/home/XXXXXXXXX/tower_experiment_stuff/experiments/RNAseq_modified.git; user=XX - workspace=XXXXXXXXXXXXXXXX
Nov-25 12:46:10.758 [io-executor-thread-48801] XXX.XXX.XXX.XXX  DEBUG i.s.t.s.pipeline.PipelineServiceImpl - Checking pipeline file:/home/XXXXXXXXX/tower_experiment_stuff/experiments/RNAseq_modified.git - SCM config=(none)
Nov-25 12:46:12.610 [io-executor-thread-48800] XXX.XXX.XXX.XXX  ERROR i.s.t.c.GlobalErrorController - Oops... Unable to process request - Error ID: 5VGiK8Xj4uOvpahre7SEY9

So I am still working on it...sort of.

[MatthiasZepper]

This set of commands does successfully download a pipeline (in this case RNA-seq) and associated configs as one combined bare cloned repo, such that it can be added to Tower via the CLI:

git clone --single-branch https://github.com/nf-core/rnaseq.git
cd rnaseq

# add conf files as submodule and get files
git submodule add https://github.com/nf-core/configs.git conf_institutional
git submodule update --init --recursive

# convert submodule to local folder
git rm --cached conf_institutional
rm -rf conf_institutional/.git
git add conf_institutional
git rm .gitmodules

# make the conf files executable
chmod -R 775 conf_institutional

# update the custom_config_base
sed -i.bak 's=\"https://raw.githubusercontent.com/nf-core/configs/${params.custom_config_version}\"=\"${projectDir}/conf_institutional\"=g' nextflow.config

# commit to version control
git add nextflow.config
git add nextflow.config.bak
git commit -m "Link pipeline with offline institutional configs."

# remove remote
git remote rm origin

# clone working directory into bare directory
git clone --bare --no-hardlinks --dissociate . ../nf-core-rnaseq-3.10.1.git

This version can then be added to Tower like so:

tw pipelines add -n "nf-core-rnaseq" -d "nf-core RNA-seq pipeline version 3.10.1" -w "XXXXXXXXXXXXXXX -c "XXXXXXXXXXXX" -p "uppmax" --work-dir="XXXXXXXXXXXX" --main-script "workflow/main.nf" --schema-name "singularity" --params-file "XXXXXXXXXXXXX/generic_params.yaml" "XXXXXXXXX/tower-pipelines/nf-core-rnaseq-3.10.1.git"

However, I have not yet been able to launch the pipeline successfully, which is likely due to our Tower instance and not due to the pipeline. But without being able to test, developing the feature is somewhat stalled.

[maxulysse]

Can we try that locally?

[MatthiasZepper]

When downloading a repo for Tower, it would be no problem to include all versions / releases of a pipeline, if not for the configs. What would be your take on the way the download should be implemented: Mimicking the current download (that is one version and the configs included in the package) functionality or not include any configs but keep all versions?

In theory, the option to keep all versions but include the configs also for all versions might be possible as well, but quite cumbersome I fear :-(

[mashehu]

What would be the use case why one would like to download more than one version?

[MatthiasZepper]

True, I suppose most people would want to run the latest or a specific older version of the pipeline, which they have validated and verified.

Within a bigger institution, however, different users might desire to run different pipeline versions and Tower just offers this kind of functionality when the pipeline is linked against a regular Git repo. Depending on the number of pipelines managed, it might clutter the UI of Tower if each version needs to be added as a separate pipeline?

[MatthiasZepper]

Didn't realize that this is still open. Done with tools 2.9 🎉 !

Main PR: #2247
Further work and bugfixes: #2308, #2336, #2345, #2346, #2353, #2364