Feat: implements nextflow_version new parameter for job submission

[dapineyro]

Overview

This PR implements the following new features:

• New --nextflow-version parameter to select the Nextflow version to execute on CloudOS
• Now it accepts cloudOS URLs with or without a trailing '/' (previous behaviour was failure when using URL with a trailing '/')

Jira

https://lifebit.atlassian.net/browse/LIF-1124

Acceptance criteria

See: https://lifebit.atlassian.net/wiki/spaces/DEL/pages/1223491604/LIF-1222+CloudOS+CLI+support+for+multiple+nextflow+versions+LP-16142#Proposed-state

Changes

• Adds the new parameter --nextflow-version :
  • It allows for only three possible string values: '22.10.8' and '24.04.4'. (current supported versions on CloudOS) and 'latest' that selects '24.04.4' too. Default='22.10.8'
  • There is a warning (same as in the platform) when selecting other than '22.10.8'.
  • System tools, target-id, etl pipelines (modules) only allow for '22.10.8' (with message if using different)
• Some messages have been improved for consistency
• Some docstrings have been improved for consistency
• UX improvement: the user can now specify the url with or without a trailing '/', so that now https://cloudos.lifebit.ai/ and https://cloudos.lifebit.ai are acceptable (previously, https://cloudos.lifebit.ai/ raised an error with no clear message)

Tests

Using docker image:

docker run --rm -it quay.io/lifebitaiorg/cloudos-cli:v2.12.0

Global variables:

MY_API_KEY="xxxx"
CLOUDOS="https://cloudos.lifebit.ai"
CLOUDOS2="https://cloudos.lifebit.ai/"
WORKSPACE_ID="xxxx"
PROJECT_NAME="davidp_testing"
WORKFLOW_NAME="Cufflinks"
WORKFLOW_NAME2="GWAS"
JOB_PARAMS="/cloudos/examples/rnatoy.config"

Normal workflow default job run

cloudos job run \
    --cloudos-url $CLOUDOS \
    --apikey $MY_API_KEY \
    --workspace-id $WORKSPACE_ID \
    --project-name "$PROJECT_NAME" \
    --workflow-name $WORKFLOW_NAME \
    --job-config $JOB_PARAMS \
    --job-name cloudos_cli_default

Normal workflow --nextflow-version=22.10.8

cloudos job run \
    --cloudos-url $CLOUDOS \
    --apikey $MY_API_KEY \
    --workspace-id $WORKSPACE_ID \
    --project-name "$PROJECT_NAME" \
    --workflow-name $WORKFLOW_NAME \
    --job-config $JOB_PARAMS \
    --nextflow-version 22.10.8 \
    --job-name cloudos_cli_22.10.8

Normal workflow --nextflow-version=24.04.4

cloudos job run \
    --cloudos-url $CLOUDOS \
    --apikey $MY_API_KEY \
    --workspace-id $WORKSPACE_ID \
    --project-name "$PROJECT_NAME" \
    --workflow-name $WORKFLOW_NAME \
    --job-config $JOB_PARAMS \
    --nextflow-version 24.04.4 \
    --job-name cloudos_cli_24.04.4

Normal workflow --nextflow-version=latest

cloudos job run \
    --cloudos-url $CLOUDOS \
    --apikey $MY_API_KEY \
    --workspace-id $WORKSPACE_ID \
    --project-name "$PROJECT_NAME" \
    --workflow-name $WORKFLOW_NAME \
    --job-config $JOB_PARAMS \
    --nextflow-version latest \
    --job-name cloudos_cli_latest

--nextflow-version=wrong.version

cloudos job run \
    --cloudos-url $CLOUDOS \
    --apikey $MY_API_KEY \
    --workspace-id $WORKSPACE_ID \
    --project-name "$PROJECT_NAME" \
    --workflow-name $WORKFLOW_NAME \
    --job-config $JOB_PARAMS \
    --nextflow-version wrong.version \
    --job-name cloudos_cli_wrong

System tool default job

cloudos job run \
    --cloudos-url $CLOUDOS2 \
    --apikey $MY_API_KEY \
    --workspace-id $WORKSPACE_ID \
    --project-name "$PROJECT_NAME" \
    --workflow-name $WORKFLOW_NAME2 \
    --nextflow-profile "test_ci_staging_cloudos_phenofile" \
    --job-name cloudos_cli_module_default

System tool --nextflow-version=24.04.4

cloudos job run \
    --cloudos-url $CLOUDOS2 \
    --apikey $MY_API_KEY \
    --workspace-id $WORKSPACE_ID \
    --project-name "$PROJECT_NAME" \
    --workflow-name $WORKFLOW_NAME2 \
    --nextflow-profile "test_ci_staging_cloudos_phenofile" \
    --nextflow-version 24.04.4 \
    --job-name cloudos_cli_module_24.04.4

[danielboloc]

Great implementation! LGTM

[s0rthak]

Overall looks good to me. Good job 👍🏼

Added one code quality suggestion that can be implemented before merging and a wider logger improvement that we can explore in a different PR.