Patch: update support for fixed queue workflows

[dapineyro]

Overview

This PR improves AWS batch queue support for Platform Workflows such as "system tools" or "data factory".
Previous to this patch, the queue selection logic for Platform Workflows was the same as for regular workflows:

1. Search for the --job-queue name in the workspace available queues.
2. If not found or if --job-queue was not provided: select the default queue.
3. If there was no default queue, select the newest suitable queue that was available in the workspace.

This logic presents an issue for Platform Workflows. These special type of workflows always run on fixed queues provided by the system and not transparent to the user. Therefore, they ignore any job queue value used. In addition to that, they can also run even if there is no normal queue available in the workspace, as the system queues are always available regardless the workspace. Therefore, the previous logic had two main issues:

1. It states at the stdout of cloudOS-cli that the job will use a queue that is not the real one the job uses.
2. It prevents Platform Workflows from running on workspaces without normal queues created. Platform Workflows are not using normal queues, but due to the cloudos-cli logic, it forces the user to create at least one queue that wouldn't be used anyway.

The present PR removes the queue selection for Platform Workflows:

• It ignores any queue selection provided by the user when using Platform Workflows (with the corresponding Message)
• It does not search for any queue and also does not provide any queue value when using Platform Workflows

Jira

https://lifebit.atlassian.net/browse/LIF-1331

Changes

• Adds the new is_module method to Cloudos class, so we can detect when a workflow is a Module (a.k.a Platform Workflow).
  • From the table generated by Cloudos.process_workflow_list method, we now also extract the isModule column. That column effectively marks all the Platform Workflows: "system-tools", "drug-discovery", "data-factory", "data-factory-omics-etl", "data-factory-omics-insights", "data-factory-data-connection-etl" and "intermediate". "system-tools" are using the fixed "system_tools" queue, "drug-discovery" is using the fixed "drug_discovery" queue and all the others are using "data_factory" fixed queue.
  • If the workflow is detected as a module (isModule=True), then queue selection won't run and no queue value will be passed to the job call.
  • A message will be displayed if the user tries to use a normal queue with a Platform Module. It informs users that the queue will be ignored.
• Adds a specific Pytest
• Updates changelog and README accordingly
• Removes one CI test (curated workflow list) as curated workflows are now deprecated in CloudOS

Tests

Using docker image:

docker run --rm -it quay.io/lifebitaiorg/cloudos-cli:v2.11.1

Global variables:

MY_API_KEY="xxxx"
CLOUDOS="https://cloudos.lifebit.ai"
WORKSPACE_ID="xxxx"
PROJECT_NAME="davidp_testing"

Running a normal workflow specifying a queue

cloudos job run \
    --cloudos-url $CLOUDOS \
    --apikey $MY_API_KEY \
    --workspace-id $WORKSPACE_ID \
    --project-name "$PROJECT_NAME" \
    --workflow-name "Cufflinks pipeline" \
    --job-config "/cloudos/examples/rnatoy.config" \
    --job-name "test_normal_workflow_w_queue" \
    --job-queue "Test_JQ_9VG4W"

Running a normal workflow without specifying a queue (default one is selected)

cloudos job run \
    --cloudos-url $CLOUDOS \
    --apikey $MY_API_KEY \
    --workspace-id $WORKSPACE_ID \
    --project-name "$PROJECT_NAME" \
    --workflow-name "Cufflinks pipeline" \
    --job-config "/cloudos/examples/rnatoy.config" \
    --job-name "test_normal_workflow_w_queue"

Running a normal workflow, specifying a non-existent queue (default one is selected)

cloudos job run \
    --cloudos-url $CLOUDOS \
    --apikey $MY_API_KEY \
    --workspace-id $WORKSPACE_ID \
    --project-name "$PROJECT_NAME" \
    --workflow-name "Cufflinks pipeline" \
    --job-config "/cloudos/examples/rnatoy.config" \
    --job-name "test_normal_workflow_w_queue" \
    --job-queue "a_queue_that_doesnt_exist"

Running a System Tool, specifying a normal queue (message displayed, queue ignored and system_tools queue used)

cloudos job run \
    --cloudos-url $CLOUDOS \
    --apikey $MY_API_KEY \
    --workspace-id $WORKSPACE_ID \
    --project-name "$PROJECT_NAME" \
    --workflow-name "GWAS" \
    --parameter "test=value" \
    --job-queue "Test_JQ_9VG4W"

Running a System Tool without specifying a queue (system_tools queue used)

cloudos job run \
    --cloudos-url $CLOUDOS \
    --apikey $MY_API_KEY \
    --workspace-id $WORKSPACE_ID \
    --project-name "$PROJECT_NAME" \
    --parameter "test=value" \
    --workflow-name "GWAS"

Running a data factory workflow, specifying a normal queue (message displayed, queue ignored and data_factory queue used)

cloudos job run \
    --cloudos-url $CLOUDOS \
    --apikey $MY_API_KEY \
    --workspace-id $WORKSPACE_ID \
    --project-name "$PROJECT_NAME" \
    --workflow-name "Variant Data ETL" \
    --parameter "test=value" \
    --job-queue "Test_JQ_9VG4W"

Running a data factory workflow without specifying a queue (data_factory queue used)

cloudos job run \
    --cloudos-url $CLOUDOS \
    --apikey $MY_API_KEY \
    --workspace-id $WORKSPACE_ID \
    --project-name "$PROJECT_NAME" \
    --parameter "test=value" \
    --workflow-name "Variant Data ETL"

[l-mansouri]

LGTM! I've left just 2 comments!

[l-mansouri]

Thank you @dapineyro