Feat: makes AWSbatch the default executor

[dapineyro]

Overview

This PR makes AWS batch the default executor, so there is no need to specify --batch for the cloudos job run command.

Jira ticket

DEL-17662 & LP-6350

Changes

• Adds new --ignite flag to allow ignite executor if available. Using it shows a warning message to alert the user the command may fail.
• Makes --batch flag unnecessary, although using it will not break the command, to maintain backwards compatibility.
• Changes the queue selection when the user do not specify --job-queue.
  • Before: cloudos-cli selected the last available queue.
  • Now: cloudos-cli selects the workspace "default" queue, if available. If not, selects the last available queue, as before.
• Bumps version to v2.7.0.
• Updates CHANGELOG.md.
• Updates CI tests for not using --batch flag.
• Updates README.md and adds a GitHub actions badge.
• Updates --help messages.

Tests

Setting up variables:

MY_API_KEY="xxxxx"
CLOUDOS="https://cloudos.lifebit.ai"
WORKSPACE_ID="xxxxx"
PROJECT_NAME="API jobs"
WORKFLOW_NAME="rnatoy"
JOB_PARAMS="cloudos/examples/rnatoy.config"

Running docker image:

docker run --rm -it quay.io/lifebitaiorg/cloudos-cli:v2.7.0

CloudOS job using the old --batch flag

cloudos job run \
    --cloudos-url $CLOUDOS \
    --apikey $MY_API_KEY \
    --workspace-id $WORKSPACE_ID \
    --project-name "$PROJECT_NAME" \
    --workflow-name $WORKFLOW_NAME \
    --job-config $JOB_PARAMS \
    --resumable \
    --spot \
    --batch \
    --job-name "test_1_cloudos_cli"
CloudOS python package: a package for interacting with CloudOS.

Version: 2.7.0

CloudOS job functionality: run and check jobs in CloudOS.

Executing run...
	No job_queue was specified, using the CloudOS default queue: On_demand_standard_stable_3TB.
	Job successfully launched to CloudOS, please check the following link: https://cloudos.lifebit.ai/app/jobs/65fb3a6d39195838201939c6
	Your assigned job id is: 65fb3a6d39195838201939c6

	Your current job status is: initializing
	To further check your job status you can either go to https://cloudos.lifebit.ai/app/jobs/65fb3a6d39195838201939c6 or use the following command:
	cloudos job status \
		--apikey $MY_API_KEY \
		--cloudos-url https://cloudos.lifebit.ai \
		--job-id 65fb3a6d39195838201939c6

CloudOS job not using any flag (new default)

cloudos job run \
    --cloudos-url $CLOUDOS \
    --apikey $MY_API_KEY \
    --workspace-id $WORKSPACE_ID \
    --project-name "$PROJECT_NAME" \
    --workflow-name $WORKFLOW_NAME \
    --job-config $JOB_PARAMS \
    --resumable \
    --spot \
    --job-name "test_2_cloudos_cli"
CloudOS python package: a package for interacting with CloudOS.

Version: 2.7.0

CloudOS job functionality: run and check jobs in CloudOS.

Executing run...
	No job_queue was specified, using the CloudOS default queue: On_demand_standard_stable_3TB.
	Job successfully launched to CloudOS, please check the following link: https://cloudos.lifebit.ai/app/jobs/65fb3b37391958382019648d
	Your assigned job id is: 65fb3b37391958382019648d

	Your current job status is: initializing
	To further check your job status you can either go to https://cloudos.lifebit.ai/app/jobs/65fb3b37391958382019648d or use the following command:
	cloudos job status \
		--apikey $MY_API_KEY \
		--cloudos-url https://cloudos.lifebit.ai \
		--job-id 65fb3b37391958382019648d

CloudOS job using an existing queue

cloudos job run \
    --cloudos-url $CLOUDOS \
    --apikey $MY_API_KEY \
    --workspace-id $WORKSPACE_ID \
    --project-name "$PROJECT_NAME" \
    --workflow-name $WORKFLOW_NAME \
    --job-config $JOB_PARAMS \
    --resumable \
    --spot \
    --job-queue "end_to_end" \
    --job-name "test_3_cloudos_cli"
CloudOS python package: a package for interacting with CloudOS.

Version: 2.7.0

CloudOS job functionality: run and check jobs in CloudOS.

Executing run...
	Job successfully launched to CloudOS, please check the following link: https://cloudos.lifebit.ai/app/jobs/65fb3c2b391958382019b9c9
	Your assigned job id is: 65fb3c2b391958382019b9c9

	Your current job status is: initializing
	To further check your job status you can either go to https://cloudos.lifebit.ai/app/jobs/65fb3c2b391958382019b9c9 or use the following command:
	cloudos job status \
		--apikey $MY_API_KEY \
		--cloudos-url https://cloudos.lifebit.ai \
		--job-id 65fb3c2b391958382019b9c9

CloudOS job using a non-existing queue

cloudos job run \
    --cloudos-url $CLOUDOS \
    --apikey $MY_API_KEY \
    --workspace-id $WORKSPACE_ID \
    --project-name "$PROJECT_NAME" \
    --workflow-name $WORKFLOW_NAME \
    --job-config $JOB_PARAMS \
    --resumable \
    --spot \
    --job-queue "a_non_existing_queue" \
    --job-name "test_4_cloudos_cli"
CloudOS python package: a package for interacting with CloudOS.

Version: 2.7.0

CloudOS job functionality: run and check jobs in CloudOS.

Executing run...
	Queue 'a_non_existing_queue' you specified was not found, using the CloudOS default queue instead: On_demand_standard_stable_3TB.
	Job successfully launched to CloudOS, please check the following link: https://cloudos.lifebit.ai/app/jobs/65fb3cdc391958382019dcb5
	Your assigned job id is: 65fb3cdc391958382019dcb5

	Your current job status is: initializing
	To further check your job status you can either go to https://cloudos.lifebit.ai/app/jobs/65fb3cdc391958382019dcb5 or use the following command:
	cloudos job status \
		--apikey $MY_API_KEY \
		--cloudos-url https://cloudos.lifebit.ai \
		--job-id 65fb3cdc391958382019dcb5

CloudOS job trying to use --ignite in a non-ignite workspace

cloudos job run \
    --cloudos-url $CLOUDOS \
    --apikey $MY_API_KEY \
    --workspace-id $WORKSPACE_ID \
    --project-name "$PROJECT_NAME" \
    --workflow-name $WORKFLOW_NAME \
    --job-config $JOB_PARAMS \
    --resumable \
    --spot \
    --ignite \
    --job-name "test_5_cloudos_cli"
CloudOS python package: a package for interacting with CloudOS.

Version: 2.7.0

CloudOS job functionality: run and check jobs in CloudOS.

Executing run...

[Warning] You have specified ignite executor. Please, note that ignite is being removed from CloudOS, so the command may fail. Check ignite availability in your CloudOS

Traceback (most recent call last):
  File "/opt/conda/bin/cloudos", line 33, in <module>
    sys.exit(load_entry_point('cloudos', 'console_scripts', 'cloudos')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/cloudos/__main__.py", line 313, in run
    j_id = j.send_job(job_config=job_config,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/cloudos/jobs/job.py", line 534, in send_job
    raise BadRequestException(r)
cloudos.utils.errors.BadRequestException: Server returned status 400. Reason: Bad Request

NOTE: I was not able to find a workspace with ignite support, but as I don't have access to all the existing workspaces, we will maintain the --ignite option until we can confirm 100% there is no ignite workspace left and we are not going to support it again in the future.

[danielboloc]

LGTM! Great new feature @dapineyro! And is great that it still maintains backwards compatibility so we don't have to change all our CI tests