Provide a mechanism to configure airflow task configurations like pool, queue.

[sravankorumilli]

Description

There's a requirement from the user, where they need to put some additional requirements in the job spec. As the current Optimus version uses Airflow as the scheduler, then the focus for this issue is to address it in Airflow specific only. Others will follow if implemented.

Expectation

• the user is allowed to put additional configuration specific to the Airflow pool and queue

[irainia]

Chosen Approach

The approach taken is to minimise changes in the job spec, but still accommodate the requirements. As it is specific to Airflow, then the focus is only for this.

The following is the approach taken for this issue, which will be implemented in the job spec. Example of the job spec:

...
behavior:
  ...
task:
  ...
metadata:
  resource:
    ...
  airflow: [struct]
    queue: queueA
    pool: poolA

Using struct for the Airflow is to make it easier to process.

References:

• pool
• queue