dbt with kubernetes

[rtooker]

Similar to #491 but I think subtly different.

I'm using

• apache-airflow==2.6.3
• astronomer-cosmos==1.0.5

DbtTaskGroup has a mandatory profile_config arg, but when set this seems to be passed through to the DbtRunKubernetesPodOperator causing in exception

with DAG(
    Path(__file__).stem,
    description="*****",
    schedule=None,
    start_date=datetime(2023,1,1),
    catchup=False,
) as dag:
    dbt_task_group = DbtTaskGroup(
        project_config=ProjectConfig(
            dbt_project_path = Path(__file__).parent / 'dbt',
            manifest_path=Path(__file__).parent / 'manifests' / 'manifest.json',
        ),
        render_config=RenderConfig(
            load_method=LoadMode.DBT_MANIFEST,
        ),
        execution_config=ExecutionConfig(
            execution_mode=ExecutionMode.KUBERNETES,
        ),
        profile_config=ProfileConfig(
            profile_name='*****',
            target_name='*****'
        ),
        operator_args={
           ...
        }
    )



if __name__ == "__main__":
    dag.test()

yields

airflow.exceptions.AirflowException: Invalid arguments were passed to DbtRunKubernetesOperator (task_id: ******). Invalid arguments were:
**kwargs: {'profile_config': ProfileConfig(profile_name='****', target_name='****', profiles_yml_filepath=None, profile_mapping=None)}

[adityasharma1685]

After moving my DBT project under Airflow Dags folder, I am getting same error.

[david-mag]

I've got the same issue. The step-by-step guide in the documentation doesn't look like its up to date either. It uses imports (from cosmos.providers.dbt.task_group import DbtTaskGroup) that don't work with the latest version of cosmos anymore.

[david-mag]

I've added a pull request for an attempt to fix this issue here.

[rtooker]

Thanks @david-mag .
I'm now getting the same issue as reported in #505 (comment)

airflow.exceptions.AirflowException: Invalid arguments were passed to DbtRunKubernetesOperator (task_id: XXXX). Invalid arguments were:
**kwargs: {'emit_datasets': True}

[david-mag]

As far as I can see, with the release of this fix, more "unknown" arguments have been added, which broke the kubernetes DAGs and TaskGroups again right away.

[adityasharma1685]

While trying latest release(1.1.1) I ran into another issue and getting below exception:

from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator
except ImportError:
    raise ImportError(
        "Could not import KubernetesPodOperator. Ensure you've installed the Kubernetes provider "
        "separately or with with `pip install astronomer-cosmos[...,kubernetes]`."
    )

https://github.com/astronomer/astronomer-cosmos/blob/main/cosmos/operators/kubernetes.py#L21-L26
I am using Airflow=2.5.3 and apache-airflow-providers-cncf-kubernetes==5.2.2 .

It seems like the latest version of the cosmos is using the Kubernetes >= 7.4 though the toml file says kubernetes version >=5.1 < 7.3

[tatiana]

Thanks for the details, @rtooker @adityasharma1685 @david-mag, for all the support and information.

I believe the release astronomer-cosmos 1.1.2 will solve the pending issues described in this thread:
https://github.com/astronomer/astronomer-cosmos/releases/tag/astronomer-cosmos-v1.1.2
https://pypi.org/project/astronomer-cosmos/1.1.2/

Please, do reach out if any of them persist.