How to use it with multi python projects on Airflow

[gudata]

Hello,
This project is excellent and gives a lot of usefulness and verbosity for airflow pipelines.

My question is probably more airflow related rather than cosmos.
I would like to know how to cosmos when all the project dependencies are self-contained in a docker image?

The architecture where the dags and the tasks share the same runtime python environment isn't scalable. I have a case where I would like another version of python, another use case is when I have old/new dbt version which is not compatible with the airflow requirements. Another use case is when airflow is running dags/tasks from different teams and technical stacks.

For those reasons, I choose to use airflow as a scheduler and pack all the task codes in docker images.

Could you please share your experience on how you use this project in airflow installation where a lot of teams runs a lot of pipelines with diffent stacks?

[jlaneve]

Hey @gudata! Good question. What we do internally is install dbt into a virtual environment in the Docker image - we've documented how to it in the docs here. However, there's been some talk about running dbt/Cosmos tasks in a similar manner to the Kubernetes Pod Operator, where you build a Docker image specifically for the task. If that's what you're looking for, feel free to add an issue (or open a PR!) and if it gets enough traction we could prioritize it.