Skip to content

Conversation

@Miretpl
Copy link
Contributor

@Miretpl Miretpl commented Jun 5, 2025

closes: #28880

Currently, in the Helm Chart, there is a workers field whose values are shared between Celery and Kubernetes executors workers. The only possible way to configure both separately is by using the podTemplate field, which overrides the whole pod-template-file content. However, there is no easy possibility to override part of the configuration or to share the same configuration between workers.

This PR introduces the workers.celery and workers.kubernetes files, which contain specific fields used by each executor worker. Every field specified under the above sections overrides the value of the respective fields under the workers section. In case of configuring the field under workers section, without configuring different values under workers.celery or workers.kubernetes sections, it will result in applying the change to both worker types. I added comments about deprecation to every field which was used only by one executor type. Also, I implemented the separation of service accounts used by pod-template-file and Airflow Celery workers with the usage of workers.useWorkerDedicatedServiceAccounts flag. By default, both workers will use a single common service account, and the fields workers.celery.serviceAccount and workers.kubernetes.serviceAccount will have no effect.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@boring-cyborg boring-cyborg bot added the area:helm-chart Airflow Helm Chart label Jun 5, 2025
@Miretpl Miretpl changed the title Separate workers config Introduce workers.celery and workers.kubernetes config sections Jun 5, 2025
@Miretpl
Copy link
Contributor Author

Miretpl commented Jun 5, 2025

Currently, the introduction of workers.celery section should be fully finished.

@Miretpl Miretpl force-pushed the separate-workers-config branch from 94a1b5a to 12ba007 Compare June 13, 2025 17:51
@Miretpl
Copy link
Contributor Author

Miretpl commented Jun 13, 2025

Both sections workers.celery and workers.kubernetes should be finished. I would appreciate any help with tests, due to the number of changes in the PR.

@github-actions
Copy link

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Jul 30, 2025
@Miretpl
Copy link
Contributor Author

Miretpl commented Jul 30, 2025

Not stale - I'm during rebase 😄

@github-actions github-actions bot removed the stale Stale PRs per the .github/workflows/stale.yml policy file label Jul 31, 2025
@Miretpl Miretpl force-pushed the separate-workers-config branch from 401b8d1 to b7067b0 Compare August 7, 2025 19:31
@github-actions
Copy link

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Sep 25, 2025
@github-actions github-actions bot closed this Sep 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:helm-chart Airflow Helm Chart stale Stale PRs per the .github/workflows/stale.yml policy file

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Can't configure Kubernetes and Celery workers in Helm Chart

1 participant