Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support of custom celery configs #45037

Open
2 tasks done
arorasachin9 opened this issue Dec 18, 2024 · 1 comment · May be fixed by #45038
Open
2 tasks done

Support of custom celery configs #45037

arorasachin9 opened this issue Dec 18, 2024 · 1 comment · May be fixed by #45038
Labels
area:providers kind:feature Feature Requests provider:celery

Comments

@arorasachin9
Copy link

Description

Currently airflow supports few celery configs only like worker concurrency, worker prefetch multiplier etc. All others config are not supported. One of main out of them is worker max tasks per child which helps in recycling the celery worker processes. It helps in reclaiming the resources of the processes which got consumed due to high resource utilising task.

Use case/motivation

We have a task which consumes very high memory. As task keeps on changing the processes so our memory keeps on increasing. To recycle the process after certain number of task is the celery support but currently not supported in airflow.

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct
@arorasachin9 arorasachin9 added kind:feature Feature Requests needs-triage label for new issues that we didn't triage yet labels Dec 18, 2024
@boring-cyborg boring-cyborg
Copy link

boring-cyborg bot commented Dec 18, 2024

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

arorasachin9 pushed a commit to arorasachin9/airflow that referenced this issue Dec 18, 2024
@sachin-arora-cashfree
apache#45037: Support for additional celery config directly from airf…
1a172e5 
…low.cfg file
@arorasachin9 arorasachin9 linked a pull request Dec 18, 2024 that will close this issue
Open
arorasachin9 pushed a commit to arorasachin9/airflow that referenced this issue Dec 18, 2024
@sachin-arora-cashfree @arorasachin9
apache#45037: Support for additional celery config directly from airf…
28d1806 
…low.cfg file
arorasachin9 pushed a commit to arorasachin9/airflow that referenced this issue Dec 18, 2024
@sachin-arora-cashfree @arorasachin9
apache#45037: Support for additional celery config directly from airf…
6ad3bc4 
…low.cfg file
@nathadfield nathadfield removed the needs-triage label for new issues that we didn't triage yet label Dec 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area:providers kind:feature Feature Requests provider:celery
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add support for custom celery configs arorasachin9/airflow
2 participants
@nathadfield @arorasachin9