Skip to content

Add CLI command to remove all queues from Celery worker #56195

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
jscheffl merged 1 commit into from
Sep 30, 2025
Merged

Add CLI command to remove all queues from Celery worker #56195

jscheffl merged 1 commit into from
Sep 30, 2025

Conversation

@dheerajturaga
Copy link
Member

@dheerajturaga dheerajturaga commented Sep 29, 2025 •
edited
Loading

Implement command that unsubscribes a Celery worker from all its active queues. This complements the existing command by providing a bulk operation for queue management. Idea is to quickly move a worker out from service without actually shutting it down. Intent is to create a similar effect as Edge Executor's Maintenance Mode

image

cc: @jscheffl

@dheerajturaga
Add CLI command to remove all queues from Celery worker
14568ca 
  Implement  command that unsubscribes
  a Celery worker from all its active queues. This complements the existing
   command by providing a bulk operation for queue management.
jscheffl
jscheffl approved these changes Sep 29, 2025
View reviewed changes
Copy link
Contributor

@jscheffl jscheffl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting feature! Looks good and valid for me.

Would it be a use case as well to add a list of queues again as batch to restore after maintenance?

@dheerajturaga
Copy link
Member Author

dheerajturaga commented Sep 30, 2025 •
edited
Loading

Interesting feature! Looks good and valid for me.

Would it be a use case as well to add a list of queues again as batch to restore after maintenance?

We already have a add-queues method that takes a comma separated queue.

@jscheffl jscheffl merged commit 964997a into apache:main Sep 30, 2025
78 checks passed
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 1, 2025
@dheerajturaga
Add CLI command to remove all queues from Celery worker (apache#56195)
dea3977 
Implement  command that unsubscribes
  a Celery worker from all its active queues. This complements the existing
   command by providing a bulk operation for queue management.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 2, 2025
@dheerajturaga
Add CLI command to remove all queues from Celery worker (apache#56195)
4a95bc1 
Implement  command that unsubscribes
  a Celery worker from all its active queues. This complements the existing
   command by providing a bulk operation for queue management.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 3, 2025
@dheerajturaga
Add CLI command to remove all queues from Celery worker (apache#56195)
f97e30f 
Implement  command that unsubscribes
  a Celery worker from all its active queues. This complements the existing
   command by providing a bulk operation for queue management.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 4, 2025
@dheerajturaga
Add CLI command to remove all queues from Celery worker (apache#56195)
ede60f6 
Implement  command that unsubscribes
  a Celery worker from all its active queues. This complements the existing
   command by providing a bulk operation for queue management.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 5, 2025
@dheerajturaga
Add CLI command to remove all queues from Celery worker (apache#56195)
f209d0a 
Implement  command that unsubscribes
  a Celery worker from all its active queues. This complements the existing
   command by providing a bulk operation for queue management.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 5, 2025
@dheerajturaga
Add CLI command to remove all queues from Celery worker (apache#56195)
0d3dff5 
Implement  command that unsubscribes
  a Celery worker from all its active queues. This complements the existing
   command by providing a bulk operation for queue management.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 7, 2025
@dheerajturaga
Add CLI command to remove all queues from Celery worker (apache#56195)
a9d44b0 
Implement  command that unsubscribes
  a Celery worker from all its active queues. This complements the existing
   command by providing a bulk operation for queue management.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 8, 2025
@dheerajturaga
Add CLI command to remove all queues from Celery worker (apache#56195)
16d2a22 
Implement  command that unsubscribes
  a Celery worker from all its active queues. This complements the existing
   command by providing a bulk operation for queue management.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 9, 2025
@dheerajturaga
Add CLI command to remove all queues from Celery worker (apache#56195)
71ed456 
Implement  command that unsubscribes
  a Celery worker from all its active queues. This complements the existing
   command by providing a bulk operation for queue management.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 10, 2025
@dheerajturaga
Add CLI command to remove all queues from Celery worker (apache#56195)
d59033d 
Implement  command that unsubscribes
  a Celery worker from all its active queues. This complements the existing
   command by providing a bulk operation for queue management.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 11, 2025
@dheerajturaga
Add CLI command to remove all queues from Celery worker (apache#56195)
7308787 
Implement  command that unsubscribes
  a Celery worker from all its active queues. This complements the existing
   command by providing a bulk operation for queue management.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 12, 2025
@dheerajturaga
Add CLI command to remove all queues from Celery worker (apache#56195)
e4f7b72 
Implement  command that unsubscribes
  a Celery worker from all its active queues. This complements the existing
   command by providing a bulk operation for queue management.
dabla pushed a commit to dabla/airflow that referenced this pull request Oct 12, 2025
@dheerajturaga @dabla
Add CLI command to remove all queues from Celery worker (apache#56195)
b3df91b 
Implement  command that unsubscribes
  a Celery worker from all its active queues. This complements the existing
   command by providing a bulk operation for queue management.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 14, 2025
@dheerajturaga
Add CLI command to remove all queues from Celery worker (apache#56195)
65b566f 
Implement  command that unsubscribes
  a Celery worker from all its active queues. This complements the existing
   command by providing a bulk operation for queue management.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 15, 2025
@dheerajturaga
Add CLI command to remove all queues from Celery worker (apache#56195)
e416115 
Implement  command that unsubscribes
  a Celery worker from all its active queues. This complements the existing
   command by providing a bulk operation for queue management.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 17, 2025
@dheerajturaga
Add CLI command to remove all queues from Celery worker (apache#56195)
d023488 
Implement  command that unsubscribes
  a Celery worker from all its active queues. This complements the existing
   command by providing a bulk operation for queue management.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 19, 2025
@dheerajturaga
Add CLI command to remove all queues from Celery worker (apache#56195)
bff59a2 
Implement  command that unsubscribes
  a Celery worker from all its active queues. This complements the existing
   command by providing a bulk operation for queue management.
@eladkal eladkal mentioned this pull request Oct 22, 2025
Closed
69 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jscheffl jscheffl jscheffl approved these changes

@hussein-awala hussein-awala Awaiting requested review from hussein-awala hussein-awala is a code owner

Assignees

No one assigned

Labels

area:providers provider:celery

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@dheerajturaga @jscheffl