Enable triggerer queues

[zach-overflow]

---

Closes: #33818
Partially addresses / related: #59871

What?

• Adds optional support for task-based trigger queue assignment, which allows a given triggerer to consume from a user-configured subset of triggers, based on their origin tasks' queue value.
  • Adds an optional --queues CLI option to the triggerer, to constrain a triggerer to only consume Triggers from a given list of task queues.
  • Adds a new triggerer.queues_enabled config entry, which when set to True enables the trigger queue feature (False by default).
  • Adds a queue column to the trigger table, which is set automatically by the task runner when triggerer.queues_enabled is set to True.
• Adds relevant documentation updates for this new feature.

Why?

• AIP-67 (multi-tenancy support), to enable team-specific triggerer assignment.
• More generally, and not necessarily specific to multi-tenancy use-cases, to allow for other triggerer assignment strategies in an HA setup.

Testing

• Tested the example HITL dag in docker-compose; once with triggerer.queues_enabled=False and once with triggerer.queues_enabled=True to ensure deferrable tasks worked in either setup.
• standard collection of breeze and prek testing commands
• Ran uv run --group docs build-docs --autobuild to check the new doc section rendering.

Note on event-based triggers and async Callbacks

Following from the helpful discussion here, this PR aims to assign a trigger's queue from its deferring task's queue. As such, this PR does not add any triggerer assignment controls for event-based triggers or TriggererCallback-based triggers. This was done intentionally both to keep this PR's scope reasonable, and because the latter 2 types of triggers do not necessarily have a relation to a singular task / task queue. If it is preferred, I'm happy to create a followup issue for adding queue support in these types of triggers. I've also made sure to make this explicit in the user documentation, and added details in the docs on how to use this feature without disrupting any potential usage of event-based or callback-based triggers.

[jason810496]

Thanks for adding the nice feature!

Would it be better that Triggerer only add the SQL filter if user specify --consume-trigger-queues from CLI without adding the new configs?

Not sure if this would be not only less ambiguous for users but also make the SQL unchanged by only applying the trigger_queue.in_ filter when users explicitly pass the --consume-trigger-queues flag. In my opinion, this preserves the feature while minimizing changes.

[zach-overflow]

Thanks for adding the nice feature!

Would it be better that Triggerer only add the SQL filter if user specify --consume-trigger-queues from CLI without adding the new configs?

Not sure if this would be not only less ambiguous for users but also make the SQL unchanged by only applying the trigger_queue.in_ filter when users explicitly pass the --consume-trigger-queues flag. In my opinion, this preserves the feature while minimizing changes.

Thanks for the suggestion, I like that idea and am all in favor of reduced ambiguity. It also seems like what your suggesting will fall more in line with the celery queues controls (i.e. the queue(s) a celery worker consumes from are optionally set by the CLI command).

[jscheffl]

For me it is good then

[jason810496]

#protm , it's a really nice feature for 3.2!

[Dev-iL]

Great job!

[potiuk]

Indeed!

[o-nikolas]

AIP-67 (multi-tenancy support), to enable team-specific triggerer assignment.

We won't be able to rely on the queue field for the multi-tenancy use-case. But I'm sure there will be lots of plumbing we can reuse from this!