Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for worker pools that exclude specific queues [Green specs] #1198

Open
wants to merge 10 commits into
base: master
Choose a base branch
from

Conversation

dmytro-strukov
Copy link

Copy of #1148, but with cleanup and green specs

Delayed Job already supports worker pools that draw from specific queues. This is useful to ensure those tasks are processed in a timely manner and do not starve. Coupling that with a pool that processes all jobs is a common practice. However, it is possible to create so many jobs for a specific that other jobs are delayed to an unacceptable degree. In this case we want to have a worker pool that only processes jobs that are not from that queue (or queues). There is currently no mechanism for this, hence this PR.

The changes allow worker pools to be named with a ! prefix which would indicate that we want this pool to process any jobs except those in the given queues. It is intended to be coupled with a change at the adapter level which would read the exclude_specified_queues flag and return the appropriate jobs based on whether the list of queues is intended as an inclusion or exclusion filter.

See this PR for changes made to DJ active_record adapter to support this new feature collectiveidea/delayed_job_active_record#194.

one-more-alex and others added 10 commits March 18, 2019 11:06
Co-authored-by: Nicholas Jakobsen <nicholas.jakobsen@gmail.com>
Co-authored-by: Nicholas Jakobsen <nicholas.jakobsen@gmail.com>
The addition of `exclude_specified_queues` allowed all specified queues to be treated as exclusions, implying that all jobs in those queues not be handled by the workers. However, this only allows application of this behaviour globally. What we want is the ability to create two or more pools, where some pools handle jobs from the given queues and other pools handle jobs that are not from the given queues.

To implement this, we check if the `exclude_specified_queues` is not set and treat this as a request for automatic detection of exclusions. By adding a "!" to the beginning of a queue list, only the workers that handle that list of queues will behaves as if the `exclude_specified_queues` flag had been set.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants