This repository has been archived by the owner on Oct 24, 2024. It is now read-only.
i718 Add auxiliary worker to run different queues #723
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add a second worker that runs all GoodJob queues. The "base" worker will run all queues except the :auxiliary queue. This, combined with the high priority assigned to all jobs in the :auxiliary queue, effectively builds a fast lane; all jobs run in the :auxiliary queue will be executed before all other jobs by the auxiliary worker. The fact that the base worker does not run any jobs in the :auxiliary queue means we can have this fast lane without putting every other job on hold for, potentially, long periods of time. P.S. The reason we use `before_enqueue` in ApplicationJob is because GoodJob does not support setting a default priority per queue.
This resolves the "duplicate name" warning when deploying. However, it may remove all other ENV vars from the base worker, so this change may not stick around
This is an attempt to resolve this helm upgrade error: > The order in patch list: ... doesn't match $setElementOrder list: ...
|
| GitGuardian id | GitGuardian status | Secret | Commit | Filename | |
|---|---|---|---|---|---|
| 9527551 | Triggered | Generic Database Assignment | f1aa344 | ops/dev-deploy.tmpl.yaml | View secret |
| 9527551 | Triggered | Generic Database Assignment | 4afc1df | ops/dev-deploy.tmpl.yaml | View secret |
🛠 Guidelines to remediate hardcoded secrets
- Understand the implications of revoking this secret by investigating where it is used in your code.
- Replace and store your secrets safely. Learn here the best practices.
- Revoke and rotate these secrets.
- If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
- following these best practices for managing and storing secrets including API keys and other credentials
- install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.
🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.
Our GitHub checks need improvements? Share your feedbacks!
laritakr
force-pushed
the
i718-separate-fast-lane-import-queue-and-worker
branch
from
92956e3
to
f1aa344
Compare
Attempting to merge/override the same ENV variable in different deployments ended up being very difficult. This is perhaps less elegant, but simpler
Attempting to use the --queues flag didn't work, but using the ENV variable does
jeremyf
suggested changes
Feb 22, 2024
jeremyf
reviewed
Feb 22, 2024
| def set_auxiliary_queue_priority | ||
| return unless queue_name.to_sym == :auxiliary | ||
|
|
||
| self.priority = ENV.fetch('AUXILIARY_QUEUE_PRIORITY', 100).to_i |
There was a problem hiding this comment.
This setting of priority will clobber the nuanced priority of a later PR.
bkiahstroud
commented
Feb 23, 2024
There was a problem hiding this comment.
@jeremyf here are the results of QA'ing locally with these changes combined with LaRita's. No all the Queue Name values are correct for SDAPI, but all the priorities do get adjusted as expected, which makes me think it will work as intended.
| Tenant | Job Class | Queue Name | Priority |
|---|---|---|---|
| SDAPI | AttachFilesToWorkJob | auxiliary | 109 |
| Dev | AttachFilesToWorkJob | ingest | -1 |
| SDAPI | BatchCreateJob | auxiliary | 110 |
| Dev | BatchCreateJob | ingest | 0 |
| SDAPI | Bulkrax::ImportWorkJob | auxiliary | 105 |
| Dev | Bulkrax::ImportWorkJob | ingest | -5 |
| SDAPI | Bulkrax::ImporterJob | auxiliary | 90 |
| Dev | Bulkrax::ImporterJob | ingest | -20 |
| SDAPI | Bulkrax::ScheduleRelationshipsJob | default | 160 |
| Dev | Bulkrax::ScheduleRelationshipsJob | default | 50 |
| SDAPI | CharacterizeJob | auxiliary | 140 |
| Dev | CharacterizeJob | ingest | 30 |
| SDAPI | ContentDepositEventJob | auxiliary | 60 |
| Dev | ContentDepositEventJob | ingest | -50 |
| SDAPI | CreateDerivativesJob | auxiliary | 150 |
| Dev | CreateDerivativesJob | ingest | 40 |
| SDAPI | CreateWorkJob | auxiliary | 110 |
| Dev | CreateWorkJob | ingest | 0 |
| SDAPI | FileSetAttachedEventJob | auxiliary | 110 |
| Dev | FileSetAttachedEventJob | ingest | 0 |
| SDAPI | Hyrax::GrantEditToMembersJob | auxiliary | 120 |
| Dev | Hyrax::GrantEditToMembersJob | ingest | 10 |
| SDAPI | IiifPrint::Jobs::CreateRelationshipsJob | default | 90 |
| Dev | IiifPrint::Jobs::CreateRelationshipsJob | default | -20 |
| SDAPI | IngestJob | auxiliary | 120 |
| Dev | IngestJob | ingest | 10 |
| SDAPI | InheritPermissionsJob | default | 110 |
| Dev | InheritPermissionsJob | default | 0 |
| SDAPI | StreamNotificationsJob | default | 110 |
| Dev | StreamNotificationsJob | default | 0 |
jeremyf
approved these changes
Feb 23, 2024
bkiahstroud
deleted the
i718-separate-fast-lane-import-queue-and-worker
branch
2 tasks
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Story
Refs:
Add a second worker that runs all GoodJob queues. The "base" worker will run all queues except the :auxiliary queue.
This, combined with the high priority assigned to all jobs in the :auxiliary queue, effectively builds a fast lane; all jobs run in the :auxiliary queue will be executed before all other jobs by the auxiliary worker.
The fact that the base worker does not run any jobs in the :auxiliary queue means we can have this fast lane without putting every other job on hold for, potentially, long periods of time.
Expected Behavior Before Changes
A single GoodJob process runs all background job queues (
*)Expected Behavior After Changes
Two GoodJob processes each run different queues:
:auxiliaryqueue (-auxiliary)*)Screenshots / Video
Notes
The reason we use
before_enqueuein ApplicationJob is because GoodJob does not support setting a default priority per queue.Newer versions of GoodJob do support the ability to run queues in the order they're listed, but this currently has problems.