[QOLSVC-4633] allow configuration of multiple target queues #98
Conversation
ThrawnCA
commented
Jun 14, 2024
ThrawnCA
commented
- Select a queue by hashing the package ID, so jobs for the same package will always go to the same queue to minimise deadlock risks.
- Also add the queue name to the job metadata so retries can remain on the same queue.
- Select a queue by hashing the package ID, so jobs for the same package will always go to the same queue to minimise deadlock risks. - Also add the queue name to the job metadata so retries can remain on the same queue.
Then there is still a possibility that they will clash. Can't really be helped. I suppose it would be possible to configure the two boxes to use different sets of queues, if you really wanted to go that far. |
|
Lets take it back a level, we want to follow the KISS principal. Since we are getting into lock issues and xloader and co already take a lock on the dataset (not just the resource). Should we not look at a way to flag that the dataset is being worked on before a worker starts processing and checks the job to the back of the queue to go through the other resources quicker. Also the issue we are running into may not be based on xloader only but also the validation scanner and other job queued products + the main thread altering the resource/s if its a upload or a re-ordering. |
That would likely be much more complex to implement in a robust fashion, actually.
Being able to customise the queue name from "default" to something else still has value. |
| @@ -128,6 +128,13 @@ groups: | |||
| to True. | |||
| type: bool | |||
| required: false | |||
| - key: ckanext.xloader.queue_names | |||
There was a problem hiding this comment.
Add to the readme and lets get it included.
… into QOLSVC-4633-queue-performance
