Multiple concurrency keys

[wnm]

We are migrating to Solid Queue and trying to find out how to support our use case:

We currently use sidekiq and a custom currency control that makes sure there is always 1 job running per customer, and only 1 job with the same arguments. We do this by having multiple concurrency keys, and I was hoping I could do something similiar with Solid Queue, eg:

class FirstJob < ApplicationJob
  # make sure only 1 job per customer across all jobs
  limits_concurrency to: 1, key: ->(customer) group: "CustomerJobs"
  # make sure only 1 job with the same arguments
  limits_concurrency to: 1, key: ->(customer, project, other_argument)

class SecondJob < ApplicationJob
  limits_concurrency to: 1, key: ->(customer) group: "CustomerJobs"
  limits_concurrency to: 1, key: ->(customer, mailbox, other_argument)

If I understood correctly, there is only one concurrency key per job in Solid Queue? Would I be able to do it in another way? Thanks in advance! ❤️

[rosa]

If I understood correctly, there is only one concurrency key per job in Solid Queue?

Yes, that's right, you can only specify a single concurrency key per job.

Would I be able to do it in another way?

🤔 It depends on your case, but would it work just to limit on the customer? For example, if you have FirstJob enqueued with customer ID 1, you can't have SecondJob enqueued with customer ID 1, but also you can't have another FirstJob enqueued with the same arguments because the customer would be included in those arguments.

[wnm]

If I understood correctly, there is only one concurrency key per job in Solid Queue?

Yes, that's right, you can only specify a single concurrency key per job.

Would I be able to do it in another way?

🤔 It depends on your case, but would it work just to limit on the customer? For example, if you have FirstJob enqueued with customer ID 1, you can't have SecondJob enqueued with customer ID 1, but also you can't have another FirstJob enqueued with the same arguments because the customer would be included in those arguments.

In terms of concurrency yes, that would be enough. But we are using one concurrency key to make sure identical jobs don't even get enqueued, just to not do unneccessary work, similar to the on_conflict: :discard you were thinking about in #176

So maybe my question isn't about multiple concurrency keys, but rather how do I make sure to not enqueue duplicate jobs. And maybe #105 will solve this!

[rosa]

Ahh got it. I think #176 might not quite work for your case, then, because it'd still be just with a single concurrency key; the only difference would be that you could choose what to do if more than one job is enqueued, but either by all arguments or by the customer. For Solid Queue, I think having more than one concurrency key per job adds too much overhead in the way this is implemented with semaphores (the best way we could come up with, considering our database constraints), so I don't see that being added in the near future, at least not in the way this works now 😞

[rosa]

About #105, yeah, that would work if we do something else, different to #176. I still haven't made up my mind on that one 😅 For example, in your case, if a job is scheduled in the future, do you allow other jobs to be enqueued with the same arguments before that other job is due? Or is it not allowed at all, even if the scheduled job is not due yet?

[nhorton]

@rosa - maybe it is worth designing in public on this.

For handling of your above situation, my implementation instinctively would be:
Have a uniqueness_key method on the job classes that you use to generate a key at enqueue that is put into a DB column, then have either explicit match only or, somewhat better, support for an Arel statement so people could do complex stuff like OR conditions on the string. The default could include all params including queue and perform_at timing or exclude those attributes - either is fine - as users could implement different versions as needed. This would also allow one to specify different queries to use on the key for handling of enqueue conflicts and dequeueing / performing conflicts.

In the future, one could easily wrap the above with some syntactic sugar to handle common cases.