Multi-tenant distribution

[douglara]

Hi community!

Im search approach to resolve a problem and I thought to share to get help and feedback

Exemple:

• 1 GoodJob Worker with 10 threads
• User 1 call action to enqueue 10.000 jobs on queue_1 and this jobs there done in 1 hours.
• 10 minutes later user 2 call action to enqueue 20.000 jobs on queue_2 and this jobs there done in 2 hours.

With GoodJob, has one possibility do this worker process jobs from user 1 with 10 threads and after user 2 enque your jobs split thereads for 5 jobs from user 1 and 5 jobs from user 2?

I have problems with long delay to process jobs from user 2 it start after user 1 jobs is clean.

Has anyone had an similares problem?
Thanks!

[bensheldon]

Hi @douglara! I just wanted to double-check:

• do you think you're seeing a bug? ("problems with long delay to process jobs from user 2 it start after user 1 jobs is clean")
• or a feature request? ("possibility do this worker process jobs from user 1 with 10 threads and after user 2 enque your jobs split thereads for 5 jobs from user 1 and 5 jobs from user 2")

If it's a bug report:

I have problems with long delay to process jobs from user 2 it start after user 1 jobs is clean.

I think this is related to max_cache. The default is that 10,000 jobs will be cached in the queue, and otherwise will rely on polling to pick up additional jobs. You can change this value, and the only impact is that your application will use slightly more memory (or reduce the polling interval):

# config/application.rb
config.good_job.max_cache = 50_000

If it's a feature request:

GoodJob will execute jobs in the order that they're enqueued after accounting for priority and schedule. This would explain why you're seeing queue_1 finish all of its jobs before starting jobs from queue_2.

GoodJob could have a feature to instead execute jobs in random order. I think that would address your need.

[douglara]

Hi @bensheldon

Thanks for your ideias, I was wondering any possibility to do this hehe but now I see a feature request

[sandstrom]

This is a somewhat common scenario for async queues.

I'm not sure it's worth the effort to build out GoodJob to support this, since it's quite a lot of work.

Common approaches are (that would work with GoodJob today) are:

• Keep track of jobs in DB, and use application logic for "fairness" (whatever that means to you), see Holistic blog post below.
• One named queue per tenant (doesn't scale with huge number of tenants).
• Rate limit queuing of jobs.
• Use two (or more) named queues, e.g. priority1 and priority2 etc. and keep count of active jobs per tenant in DB. When one tenant have less than X active jobs, use priority1 queue, with more than X jobs, use priority2 (could use more than two tiers if needed). That way, the first N jobs for a single tenant will always complete quickly, such that a single tenant cannot "block" regular use for others.
• Instead of focusing on "fairness", simply ramp up processing power (e.g. auto-scaling groups or similar) when there are many pending tasks (or by measuring throughput of the queue every 1 or 10 minutes). If you have a high enough throughput it doesn't matter in which order you are processing tasks, because they will be completed quickly anyway (which is what the users care about).

More background:

• https://www.holistics.io/blog/how-we-built-a-multi-tenant-job-queue-system-with-postgresql-ruby/
• https://medium.com/thron-tech/multi-tenancy-and-fairness-in-the-context-of-microservices-sharded-queues-e32ee89723fc

[douglara]

Hi @sandstrom
Thanks for your suggestion and nice blogposts, Im studying it

About approaches:

• Keep track of jobs in DB, and use application logic for "fairness" (whatever that means to you), see Holistic blog post below.

Maybe It this would be best way, I think to change method the get next job, randomize where by user identify, that would be enough
I see this method:

good_job/lib/good_job/job_performer.rb

Line 29 in db950b1

def next

good_job/lib/good_job/job.rb

Line 160 in db950b1

def self.perform_with_advisory_lock

Does create new scope/method that random jobs where by field good_job_concurrency_key or queue if using dynamic queues name for each user.
@bensheldon It makes sense?

One named queue per tenant (doesn't scale with huge number of tenants).
I tried this using dynamic queue names by user id but Good Job get job by order enqueued. This works if create one process by user excluding others queues.

Use two (or more) named queues, e.g. priority1 and priority2 etc. and keep count of active jobs per tenant in DB. When one tenant have less than X active jobs, use priority1 queue, with more than X jobs, use priority2 (could use more than two tiers if needed). That way, the first N jobs for a single tenant will always complete quickly, such that a single tenant cannot "block" regular use for others.

Great idea i liked her

[sandstrom]

I can't really follow your response (I don't understand what you're trying to say). But I hope my suggestions + the blog posts are helpful.

Basically, I would try to solve this as an "application concern", i.e. by using the GoodJob library as it is, and make amendments in your application code.

Also, an additional suggestion is to simply scale up processing (dynamically, based on load). So when there are many queued jobs, you spin up additional machines (auto scaling groups or similar). Basically, instead of focusing on "fairness", just bring more processing power online and focus on the processing itself. If you have a high enough throughput it doesn't matter in which order you are processing tasks.

[douglara]

Guys, sorry my English

I'm try a new scope that on get next job randomize queue and seems with low cost

Follow the code:
main...douglara:feature-random-queues-order

Do you see a high problem?

I'm testing with benchmark with 100 jobs and 5 queues

[bensheldon]

This is great work, I really appreciate you benchmarking it. I'm a little concerned with the additional queries inside of the scope; getting it all into a single query would be nice.

I want to suggest another option. There is currently a priority_ordered scope. I wonder if you could benchmark the difference between:

# this is the current ordering method
scope :priority_ordered, -> { order('priority DESC NULLS LAST') }
#... and this new one
scope :priority_order_randomized, -> { order('priority DESC NULLS LAST, RANDOM()') }

[douglara]

Hi @bensheldon

Thanks!
You are right, this method is incrementing two new queries

I'm testing with scope

scope :priority_order_randomized, -> { order('priority DESC NULLS LAST, RANDOM()') }

The cost now is minimum hehe but it can be a problem if you need FIFO order on queues
Anyway can be solved using priority

[douglara]

I was got it to encapsulated in one query 😃
douglara@0b10fbf

I will test in production at next week's, thanks for your help.
If you consider a nice feature for GoodJob, I'm happy to work

Benchmark 1

 ------------------- Benchmark runs all jobs ------------------- 


Warming up --------------------------------------
 default queue order     1.000  i/100ms
        random order     1.000  i/100ms
  random queue order     1.000  i/100ms
random queue order v2
                         1.000  i/100ms
Calculating -------------------------------------
 default queue order      0.184  (± 0.0%) i/s -      1.000  in   5.431260s
        random order      0.182  (± 0.0%) i/s -      1.000  in   5.484374s
  random queue order      0.139  (± 0.0%) i/s -      1.000  in   7.218408s
random queue order v2
                          0.171  (± 0.0%) i/s -      1.000  in   5.843002s

Comparison:
 default queue order:        0.2 i/s
        random order:        0.2 i/s - 1.01x  (± 0.00) slower
random queue order v2:        0.2 i/s - 1.08x  (± 0.00) slower
  random queue order:        0.1 i/s - 1.33x  (± 0.00) slower

       user     system      total        real
default queue order  2.839895   0.167844   3.007739 (  5.392240)
random order  2.833887   0.171266   3.005153 (  5.197657)
random queue order  3.907435   0.220234   4.127669 (  6.771039)
random queue order v2  2.920526   0.181284   3.101810 (  5.318244)


 ------------------- Benchmark runs until the first job of the last queue  ------------------- 


Warming up --------------------------------------
 default queue order     1.000  i/100ms
        random order     1.000  i/100ms
  random queue order     1.000  i/100ms
random queue order v2
                         1.000  i/100ms
Calculating -------------------------------------
 default queue order      0.223  (± 0.0%) i/s -      2.000  in   8.973058s
        random order      6.232  (±16.0%) i/s -     30.000  in   5.079735s
  random queue order      5.453  (±18.3%) i/s -     26.000  in   5.167433s
random queue order v2
                          6.315  (±31.7%) i/s -     30.000  in   5.121034s

Comparison:
random queue order v2:        6.3 i/s
        random order:        6.2 i/s - same-ish: difference falls within error
  random queue order:        5.5 i/s - same-ish: difference falls within error
 default queue order:        0.2 i/s - 28.32x  (± 0.00) slower

       user     system      total        real
default queue order  2.314872   0.111017   2.425889 (  4.484797)
random order  0.013566   0.000000   0.013566 (  0.021516)
random queue order  0.082576   0.000395   0.082971 (  0.157051)
random queue order  0.044316   0.006431   0.050747 (  0.105662)

Benchmark 2

 ------------------- Benchmark runs all jobs ------------------- 


Warming up --------------------------------------
 default queue order     1.000  i/100ms
        random order     1.000  i/100ms
  random queue order     1.000  i/100ms
random queue order v2
                         1.000  i/100ms
Calculating -------------------------------------
 default queue order      0.157  (± 0.0%) i/s -      1.000  in   6.374150s
        random order      0.149  (± 0.0%) i/s -      1.000  in   6.717841s
  random queue order      0.120  (± 0.0%) i/s -      1.000  in   8.310898s
random queue order v2
                          0.132  (± 0.0%) i/s -      1.000  in   7.572954s

Comparison:
 default queue order:        0.2 i/s
        random order:        0.1 i/s - 1.05x  (± 0.00) slower
random queue order v2:        0.1 i/s - 1.19x  (± 0.00) slower
  random queue order:        0.1 i/s - 1.30x  (± 0.00) slower

       user     system      total        real
default queue order  3.027592   0.216322   3.243914 (  5.971565)
random order  3.080842   0.157146   3.237988 (  5.800498)
random queue order  4.416566   0.289069   4.705635 (  7.993095)
random queue order v2  3.304799   0.221477   3.526276 (  7.139496)


 ------------------- Benchmark runs until the first job of the last queue  ------------------- 


Warming up --------------------------------------
 default queue order     1.000  i/100ms
        random order     1.000  i/100ms
  random queue order     1.000  i/100ms
random queue order v2
                         1.000  i/100ms
Calculating -------------------------------------
 default queue order      0.219  (± 0.0%) i/s -      2.000  in   9.131966s
        random order      6.264  (±31.9%) i/s -     29.000  in   5.172901s
  random queue order      5.724  (±17.5%) i/s -     28.000  in   5.151360s
random queue order v2
                          6.506  (±30.7%) i/s -     30.000  in   5.068973s

Comparison:
random queue order v2:        6.5 i/s
        random order:        6.3 i/s - same-ish: difference falls within error
  random queue order:        5.7 i/s - same-ish: difference falls within error
 default queue order:        0.2 i/s - 29.69x  (± 0.00) slower

       user     system      total        real
default queue order  2.316350   0.124364   2.440714 (  4.509597)
random order  0.005188   0.000000   0.005188 (  0.007894)
random queue order  0.010194   0.000000   0.010194 (  0.021899)
random queue order  0.082721   0.000000   0.082721 (  0.165828)