-
Notifications
You must be signed in to change notification settings - Fork 1
/
README
104 lines (72 loc) · 5.17 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
Queue benchmark
This app was designed to benchmark Resque against Delayed::Job for the purpose of proving that we need to change out the queue system on an inventory project we have been working on.
I created two types of jobs. One as a simple baseline and the other to represent a facet of the tasks we queue up currently.
1) a simple AR find
2) fetch an image from an akamai server and then save and create a thumb with attachment_fu
Each of these tasks were created as jobs in Dj and Resque a varying number of times. To have a baseline we ran these tasks without a queue. We then ran several tests adding each task as a job 5,000 and 500,000 times into Dj. We did this with one worker and three workers to see the overhead of the queue as well as see how the performance improves with each worker. We then queued the same tasks in Resque 500,000 and ran with one worker and three workers respectively.
Results Summary:
We essentially confirmed our hypothesis that Dj doesn't scale when the queue becomes large. Synchronously fetching and saving images took took 91.7 seconds inline. Processing 1000 jobs of the that same task from a 500,000 job queue with one worker took 691.1 seconds in Dj, and 93.8 seconds in Resque. This means that Resque has very little overhead when compared to Dj which has huge database overhead. When we ran the same 1000 jobs with the same total number in the queue with 3 workers Dj took roughly 411 seconds and Resque took roughly 36 seconds. Thus Resque scales much better than Dj with more workers.
Method:
There are several rake tasks for seeding the database and the queues. After seeding the data I ran some benchmark tasks to time the execution. Take a look at benchmark.rake and seed.rake for examples.
Raw Results:
* 5,000 row image table straight synchronous processing:
basic AR find with no queue ( 1,000 queries )
user system total real
0.510000 0.060000 0.570000 ( 2.094698) [secs]
fetching an image off of akamai and doing an insert ( 1,000 inserts and image fetches )
user system total real
15.080000 2.900000 17.980000 ( 91.660208)
* 5,000 row image table 5,000 row jobs table dj queue single worker process:
Dj with just a basic AR find ( 1000 jobs )
user system total real
3.140000 0.290000 3.430000 ( 10.598753)
Dj with fetching an image off of akamai and doing an insert ( 1000 jobs )
user system total real
18.810000 3.090000 21.900000 ( 95.330724)
* 5,000 row image table 500,000 row jobs table dj queue single worker process:
Dj with just a basic AR find ( 1000 jobs )
user system total real
3.710000 0.720000 4.430000 (473.825784) [secs]
Dj with fetching an image off of akamai and doing an insert ( 1000 jobs )
user system total real
19.610000 3.730000 23.340000 (691.071522) [secs]
* 5,000 row image table 5,000 row jobs table dj queue 3 worker processes:
Dj with just a basic AR find ( 1000 jobs / 333 per worker ) [This has a rather large standard deviation which is due to failed lock attempts]
user system total real
84180 => 0.290000 0.170000 0.460000 ( 1.257616)
84178 => 1.100000 0.260000 1.360000 ( 4.334029)
84179 => 1.390000 0.280000 1.670000 ( 5.335976)
Dj with fetching an image off of akamai and doing an insert ( 1000 jobs / 333 per worker )
user system total real
84307 => 7.440000 1.370000 8.810000 ( 41.680948)
84309 => 7.500000 1.360000 8.860000 ( 42.371396)
84308 => 7.480000 1.360000 8.840000 ( 42.777726)
* 5,000 row image table 500,000 row jobs table dj queue 3 worker processes:
Dj with just a basic AR find ( 1000 jobs )
user system total real
84970 => 1.940000 0.590000 2.530000 (333.338117)
84971 => 1.940000 0.590000 2.530000 (333.344340)
84969 => 1.990000 0.580000 2.570000 (334.971635)
Dj with fetching an image off of akamai and doing an insert ( 1000 jobs / 333 per worker )
user system total real
85809 => 7.780000 1.660000 9.440000 (409.771486)
85811 => 7.830000 1.630000 9.460000 (410.615919)
85810 => 7.840000 1.670000 9.510000 (411.344920)
* 5,000 row image table 500,000 row jobs table Resque queue single worker process:
Resque with just a basic AR find ( 1000 jobs )
user system total real
1.080000 0.200000 1.280000 ( 3.261001)
Resque with fetching an image off of akamai and doing an insert ( 1000 jobs )
user system total real
15.610000 3.150000 18.760000 ( 93.730797)
* 5,000 row image table 500,000 row jobs table Resque queue 3 worker processes:
Resque with just a basic AR find ( 1000 jobs / 333 per worker )
user system total real
0.480000 0.180000 0.660000 ( 2.446347)
0.480000 0.170000 0.650000 ( 2.542048)
0.490000 0.180000 0.670000 ( 2.551230)
Resque with fetching an image off of akamai and doing an insert ( 1000 jobs / 333 per worker )
user system total real
5.930000 1.230000 7.160000 ( 35.850032)
6.010000 1.230000 7.240000 ( 36.007538)
5.950000 1.240000 7.190000 ( 36.010807)