-
Notifications
You must be signed in to change notification settings - Fork 0
/
.env
91 lines (59 loc) · 2.44 KB
/
.env
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
SERVER_PORT=8080
WORKER_COUNT=500
SERVER_ENDPOINT="http://127.0.0.1:8080"
NEXT_PUBLIC_SERVER_ENDPOINT="http://127.0.0.1:8080"
DB_HOST=127.0.0.1
DB_PORT=5432
DB_DATABASE=tasks
DB_USERNAME=admin
DB_PASSWORD="admin"
DB_SSL_MODE=disable
DB_POOL_MAX_CONNS=50
TASK_TIME_OUT=3
# OAUTH2_ISSUER=https://dev-736553.okta.com
OAUTH2_CLIENT_ID=64660401062-s9nm4vp7esak8g9a6im8c9712jkk2lbb.apps.googleusercontent.com
OAUTH2_CLIENT_SECRET=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
OAUTH2_PROVIDER=google
# // Tasks 1M (run_query)
# // Worker Count per deployment = 10000
# // Worker Deployment = 1M/10000 = 100
# // Timeout = 4s
# // Retry Count = 5
# // Delay between each retry = 2s
# // Max Time = 12s
# // Min Time = 4s
# // Failed Task = 20% (Cascade effect) Retry Required
# Initial run:
# Successful tasks (800,000): 800,000 * 4s = 3,200,000s
# Failed tasks (200,000): 200,000 * 1s = 200,000s
# Total time for initial run: 3,400,000s
# Retry 1:
# Total Retry Task Failed 1 = 200,000 * 20% = 40,000
# Total Retry Task Success 1 = 200,000 - 40,000 = 160,000
# Time: (160,000 * 4s) + (40,000 * (4s + 2s)) = 880,000s
# Retry 2:
# Total Retry Task Failed 2 = 40,000 * 20% = 8,000
# Total Retry Task Success 2 = 40,000 - 8,000 = 32,000
# Time: (32,000 * 4s) + (8,000 * (4s + 2s + 2s)) =
# Retry 3:
# Total Retry Task Failed 3 = 8,000 * 20% = 1,600
# Total Retry Task Success 3 = 8,000 - 1,600 = 6,400
# Time: (6,400 * 4s) + (1,600 * (4s + 2s + 2s + 2s)) = 41,600s
# Retry 4:
# Total Retry Task Failed 4 = 1,600 * 20% = 320
# Total Retry Task Success 4 = 1,600 - 320 = 1,280
# Time: (1,280 * 4s) + (320 * (4s + 2s + 2s + 2s + 2s)) = 8,960s
# Retry 5:
# Total Retry Task Failed 5 = 320 * 20% = 64
# Total Retry Task Success 5 = 320 - 64 = 256
# Time: (256 * 4s) + (64 * (4s + 2s + 2s + 2s + 2s + 2s)) = 1,920s
# Total Task Failed = 64
# Total Time = 3,400,000s + 880,000s + 192,000s + 41,600s + 8,960s + 1,920s = 4,424,480s
# Total Time in hours: 4,424,480s / 3600 ≈ 1,230.13 hours
# Total Time in days: 1,230.13 hours / 24 ≈ 51.25 days
# // TODO(Reconcile Improvment):
# // To Scale up run the system with 1M Dummy Task and Get insights of the system behavior
# // 1. Track the retry number in history table along with task table
# // 2. Reconcile based on history of the task rather than task table
# // 3. Reconcile based on the time difference of state rather than fixed time interval
# // 4. Check the task latest state and find the time difference and reconcile based on that