Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: regulation worker avoid panic in case of timeout #2657

Merged
merged 5 commits into from
Nov 8, 2022

Conversation

saurav-malani
Copy link
Contributor

@saurav-malani saurav-malani commented Nov 7, 2022

Description

  • Modified regulation-worker to avoid panic if regulation-manager API call times out. Instead we are simply retrying after a minute & pushing the stats for it.
  • Added stats to get metrics like number of users deleted till now, deletion time, total loop time.
  • Called from main with bugsnag to get notified in case of crash.

Notion Ticket

https://www.notion.so/rudderstacks/regulation-manager-worker-fail-safe-with-metrics-alert-a6c1eee073924651a15e9caad45750d7

@atzoum atzoum force-pushed the chore.regulationWorkerFailSafe branch from 6b6a4dd to a7a9796 Compare November 7, 2022 11:07
@codecov
Copy link

codecov bot commented Nov 7, 2022

Codecov Report

Base: 45.60% // Head: 45.46% // Decreases project coverage by -0.13% ⚠️

Coverage data is based on head (4e769b7) compared to base (0d061ff).
Patch coverage: 55.26% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2657      +/-   ##
==========================================
- Coverage   45.60%   45.46%   -0.14%     
==========================================
  Files         287      289       +2     
  Lines       47790    47937     +147     
==========================================
+ Hits        21793    21794       +1     
- Misses      24619    24760     +141     
- Partials     1378     1383       +5     
Impacted Files Coverage Δ
regulation-worker/internal/service/looper.go 52.63% <9.09%> (-17.74%) ⬇️
regulation-worker/internal/delete/api/api.go 77.92% <25.00%> (+0.91%) ⬆️
regulation-worker/internal/client/client.go 66.42% <50.00%> (-0.97%) ⬇️
regulation-worker/cmd/main.go 60.34% <100.00%> (+1.41%) ⬆️
regulation-worker/internal/service/service.go 73.07% <100.00%> (+2.24%) ⬆️
processor/stash/stash.go 41.56% <0.00%> (-24.28%) ⬇️
jobsdb/unionQueryLegacy.go 70.49% <0.00%> (-4.92%) ⬇️
config/backend-config/namespace_config.go 70.83% <0.00%> (-3.13%) ⬇️
services/rsources/handler.go 70.83% <0.00%> (-1.39%) ⬇️
... and 15 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@atzoum
Copy link
Contributor

atzoum commented Nov 8, 2022

@saurav-malani @chandumlg do we want this to be included in 1.3.0?

@saurav-malani
Copy link
Contributor Author

@saurav-malani
Copy link
Contributor Author

@atzoum if everything looks good, then it would be great to include it in release, otherwise we will have the use the image of the PR in regulation-worker deployment.

@saurav-malani saurav-malani merged commit 073a035 into master Nov 8, 2022
@saurav-malani saurav-malani deleted the chore.regulationWorkerFailSafe branch November 8, 2022 15:03
This was referenced Nov 8, 2022
atzoum added a commit that referenced this pull request Dec 1, 2022
feat(warehouse): added support for bigquery custom partition for workspaceIDs (#2679)
chore: by default enable max concurrent gw request limit. (#2648)
doc: create SECURITY.md (#2656)
chore: use bugsnag in router and processor goroutines (#2686)
test: using arm64 compatible images if necessary (#2670)
chore: regulation worker avoid panic in case of timeout (#2657)
feat(warehouse): degraded workspace id (#2627)
Release-As: 1.4.0
atzoum added a commit that referenced this pull request Dec 1, 2022
feat(warehouse): added support for bigquery custom partition for workspaceIDs (#2679)
chore: by default enable max concurrent gw request limit. (#2648)
doc: create SECURITY.md (#2656)
chore: use bugsnag in router and processor goroutines (#2686)
test: using arm64 compatible images if necessary (#2670)
chore: regulation worker avoid panic in case of timeout (#2657)
feat(warehouse): degraded workspace id (#2627)
Release-As: 1.4.0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants