-
Notifications
You must be signed in to change notification settings - Fork 556
fix(dedupe): Make DedupeIntegration
more memory efficient.
#4446
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
fix(dedupe): Make DedupeIntegration
more memory efficient.
#4446
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #4446 +/- ##
==========================================
- Coverage 84.60% 84.58% -0.02%
==========================================
Files 158 158
Lines 16463 16463
Branches 2850 2850
==========================================
- Hits 13928 13926 -2
- Misses 1694 1696 +2
Partials 841 841
|
DedupeIntegration
more memory efficient.
@antonpirker can't you just |
i thought about that too. but garbage collection moves objects around. so id(exc) could be different when we safe it and when we compare it... |
The |
DedupeIntegration
more memory efficient.DedupeIntegration
more memory efficient.
Users reported that the
DedupeIntegration
can use up a lot of memory, because it keeps a full exception in memory for checking if it has seen this exception already.Depending on the users code those exception objects can be big because they also include the traceback and local variables (which can be huge).
Idea is now to not save the whole exception but just a hash of the important parts of the exception to decide if we have seen this exception again.
fixes #3165
fixes #4327