Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(event-queue): improve memory usage during the load phase #744

Merged
merged 1 commit into from
Aug 3, 2023

Conversation

glevco
Copy link
Contributor

@glevco glevco commented Aug 2, 2023

Motivation

The previous code was consuming too much memory when loading events from an existing database.

Acceptance Criteria

  • Create new batch_iterator() utils function that converts an iterator of items into an iterator of batches.
  • Change EventManager.handle_load_phase_vertices() so it saves events to RocksDB in write batches of 10.000 events instead of a single batch with all events.

Checklist

  • If you are requesting a merge into master, confirm this code is production-ready and can be included in future releases as soon as it gets merged

@glevco glevco self-assigned this Aug 2, 2023
@glevco glevco added the bug Something isn't working label Aug 2, 2023
@glevco glevco force-pushed the fix/event-queue/improve-memory-usage branch from 22ef998 to 6ab0e74 Compare August 3, 2023 14:40
@glevco glevco marked this pull request as ready for review August 3, 2023 14:55
msbrogli
msbrogli previously approved these changes Aug 3, 2023
hathor/event/event_manager.py Show resolved Hide resolved
@codecov
Copy link

codecov bot commented Aug 3, 2023

Codecov Report

Merging #744 (2fb77fc) into master (c95834d) will decrease coverage by 0.05%.
Report is 2 commits behind head on master.
The diff coverage is 100.00%.

❗ Current head 2fb77fc differs from pull request most recent head a909ab2. Consider uploading reports for the commit a909ab2 to get more accurate results

@@            Coverage Diff             @@
##           master     #744      +/-   ##
==========================================
- Coverage   84.56%   84.51%   -0.05%     
==========================================
  Files         252      253       +1     
  Lines       21688    21705      +17     
  Branches     2937     2941       +4     
==========================================
+ Hits        18341    18345       +4     
- Misses       2709     2719      +10     
- Partials      638      641       +3     
Files Changed Coverage Δ
hathor/event/event_manager.py 92.00% <100.00%> (+0.33%) ⬆️
hathor/utils/iter.py 100.00% <100.00%> (ø)

... and 4 files with indirect coverage changes

@glevco glevco force-pushed the fix/event-queue/improve-memory-usage branch from 2fb77fc to a909ab2 Compare August 3, 2023 21:59
@msbrogli msbrogli merged commit a909ab2 into master Aug 3, 2023
@msbrogli msbrogli deleted the fix/event-queue/improve-memory-usage branch August 3, 2023 23:21
This was referenced Aug 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

3 participants