Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(tianmu): Avoid OOM when data too large and aggregated (#949) #1355

Merged
merged 2 commits into from
Mar 10, 2023
Merged

fix(tianmu): Avoid OOM when data too large and aggregated (#949) #1355

merged 2 commits into from
Mar 10, 2023

Conversation

adofsauron
Copy link
Collaborator

@adofsauron adofsauron commented Mar 7, 2023

1. Use memory mapping
2. Set the maximum number of concurrent threads for aggregation

Summary about this PR

Issue Number: close #949 #1291

Tests Check List

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Changelog

  • New Feature
  • Bug Fix
  • Performance Improvement
  • Build/Testing/CI/CD
  • Documentation
  • Not for changelog (changelog entry is not required)

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features

@mergify
Copy link
Contributor

mergify bot commented Mar 7, 2023

Thanks for the contribution!
I have applied any labels matching special text in your PR Changelog.

Please review the labels and make any necessary changes.

@mergify mergify bot added the PR-bug bug for pull request label Mar 7, 2023
@codecov
Copy link

codecov bot commented Mar 8, 2023

Codecov Report

Patch coverage: 69.79% and project coverage change: +0.04 🎉

Comparison is base (106431a) 42.89% compared to head (db55498) 42.93%.

❗ Current head db55498 differs from pull request most recent head c2c0676. Consider uploading reports for the commit c2c0676 to get more accurate results

Additional details and impacted files
@@                 Coverage Diff                 @@
##           stonedb-5.7-dev    #1355      +/-   ##
===================================================
+ Coverage            42.89%   42.93%   +0.04%     
===================================================
  Files                 1836     1838       +2     
  Lines               397300   397390      +90     
===================================================
+ Hits                170411   170621     +210     
+ Misses              226889   226769     -120     
Impacted Files Coverage Δ
storage/tianmu/core/aggregation_algorithm.h 60.00% <ø> (ø)
storage/tianmu/core/engine.h 30.43% <ø> (ø)
storage/tianmu/handler/ha_tianmu.cpp 54.24% <0.00%> (+4.29%) ⬆️
storage/tianmu/mm/memory_handling_policy.cpp 47.56% <0.00%> (+0.23%) ⬆️
storage/tianmu/system/configuration.cpp 100.00% <ø> (ø)
storage/tianmu/system/configuration.h 0.00% <ø> (ø)
storage/tianmu/core/aggregation_algorithm.cpp 55.95% <30.00%> (-0.03%) ⬇️
storage/tianmu/mm/huge_heap_policy.cpp 41.66% <44.44%> (+1.66%) ⬆️
storage/tianmu/core/engine.cpp 51.31% <81.81%> (+2.71%) ⬆️
storage/tianmu/mm/memory_statistics.h 86.79% <86.79%> (ø)
... and 1 more

... and 27 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

storage/tianmu/mm/huge_heap_policy.cpp Show resolved Hide resolved
storage/tianmu/core/aggregation_algorithm.cpp Outdated Show resolved Hide resolved
storage/tianmu/core/aggregation_algorithm.cpp Outdated Show resolved Hide resolved
storage/tianmu/handler/ha_tianmu.cpp Show resolved Hide resolved
storage/tianmu/handler/ha_tianmu.cpp Show resolved Hide resolved
storage/tianmu/mm/memory_statistics.h Outdated Show resolved Hide resolved
@adofsauron adofsauron requested a review from wisehead March 10, 2023 08:53
    1. Use memory mapping
    2. Set the maximum number of concurrent threads for aggregation
@mergify mergify bot merged commit f5cd1f9 into stoneatom:stonedb-5.7-dev Mar 10, 2023
@Nliver Nliver added this to the StoneDB_5.7_v1.0.3 milestone Apr 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
PR-bug bug for pull request
Projects
Development

Successfully merging this pull request may close these issues.

feature: Cache disk files for intermediate results of aggregate operations to avoid OOM
4 participants