Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking memory spill to external storage #8696

Closed
3 tasks done
Tracked by #7823
sundy-li opened this issue Nov 8, 2022 · 2 comments
Closed
3 tasks done
Tracked by #7823

Tracking memory spill to external storage #8696

sundy-li opened this issue Nov 8, 2022 · 2 comments
Assignees
Labels
C-feature Category: feature

Comments

@sundy-li
Copy link
Member

sundy-li commented Nov 8, 2022

Summary

Description for this feature.

Some queries with large memory allocation may cause OOM which is unstable.
After we have the memory tracker, we can spill extra memory into external storage(disk, s3). This will surely make the query run slower, but it ensures we can get the final results.

tasks to do

  • spill memory in group by query (HashTable)
  • spill memory in join query (HashTable)
  • spill memory in order by query (multi-way external merge sort)
@sundy-li sundy-li added the C-feature Category: feature label Nov 8, 2022
@zhang2014
Copy link
Member

external storage(disk, s3)

Maybe use cache(automatically select s3 or disk) is better. CC: @Xuanwo

@Xuanwo
Copy link
Member

Xuanwo commented Nov 8, 2022

We can spill to a temporary storages as described in https://databend.rs/doc/contributing/rfcs/cache

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-feature Category: feature
Projects
None yet
Development

No branches or pull requests

4 participants