-
Notifications
You must be signed in to change notification settings - Fork 1.7k
Closed
Labels
enhancementNew feature or requestNew feature or request
Description
Is your feature request related to a problem or challenge?
This is a follow-up to: #15700
Part of #15271
To add a more comprehensive solution to external sort, #15700 uses an optimistic approach:
- It first uses
MultiLevelMergeBuilder
to do possible re-spills, and ensure the laterSortPreservingMergeStream
won't use memory larger than the budget - Construct a
SortPreservingMergeStream
from that builder, and this SPM stream bypasses the memory pool (don't validate the actual memory usage)
The existing memory tracking in DF is always validate in the memory-consuming operator itself, to prevent the inaccurate memory estimation.
Here is an idea (also the original discussion) to also enforce validation in this case: #15700 (comment)
Describe the solution you'd like
No response
Describe alternatives you've considered
No response
Additional context
No response
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request