Skip to content

Validate the memory consumption in SortPreservingMergeStream #16909

@2010YOUY01

Description

@2010YOUY01

Is your feature request related to a problem or challenge?

This is a follow-up to: #15700
Part of #15271

To add a more comprehensive solution to external sort, #15700 uses an optimistic approach:

  1. It first uses MultiLevelMergeBuilder to do possible re-spills, and ensure the later SortPreservingMergeStream won't use memory larger than the budget
  2. Construct a SortPreservingMergeStream from that builder, and this SPM stream bypasses the memory pool (don't validate the actual memory usage)

The existing memory tracking in DF is always validate in the memory-consuming operator itself, to prevent the inaccurate memory estimation.

Here is an idea (also the original discussion) to also enforce validation in this case: #15700 (comment)

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions