Skip to content

Track memory usage in Non Limited Operators #1569

Closed
@alamb

Description

@alamb

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Ensure that physical ExecutionPlan nodes properly report their memory usage so DataFusion's can properly stay under its memory budget

Describe the solution you'd like

  1. Figure out a pattern to make ExecutionPlans be Tracking memory consumers (api in Initial MemoryManager and DiskManager APIs for query execution + External Sort implementation #1526)
  2. Apply that pattern to all operators built into DataFusion

Describe alternatives you've considered
N/A

Context
This is follow on work from the great PR from @yjshen in #1526 and part of the story of limiting memory used by DataFusion #587

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions