Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize Memory Usage for MergeGlobalStatus's TopN #45727

Closed
hawkingrei opened this issue Aug 1, 2023 · 0 comments · Fixed by #45718 or #45993
Closed

Optimize Memory Usage for MergeGlobalStatus's TopN #45727

hawkingrei opened this issue Aug 1, 2023 · 0 comments · Fixed by #45718 or #45993
Labels
affects-6.5 affects-7.1 report/customer Customers have encountered this bug. sig/planner SIG: Planner type/enhancement The issue or PR belongs to an enhancement.

Comments

@hawkingrei
Copy link
Member

Enhancement

During the MergeGlobalStatus process, the current method of waiting for all TopN values to be collected before merging them can lead to high memory consumption. This is especially problematic when dealing with large datasets or when the number of unique values is high.

The proposed enhancement involves modifying the mergeglobalstatus process to collect and process the TopN values concurrently. Rather than waiting for all TopN values to be collected, the process should start merging and processing the available TopN values as soon as they are collected. This approach would help optimize memory usage by reducing the need to hold all TopN values in memory before processing.

@hawkingrei hawkingrei added the type/enhancement The issue or PR belongs to an enhancement. label Aug 1, 2023
@winoros winoros added the sig/planner SIG: Planner label Aug 2, 2023
@seiya-annie seiya-annie added the report/customer Customers have encountered this bug. label Jun 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects-6.5 affects-7.1 report/customer Customers have encountered this bug. sig/planner SIG: Planner type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
4 participants