Skip to content

[FEATURE] Memory limitation when merging thousands of complex indices #3750

@xinyual

Description

@xinyual

Is your feature request related to a problem?
In #3653 we merge objects type when fetching schemas from several indices using index schema. But in the testing, when the cluster meets higher concurrency (e.g. 64) and complex scenario (e.g. 1000 indices with 15 depth), it will throw out of memory. We need to figure out what is the bottleneck and see how to resolve it.

What solution would you like?
Figure out what is the bottleneck of out of memory problem.

What alternatives have you considered?
A clear and concise description of any alternative solutions or features you've considered.

Do you have any additional context?
Add any other context or screenshots about the feature request here.

Metadata

Metadata

Assignees

Labels

PPLPiped processing languageenhancementNew feature or requestperformanceMake it fast!

Type

No type

Projects

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions