Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fleet] Split asset installation into chunks to avoid high memory pressure #189043

Closed
Tracked by #187969
, #174168
...
xcrzx opened this issue Jul 24, 2024 · 4 comments · Fixed by #189045
Closed
Tracked by #187969
, #174168
...

[Fleet] Split asset installation into chunks to avoid high memory pressure #189043

xcrzx opened this issue Jul 24, 2024 · 4 comments · Fixed by #189045
Assignees
@xcrzx
Labels
8.16 candidate Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team:Fleet Team label for Observability Data Collection Fleet team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.

Comments

@xcrzx
Copy link
Contributor

xcrzx commented Jul 24, 2024

Related to: #187969

Summary

For packages with a large number of assets, when performing reinstall or upgrade, we need to limit the number of saved objects installed in a single request.

Currently, during the package installation, before and inside the savedObjectsImporter.import method, a substantial number of auxiliary objects proportional to the package size are being created. Not all of these created objects are used after the installation. We can reduce memory pressure by chunking the installation requests so that unused objects can be garbage collected, leading to less peak memory consumption.
@xcrzx xcrzx added Team:Fleet Team label for Observability Data Collection Fleet team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Detection Rule Management Security Detection Rule Management Team Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area 8.16 candidate labels Jul 24, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@elasticmachine
Copy link
Contributor

Pinging @elastic/fleet (Team:Fleet)

This was referenced Jul 24, 2024
Closed
Open
@xcrzx xcrzx self-assigned this Jul 24, 2024
@xcrzx xcrzx mentioned this issue Jul 24, 2024
Merged
xcrzx added a commit that referenced this issue Jul 25, 2024
@xcrzx
[Fleet] Chunk asset installation during package install (#189045)
b7e66eb 
**Resolves: #189043

## Summary

This PR limits the number of saved objects installed in a single
request. During saved object installation a lot of auxiliary objects are
created in memory. Chunking allows for the garbage collection of memory
objects that are not needed for response.

**Memory consumption before**

![Screenshot 2024-07-12 at 12 44
58](https://github.com/user-attachments/assets/2c7bb609-f107-46bd-bd98-43a0c58107e8)


**After**
![Screenshot 2024-07-18 at 11 53
30](https://github.com/user-attachments/assets/ff6529dd-033e-4c2b-9630-0a2ea3927c21)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@xcrzx xcrzx

Labels
8.16 candidate Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team:Fleet Team label for Observability Data Collection Fleet team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Fleet] Chunk asset installation during package install xcrzx/kibana
2 participants
@xcrzx @elasticmachine