Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

disk storage: performance improvements related to bundles #4539

Closed
srenatus opened this issue Apr 4, 2022 · 0 comments · Fixed by #4661
Closed

disk storage: performance improvements related to bundles #4539

srenatus opened this issue Apr 4, 2022 · 0 comments · Fixed by #4661
Labels
investigating Issues being actively investigated performance

Comments

@srenatus
Copy link
Contributor

srenatus commented Apr 4, 2022

When using the disk storage backend (released in 0.39.0), there are limitations related to bundles, as noted in the docs:

Bundles are loaded into memory entirely even when disk storage is used: the decompressed, parsed bundle content is then inserted into the disk store.

How can this be overcome? Are delta bundles the answer? Can we alter the bundles processing code in such a way that we stream snapshot bundles' contents into the disk store?

Follow-up to #4014.
@srenatus srenatus added investigating Issues being actively investigated performance labels Apr 4, 2022
@ashutosh-narkar ashutosh-narkar mentioned this issue May 14, 2022
Merged
ashutosh-narkar added a commit to ashutosh-narkar/opa that referenced this issue May 24, 2022
@ashutosh-narkar
Initial support for large bundle deployments
419578a 
Fixes: open-policy-agent#4539

Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
ashutosh-narkar added a commit to ashutosh-narkar/opa that referenced this issue Jun 25, 2022
@ashutosh-narkar
Initial support for large bundle deployments
4bbe9ec 
Currently bundles are loaded into memory entirely
even when disk storage is used. Then the parsed content
is written to the store. Deserializing data into Go structs
is memory consuming and even if user has configured disk
storage, OPA is still bound by the amount of memory
assigned to it. This change adds a new lazy loading mode
wherein the entire data is not deserialized while bundle
reading and hence if the bundle contains large data files
and the user has enabled disk storage, OPA should be
able to handle this scenario w/o running OOM.

Fixes: open-policy-agent#4539

Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
ashutosh-narkar added a commit that referenced this issue Jun 27, 2022
@ashutosh-narkar
Initial support for large bundle deployments
f60dfaf 
Currently bundles are loaded into memory entirely
even when disk storage is used. Then the parsed content
is written to the store. Deserializing data into Go structs
is memory consuming and even if user has configured disk
storage, OPA is still bound by the amount of memory
assigned to it. This change adds a new lazy loading mode
wherein the entire data is not deserialized while bundle
reading and hence if the bundle contains large data files
and the user has enabled disk storage, OPA should be
able to handle this scenario w/o running OOM.

Fixes: #4539

Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
investigating Issues being actively investigated performance
Projects
Open Policy Agent
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Initial support for large bundle deployments ashutosh-narkar/opa
1 participant
@srenatus