-
Notifications
You must be signed in to change notification settings - Fork 202
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Encrypted FS] Use pre-allocacted free list instead of calloc/free for file nodes #1714
Comments
I reviewed this proposal privately before, and I'm fine with it. I think it's a simple and efficient optimization. @kailun-qin Would it be possible to show some numbers on the RocksDB workload? Alternatively, can we write a micro-benchmark that shows the benefit of this optimization? I guess a single large Protected File that is constantly read from and written to is enough.
Yes, I think it should be a new manifest option. |
Since we also have #1712, and also #1576, I feel like we need to introduce a generic manifest syntax and rlimits:
See also https://linux.die.net/man/2/setrlimit. Then we can add rlimits in a generic fashion, and even allow applications to modify these rlimits at runtime. For this particular issue, we could have a new non-standard rlimit: |
We tested the effectiveness of our proposed approach by running the RocksDB bench tool created by @cloudnoize (thanks!). When the compaction happened, w/o the PoC optimization:
And w/ the PoC optimization:
|
Description of the feature
Instead of using calloc/free directly for the Encrypted FS file nodes (e.g., MHT nodes, data nodes), we can introduce a free-list cache for them (i.e., memory chunks exclusively reserved for encrypted-files usage), as a perf optimization.
PoC (for testing and illustration purposes ONLY):
master...kailun-qin:gramine:hack-file-node-free-list
TO BE DISCUSSED:
Should we expose the size of the free list as a new Gramine manifest option? For example, if the manifest e.g. sets it to 8GB, then these 8GB will be put aside as a memory pool for encrypted-files.
Why Gramine should implement it?
To resolve perf bottlenecks seen on some workloads (e.g., RocksDB).
Specifically, RocksDB has a design of leveled compaction. It operates on a key range and two levels, it opens the files that contains the ranges and iterate sequentially over the files and merges them into new sorted files where the stale keys are being omitted.
When this compaction process happened, RocksDB throughput was reduced by ~10x when running w/ Gramine's encrypted FS.
The ongoing optimization for encrypted FS (#1599, #1681) don't help much in this case. The perf analysis showed that the one of the dominant contributors was the memory allocation of file data nodes:
gramine/common/src/protected_files/protected_files.c
Line 620 in 5d38bb1
Gramine commit hash
1f72aaf
The text was updated successfully, but these errors were encountered: