Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[LibOS] When memory pages or access permissions in VMA bookkeeping are not changed, get rid of unnecessary calls to PAL memory ABI or EDMM operations #1708

Open
llly opened this issue Jan 9, 2024 · 6 comments · May be fixed by #1824
Open

[LibOS] When memory pages or access permissions in VMA bookkeeping are not changed, get rid of unnecessary calls to PAL memory ABI or EDMM operations #1708

llly opened this issue Jan 9, 2024 · 6 comments · May be fixed by #1824
Labels
enhancement New feature or request P: 2

Comments

@llly
Copy link
Contributor

llly commented Jan 9, 2024

Description of the feature

In current Gramine LibOS VMA bookkeeping, address begin, address end, prot, flags and file with offset are recorded.

However, when VMAs are created or changed successfully, only address is returned to caller. Then a corresponding PAL memory function is always called even pages of host already meet the expected status. In SGX PAL, extra Ocalls are called, When EDMM enabled, extra EDMM functions are called to set EPCM permissions.

My proposal is adding a flag to indicate what kind of page flags and permissions changes (change page flags, change file mapping, initialize permissions, restrict permissions, extend permissions, maybe more), returned by VMA bookkeeping functions and passed to PAL memory functions.
When all page flags and permissions are unchanged, LibOS doesn't call PAL memory function.
When pages flags and/or permissions are changed. LibOS call PAL memory function with this flag. PAL memory function can call mmap/mprotect/EMODPE/EMODPR accordingly.

It requires that LibOS VMA bookkeeping records exact same memory pages and access permissions as host PAL.
Need to identify and fix all violations, for example,

gramine/libos/src/bookkeep/libos_thread.c

Line 100 in 1f72aaf

ret = PalVirtualMemoryProtect(addr, PAGE_SIZE, /*prot=*/0);

only change permissions for PAL.

Why Gramine should implement it?

Performance improved. We've seen a workload that calls mmap(RW) to allocate some pages then calls mprotect(RW) on each page.
And for SGX EDMM. PTE permissions and the EPCM permissions can sync.
@dimakuv
Copy link

dimakuv commented Jan 9, 2024

@llly Good observation and good idea, thanks.

@kailun-qin implemented a new LibOS upcall get_vma_info(), which PALs can call to learn current VMA information (page protections, and I think currently that's all). See #1513 and the discussion here #1706

I think this new upcall will be enough to avoid OCALLs/EDMM calls. PALs will call this upcall, learn the current VMA info, compare with the requested info, and if the same, just exit the func.

@kailun-qin @mkow What do you think? Sounds like we have another usage candidate for get_vma_info() from #1513, so I am now highly in favor of implementing this upcall.

@llly Would this upcall be enough for your purposes?

@kailun-qin
Copy link
Contributor

Sounds like we have another usage candidate for get_vma_info() from #1513, so I am now highly in favor of implementing this upcall.

I recall that we might also have another candidate: #1527 (comment).

@llly
Copy link
Contributor Author

llly commented Jan 9, 2024

@dimakuv It's different. get_vma_info() is upcall from PAL to LibOS for PalExceptionHandler, only handle one page each time. I propose call from LibOS to PAL. LibOS can handle one VMA each time.

@kailun-qin It's similar, but focus on EDMM. I'd like LibOS VMA bookkeeping to handle address, flag and prot of LibOS memory. Currently it only handles address and passthrough flag and prot to PAL.

@dimakuv
Copy link

dimakuv commented Jan 9, 2024

@llly Hm, I may be missing something. Now that I look at your example:

We've seen a workload that calls mmap(RW) to allocate some pages then calls mprotect(RW) on each page.

Why can't the LibOS consult itself during mprotect() that this mprotect is basically a no-op? LibOS itself knows everything there is need to know about the memory ranges, so why any additional code in PAL?

@llly
Copy link
Contributor Author

llly commented Jan 9, 2024

Yes. That's what I means. I don't need an upcall. LibOS can handle prot in VMA bookkeeping.

@dimakuv
Copy link

dimakuv commented Jan 9, 2024

Thank you @llly, then I misunderstood your initial request. Yes, now it totally makes sense to me.

@llly llly linked a pull request Mar 21, 2024 that will close this issue
Open
@dimakuv dimakuv added enhancement New feature or request P: 2 labels Sep 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request P: 2
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[LibOS] VMA bookkeep calls Pal mprotect per VMA llly/gramine
3 participants
@dimakuv @llly @kailun-qin