Hugepage support

[abrestic-rivos]

The TEE APIs have placeholder support for hugepages, but our current implementation rejects anything other than 4kB pages. There's a number of places internally (page table manipulation, page tracking) that will need to be updated to take hugepages into account, and we'll likely want an interface to promote/demote hugepage mappings in a TVM.

[stillson]

I am working on this.

[abrestic-rivos]

Re-opening this since it's far from done :)

@stillson is still working on an orthogonal piece to this, which is to map Salus itself with hugepages. That should speed up our boot times.

For hugepage usage in TVMs, I'd propose the following new TH-ABI calls (largely inspired from @rsahita's draft spec):

• tvm_block_range(guest_addr, len): Invalidates the pages in the range guest_addr to guest_addr + len, marking them as "blocked" at the current TLB version.
• tvm_unblock_range(guest_addr, len): Marks the invalidated and blocked pages in the range guest_addr to guest_addr + len as valid.
• tvm_promote_page(guest_addr, page_size): Promotes the 512 physically-contiguous mappings of size page_size - 1 at guest_addr to a single mapping of page_size. Both guest_addr and the mapped physical address must be aligned to page_size. The mappings must have been blocked at a TLB version older than the current TLB version.
• tvm_demote_page(guest_addr, page_size): Demotes the mapping of size page_size + 1 at guest_addr to 512 mappings of size page_size. The mapping must have been blocked at a TLB version older than the current TLB version.

The implementation of the existing tvm_add_zero_pages(), tvm_add_shared_pages(), etc APIs would be updated to respect the page_size argument.

Note that the host VM will continue to always be mapped with 4kB pages so as to avoid having to transparently break up hugepage mappings within Salus. This also is consistent with MTT-based deployment modes where the host can operate on physical memory at a 4kB granularity. We do however need to make sure the host VM is mapped in 2MB-aligned chunks so that we can exercise hugepages; we currently only guarantee 16kB.

The flow for promotion/demotion would look like this:

• tvm_block_range(addr, len)
• tvm_initiate_fence() + IPIs
• tvm_{promote,demote}_page(addr, page_size)

Some points for discussion:

• Should unblock be implicit in promote/demote, or should the host be required to make an explicit tvm_unblock_range() call? I expect implicit unblock is what we'd want in the common case, but in the case we're demoting to support a share/unshare operation, we do not want an implicit unblock.
• Should we allow batched promote/demote, rather than just one hugepage at a time? I don't think THP would be able to easily take advantage of this as its currently written, so I'm proposing just a single page for now.
• On a promote, should we allow the host to reclaim the freed page-table page, or should it go back into the TVM's page-table page pool?