Add support for Windows virtual memory functions (VirtualAlloc, ...)

[yanchith]

Updated description

Virtual memory is something that high performance applications tend to do. I think it would be great if they had the ability to run their memory allocation test suites under miri.

#3605 already tracks changes to mmap such that we can do virtual memory things with it under miri.

On Windows, the API to deal with virtual memory is VirtualAlloc and VirtualFree:

VirtualAlloc(0, size, MEM_RESERVE, PAGE_READWRITE);  // reserve
VirtualAlloc(ptr, size, MEM_COMMIT, PAGE_READWRITE); // commit
VirtualFree(ptr, 0, MEM_RELEASE);                    // release

The minimal set relevant flags is MEM_RESERVE and MEM_COMMIT for VirtualAlloc; and MEM_RELEASE on VirtualFree. There is also MEM_DECOMMIT, which could be useful too.

https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-virtualalloc
https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-virtualfree

TODO @yanchith Provide runnable example.

[RalfJung]

Please have a look around for mmap in https://github.com/rust-lang/unsafe-code-guidelines/ and on Zulip -- many uses of mmap are actually incorrect in Rust (and in all LLVM-based compilers in general), so some foundational work in the backend and language spec will be required before Miri can support these operations. Some basic uses of mmap are already supported. It highly depends on what exactly you are doing.

So, if there's a particular pattern that you think should work but currently does not work, please provide a self-contained example demonstrating that pattern.

[RalfJung]

Also, we already have #3605 to track "support more uses of mmap", so I think this is a duplicate. (Supporting the Windows functions here would be a separate issue, and should not be mixed with discussions about the Linux APIs -- it is quite likely that they are subtly different, so this needs careful consideration.)

[yanchith]

many uses of mmap are actually incorrect in Rust (and in all LLVM-based compilers in general)

Yeah, mmap is a beast, but this usecase is pretty limited compared to what it can do.

So, if there's a particular pattern that you think should work but currently does not work, please provide a self-contained example demonstrating that pattern.

Sure, I'll do that. Meanwhile, I can give you a short summary of what this is about.

I am sorry if you already know all this, but here goes:

This is true at least on Windows, but I'd assume it to be true (or at least similar) elsewhere too. VirtualAlloc is the lowest-level memory allocation API you can call. Unlike malloc, which serves allocations of any sizes and alignments, VirtualAlloc only deals with page-granularity allocations. Higher level memory allocators use VirtualAlloc to get the memory from the OS.

One huge difference is that unlike malloc/free, the memory can be in multiple states.

• We use VirtualAlloc with MEM_RESERVE to reserve a contiguous (in virtual address space) region of memory. This makes the OS promise us that it won't hand out the same memory to someone else, but we can't actually use the memory yet. Note that we can reserve unrealistic amounts of memory (1000GB), as reserving doesn't mean the OS has to physically back it up yet.

• Only after doing VirtualAlloc with MEM_COMMIT can we actually write to and read from the memory. We don't have to commit the reserved memory all at once, but rather we can commit individual ranges of pages.

The fact that we have a contiguous region of memory that we commit on demand is super useful for writing arenas, which some people prefer using to manage their application's memory (e.g. see RADDebugger's arena https://github.com/EpicGamesExt/raddebugger/blob/master/src/base/base_arena.c). I am also pretty sure arenas or something similar is in most serious garbage collectors. I also think SpiderMoneky uses it to get memory for WASM modules and let the OS page fault handle OOB accesses (https://spidermonkey.dev/blog/2025/01/15/is-memory64-actually-worth-using.html).

[RalfJung]

Some basic arena usecases already work with mmap, so maybe this is just a case of making the same things work with VirtualAlloc. We don't have anyone with Windows experience on the team so we entirely rely on external contributions for that.

[saethlin]

I believe this use case with VirtualAlloc is exactly what's described in the issue description of #3605. So I think there are two tasks here, making the memory system able to support this pattern, then also having VirtualAlloc support that matches our mmap support.

[yanchith]

Had a look at #3605, and the code example in the first comment (https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=d2ea1888ffe60d05399f151af49bbf48) is basically what I had in mind, yes. (Except it does some huge-page specific stuff which I don't have experience with).

So I think there are two tasks here, making the memory system able to support this pattern, then also having VirtualAlloc support that matches our mmap support.

Pretty much!

FWIW, VirtualAlloc is a little different in that it can't do the following thing mmap can:

With mmap(0, size, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) we do not just reserve, but also make the kernel commit on demand as the memory is accessed.

On Windows, one has to explicitly VirtualAlloc(ptr, size, MEM_COMMIT, PAGE_READWRITE); to commit a page range. The kernel won't do it behind the scenes.

EDIT: So, if #3605 already tracks mmap virtual memory, do we repurpose this issue to do the same for Windows?
EDIT2: Ah, Ralf just did it :)

[RalfJung]

Okay, great! I made this issue about VirtualAlloc then.

It would be very useful to have a little example that can eventually be used as a test case for when someone works on the implementation, also to demonstrate which flags it would be good to support. (I assume there are tons of flags and initially we'll only support a minimal subset, just to get something to work.)

[RalfJung]

EDIT: So, if #3605 already tracks mmap virtual memory, do we repurpose this issue to do the same for Windows?
EDIT2: Ah, Ralf just did it :)

If you could update the issue description to match, that would be great. :)

[yanchith]

I'll make that example! Might take a couple of days, but will post it back here once I have it.

[RalfJung]

No rush, we don't have a timeline for when someone will be interested in implementing this anyway.