IPC Messsage Passing

[maxtyson123]

"If you’re performing this IPC as part of a system call from userspace, the memory containing the original
message is unlikely to be mapped in the receiver’s address space anyway, so we have to copy it into the kernel’s
address space, which is mapped in both processes." PG 226

But it is my understanding that whilst in usermode (CPL = 3) the process cannot directly access memory in the higher half addresses (e.g., 0xFFFFFFFF80000000 and above) because those addresses are typically reserved for the kernel and mapped with page table permissions that prevent user-space access.

TODO:

• Discuss
• Find reference in Intel Manual
• PR with proposed changes

[DeanoBurrito]

Hey! I think there's a bit of confusion here:

• first of all higher half refers to the higher half (for a system with 4 level paging/48-bit virtual addressing that would be 0xFFFF'8000'0000'0000).
• On x86 there's only one feature that I'm aware of that enforces 'higher half == kernel' (the linear masking feature intel recently added), but having said that its good practice for the sake of cleanliness.
• While in userspace (CPL=3 in x86 speak), you're right in that you can only access memory that has the user bit set in every PTE used to translate the address.

As for the part of the notes you're referring to, this is all happening from the perspective of the kernel. Like you hinted at, it would be bad if userspace could write to kernel memory, which is why we have system calls. They're a fixed entry-point into the kernel that allow you to thoroughly inspect and sanitize any requests from userspace, like an attempt to do IPC. The kernel carries out the work requested, after doing it's checks.
The important part here is that the kernel is the one running this code (ipc_send() and friends), so it can access memory marked as supervisor-only. None of the code in this chapter (or really any chapters, besides the example userspace program) is intended to run in userspace.

So, in less words:

• [userspace] program wants to send IPC message, performs a system call
• [kernel] starts in syscall entry function, checks arguments
• [kernel] sees its an IPC syscall, calls ipc_send() - performs the copy in into the bounce buffer.
• [kernel] ipc_send() is done, returns to syscall entry function, which also does a sysret/sysexit/iret (whichever mechanism you're using)
• [userspace] user program regains control, carries on.

[maxtyson123]

I think we are talking about two different parts, my apologies for not being clear enough in the initial message.

What I'm trying to understand is how:

void* msg_copy = malloc(length);
memcpy(msg_copy, buffer, length);

Is then able to be read from the userspace program because once the int is handled it should return to userspace. Now that buffer should be pointing to kernel memory we can't access? Or would the example from 36.4 Receiving somehow have the endpoint information mapped into its address space. I have further questions about this but I'll save the tanget until I have more information to base those off.

So within send function, we would need to make sure that there is a switch to use the endpoint's malloc so it can then see the copied buffer?

[DeanoBurrito]

All good, maybe the notes could be clearer on this too.

That code you're referencing is an example for how ipc_send() might look, which would run from inside the system call (so still in the kernel). If you're using a software interrupt for your system call, this all runs in the interrupt handler.

What happens here is during the system call the kernel makes a copy of the message in msg_copy. The original data in buffer no longer matters, and the user thread can do whatever it wants with that after the system call returns. The kernel has its own copy of the message content in msg_copy, which continues to exist after the system call finishes, andis then used to fill the receivers buffer.

There ends up being 3 total copies of the message: one in senders address space, which the kernel copies into a buffer in the higher half, and then one in the receivers address space. The kernel copy is necessary because the time when the sender and receiver are ready may not overlap. Also the higher half should be the same in all address spaces, so the kernel can smuggle things between two different address spaces.

[maxtyson123]

I think the part I am confused with is the third buffer, this is how I currently see it:

1. Proc A (sender) makes a buffer of data and then calls the syscall to send
2. Kernel copies Proc A's buffer into it's own buffer and then updates the endpoint's message address to the new buffer
3. Proc B (endpoint) copies the message info (kernel buffer) into a new buffer so that it can do xyz with that information and any new messages can be sent whilst it is doing xyz without having to wait

My issue is how does Proc B get the copy of the message buffer from the kernel? "36.4 Receiving" does not specify how ipc_endpoint* endpoint; is obtained.

[DeanoBurrito]

Minor thing with point 3: the kernel copies from its own buffer into Proc B's buffer (since the proc wouldnt have access to kernel memory).
The kernel does this copy in response to a syscall from Proc B, what this looks like again depends on how you want to expose this to userspace. If you were going with a posix-flavoured experience it might be a read(), and you would have a file descriptor representing the ipc endpoint.

The kernel would need a way to associate the system call from Proc B with a particular endpoint. So this would depend a bit on the design of the rest of your system. You could have calls for creating endpoints within a process, and return handles/file descriptors that can be passed to further syscalls (like ipc_receive()). The kernel can then use that info to lookup the endpoint info. If you want something simpler you could allow a single endpoint per process and store it in your struct process_t. These options lock the endpoint to whichever process created it, if you want the ability to move them between processes, maybe look at leveraging the VFS (this is a common unix thing).

[maxtyson123]

Thank you for you detailed replys, I understand it much better now. Would you like me to try clear that that up and add it In a PR so future users can understand better?

[DeanoBurrito]

All good, glad I could help.

Yeah I think there's a few dots that are missing. Honestly I think should be covered by a section in the userspace chapter, which covers 'big picture' resource management. How the kernel and userspace make resources available to each other, and how the kernel refers to them. It could be applied more specifically to the IPC endpoint example.

If you wanted to write that I'd be happy to merge it, but I would want to get @dreamos82 thoughts on it first. Otherwise I can take a look at adding that section myself, but I'm quite time poor these days (haha see my elf loader rewrite PR), so it might take a bit.

[dreamos82]

I think that if there are some improvements for the current implementation they could be useful to be added (especially clarifications), so yeah if you have something in mind better to create a PR (i see you already did) with the proposed changes and we can discuss them.

But i agree with @DeanoBurrito and we should create a new chapter in the userspace section about how to make resources available in userspace too, we were talking about it few weeks ago (and is a part that i need to understand better too :D)

So the future plans are to expand the explanation with a real world example. But so far i welcome improveements on what we currently have, given our time constraints i don't think we are going to write this new paragraph very soon.

Of course ifyou have the knowledge @maxtyson123 and want to draft something you are more than welcome to do it!

Meanwhile thanks for the help! :)