windows: support custom completion port integrations

[piscisaureus]

The new "wepoll" strategy is great for sockets but it makes it impossible to use the thread's I/O completion port for other purposes.
I would suggest to add a low-level "plugin" system so that other crates can share the completion port for different purposes.

This could be used for supporting named pipes and files, which can use overlapped I/O but their APIs work differently; more exotic examples are ReadDirectoryChangesW (file change notifications), GetAddrInfoExW (async DNS name lookup), and Job Objects (notifications about child processes).

Job objects in particular are exotic in that windows will send messages to the completion port where lpOverlapped does not actually point to a struct OVERLAPPED, but rather contains an indication of the event that happened.

So I would propose to to "switch" based on the lpCompletionKey value. The most simple solution would be to store a function pointer in it which receives the raw completion information, e.g. fn handle_completion(&OVERLAPPED_ENTRY).

To illustrate what I mean, here's a crude prototype: piscisaureus@ac40cd7

Notice how the Selector is no longer knows anything about Wake events.
They work simply because the lpCompletionKey contains an appropriate function pointer

[Thomasdezeeuw]

Related #526.

[Thomasdezeeuw]

I think this is related to a SourceSocket idea as discussed in #880 (comment).

[Thomasdezeeuw]

Punting to v1.0.

[carllerche]

@piscisaureus I have a question re: this proposal.

My understanding is that, when an OVERLAPPED operation is submitted to the OS, the OS needs "ownership" of the pointer. Ownership is returned back to the caller when the operation is completed and the overlapped is received during a "poll".

This implies that the selector needs to know about "in-flight" operations. This needs to happen so that when the Poll handle is dropped, it knows to wait until "in-flight" operations complete.

It gets a bit more complicated with "read" like operations that may not complete... in those cases, Poll needs to know about all inflight operations, signal to them to start shutting down, then wait to receive the OVERLAPS before dropping.

Do you have thoughts on how to model this?

[dtacalau]

@Thomasdezeeuw please assign this issue to me, will start working on it.

[carllerche]

@dtacalau before writing code we should have a solid proposal in place. I think bert’s sketch is a good starting point but we want to figure out the answers to my previous comment as well as a nice rust API to expose.

[piscisaureus]

@carllerche

My understanding is that, when an OVERLAPPED operation is submitted to the OS, the OS needs "ownership" of the pointer. Ownership is returned back to the caller when the operation is completed and the overlapped is received during a "poll".
Do you have thoughts on how to model this?

Make sure the Overlapped is wrapped (either directly or indirectly) wrapped in an Arc.

When an async I/O operation is successfully started (and you know the kernel now implicitly owns the OVERLAPPED) you artificially increase the Arc's reference count by cloning and forgetting the Arc.

When the operation completes and you receive a OVERLAPPED* from GetQueuedCompletionStatus, do some pointer math if necessary to get back at the wrapper struct, and then transmute/cast that back to an Arc, thereby effectively reversing the std::mem::forget::() we did after starting the operation.

I think the most elegant way was in this (rejected) patch:

• 88407fb#diff-8e89b097d3e3119d177930239584d29eR162
• 88407fb#diff-8e89b097d3e3119d177930239584d29eR181-R189
• 88407fb#diff-8e89b097d3e3119d177930239584d29eR569

[Thomasdezeeuw]

@piscisaureus that patch, with modifications, made it into the code base:

mio/src/sys/windows/selector.rs

Lines 288 to 301 in 3ebce3e

	/// Converts the pointer to a `SockState` into a raw pointer.
	/// To revert see `from_overlapped`.
	fn into_overlapped(sock_state: Pin<Arc<Mutex<SockState>>>) -> PVOID {
	let overlapped_ptr: *const Mutex<SockState> =
	unsafe { Arc::into_raw(Pin::into_inner_unchecked(sock_state)) };
	overlapped_ptr as *mut _
	}
	/// Convert a raw overlapped pointer into a reference to `SockState`.
	/// Reverts `into_overlapped`.
	fn from_overlapped(ptr: *mut OVERLAPPED) -> Pin<Arc<Mutex<SockState>>> {
	let sock_ptr: *const Mutex<SockState> = ptr as *const _;
	unsafe { Pin::new_unchecked(Arc::from_raw(sock_ptr)) }
	}

. I think we're doing it correctly now, the way you've explained it.

[carllerche]

Instead of making an API to hook into IOCP, would it be plausible to provide bindings to all the IOCP compatible types? The various types could be guarded w/ feature flags.

[Thomasdezeeuw]

Not blocking a v1.0 release.

[Thomasdezeeuw]

We have/had a partially working solution in #1345, but there is not enough interest to make this work. Because of this I'm closing this.