Add Pinning section to the Unsafe Deep Dive (WIP)

src/SUMMARY.md

@@ -496,6 +496,23 @@
   - [Data structures are safe](unsafe-deep-dive/foundations/data-structures-are-safe.md)
   - [Actions might not be](unsafe-deep-dive/foundations/actions-might-not-be.md)
   - [Less powerful than it seems](unsafe-deep-dive/foundations/less-powerful.md)
+- [Pinning](unsafe-deep-dive/pinning.md)
+  - [What pinning is](unsafe-deep-dive/pinning/what-pinning-is.md)
+  - [What a move is](unsafe-deep-dive/pinning/what-a-move-is.md)
+  - [Definition of Pin<Ptr>](unsafe-deep-dive/pinning/definition-of-pin.md)
+  - [Why it's difficult](unsafe-deep-dive/pinning/why-difficult.md)
+  - [`Unpin` trait](unsafe-deep-dive/pinning/unpin-trait.md)
+  - [`PhantomPinned`](unsafe-deep-dive/pinning/phantompinned.md)
+  - [Self-Referential Buffer Example](unsafe-deep-dive/pinning/self-referential-buffer.md)
+    - [C++ implementation](unsafe-deep-dive/pinning/self-referential-buffer/cpp.md)
+    - [Modeled in Rust](unsafe-deep-dive/pinning/self-referential-buffer/rust.md)
+      - [With a raw pointer](unsafe-deep-dive/pinning/self-referential-buffer/rust-raw-pointers.md)
+      - [With an integer offset](unsafe-deep-dive/pinning/self-referential-buffer/rust-offset.md)
+      - [With `Pin<Ptr>`](unsafe-deep-dive/pinning/self-referential-buffer/rust-pin.md)
+  - [`Pin<Ptr>` and `Drop`](unsafe-deep-dive/pinning/pin-and-drop.md)
+    - [Worked Example](unsafe-deep-dive/pinning/drop-and-not-unpin-worked-example.md)
+- [Case Studies](unsafe-deep-dive/case-studies.md)
+  - [Tokio's Intrusive Linked List](unsafe-deep-dive/case-studies/intrusive-linked-list.md)
 
 ---
 

src/unsafe-deep-dive/case-studies/intrusive-linked-list.md

@@ -0,0 +1,106 @@
+# Tokio's Intrusive Linked List
+
+The Tokio project maintains an [intrusive linked list implementation][ill] that
+demonstrates many use cases of `unsafe` and a number of types and traits from
+Rust's unsafe ecosystem, including `cell::UnsafeCell`, `mem::ManuallyDrop`,
+[pinning](../pinning/what-pinning-is.md), and unsafe traits.
+
+A linked list is a difficult data structure to implement in Rust, because it
+relies heavily on stable memory addresses remaining stable. This isn't something
+that happens naturally, as Rust types change their memory address every time
+they move.
+
+## Introductory Walkthrough
+
+> Code snippets are taken from tokio v1.48.0
+
+The public API is provided by the `LinkedList<L, T>` type, which contains fields
+for the start and the end of the list. `Option<NonNull<T>>` could be read as a
+`Option<*mut T>`, with the assurance that null pointers will never be created.
+
+`NonNull<T>` is "_covariant_ over `T`", which means that `NonNull<T>` inherits
+the [variance] relationships from `T`.
+
+```rust,ignore
+use core::marker::PhantomData;
+
+// ...
+
+/// An intrusive linked list.
+///
+/// Currently, the list is not emptied on drop. It is the caller's
+/// responsibility to ensure the list is empty before dropping it.
+pub(crate) struct LinkedList<L, T> {
+    /// Linked list head
+    head: Option<NonNull<T>>,
+
+    /// Linked list tail
+    tail: Option<NonNull<T>>,
+
+    /// Node type marker.
+    _marker: PhantomData<*const L>,
+}
+```
+
+`LinkedList` is neither `Send` nor `Sync`, unless its targets are.
+
+```rust,ignore
+unsafe impl<L: Link> Send for LinkedList<L, L::Target> where L::Target: Send {}
+unsafe impl<L: Link> Sync for LinkedList<L, L::Target> where L::Target: Sync {}
+```
+
+The `Link` trait used the those trait bounds is defined next. `Link` is an
+unsafe trait that manages the relationships between nodes when the list needs to
+pass references externally to callers.
+
+Here's trait's definition. The most significant method is `pointers()`, which
+returns a `Pointers` struct. `Pointers` provides access to the two ends of the
+link by marking itself as `!Unpin`.
+
+```rust,ignore
+pub unsafe trait Link {
+    type Handle;
+
+    type Target;
+
+    fn as_raw(handle: &Self::Handle) -> NonNull<Self::Target>;
+
+    unsafe fn from_raw(ptr: NonNull<Self::Target>) -> Self::Handle;
+
+    /// # Safety
+    ///
+    /// The resulting pointer should have the same tag in the stacked-borrows
+    /// stack as the argument. In particular, the method may not create an
+    /// intermediate reference in the process of creating the resulting raw
+    /// pointer.
+    unsafe fn pointers(
+        target: NonNull<Self::Target>,
+    ) -> NonNull<Pointers<Self::Target>>;
+}
+```
+
+`Pointers` is where the magic happens:
+
+```rust,ignore
+pub(crate) struct Pointers<T> {
+    inner: UnsafeCell<PointersInner<T>>,
+}
+
+struct PointersInner<T> {
+    prev: Option<NonNull<T>>,
+    next: Option<NonNull<T>>,
+
+    /// This type is !Unpin due to the heuristic from:
+    /// <https://github.com/rust-lang/rust/pull/82834>
+    _pin: PhantomPinned,
+}
+```
+
+## Remarks
+
+Understanding the whole implementation will take some time, but it's a rewarding
+experience. The code demonstrates composing many parts of unsafe Rust's
+ecosystem into a workable, high performance data structure. Enjoy exploring!
+
+[ill]: https://docs.rs/tokio/1.48.0/src/tokio/util/linked_list.rs.html
+[variance]: https://doc.rust-lang.org/reference/subtyping.html

src/unsafe-deep-dive/pinning.md

@@ -0,0 +1,30 @@
+# Welcome
+
+This segment of the course covers:
+
+- What "pinning" is
+- Why it is necessary
+- How Rust implements it
+- How it interacts with unsafe and FFI
+
+<details>
+
+"Pinning, or holding a value's memory address in a fixed location,is one of the
+more challenging concepts in Rust."
+
+"Normally only seen within async code, i.e. [`poll(self: Pin<&mut Self>)`],
+pinning has wider applicability."
+
+Some some data structures that are difficult or impossible to write without the
+unsafe keyword, including self-referential structs and intrusive data
+structures.
+
+FFI with C++ is a prominent use case that's related to this. Rust must assume
+that any C++ with a reference might be a self-referential data structure.
+
+"To understand this conflict in more detail, we'll first need to make sure that
+we have a strong understanding of Rust's move semantics."
+
+<details>
+
+[poll]: https://doc.rust-lang.org/std/future/trait.Future.html#tymethod.poll

src/unsafe-deep-dive/pinning/README.md

@@ -0,0 +1,19 @@
+# pinning
+
+> **Important Note**
+>
+> To not add this section to the project's SUMMARY.md yet. Once CLs/PRs to
+> accept all the new segments for the Unsafe Deep Dive have been included in the
+> repository, an update to SUMMARY.md will be made.
+
+## About
+
+This segment explains pinning, Rust's `Pin<Ptr>` type and concepts that relate
+to FFI rather than its async use case. Treatment of the `Unpin` trait and the
+`PhantomPinned` type is provided.
+
+## Status
+
+Provisional/beta.
+
+## Outline

src/unsafe-deep-dive/pinning/definition-of-pin.md

@@ -0,0 +1,37 @@
+---
+minutes: 5
+---
+
+# Definition of Pin
+
+```rust,ignore
+#[repr(transparent)]
+pub struct Pin<Ptr> {
+    pointer: Ptr,
+}
+
+impl<Ptr: Deref<Target: Unpin>> Pin<Ptr> {
+    pub fn new(pointer: Ptr) -> Pin<Ptr> { ... }
+}
+
+impl<Ptr: Deref> Pin<Ptr> {
+    pub unsafe fn new_unchecked(pointer: Ptr) -> Pin<Ptr> { ... }
+}
+```
+
+<details>
+
+`Pin` is a minimal wrapper around a _pointer type_, which is defined as a type
+that implements `Deref`.
+
+However, `Pin::new()` only accepts types that dereference into a target that
+implements `Unpin` (`Deref<Target: Unpin>`). This allows `Pin` to rely on the
+the type system to enforce its guarantees.
+
+Types that do not implement `Unpin`, i.e. types that require pinning, must
+create a `Pin` via the unsafe `Pin::new_unchecked()`.
+
+Aside: Unlike other `new()`/`new_unchecked()` method pairs, `new` does not do
+any runtime checking. The check is a zero-cost compile-time check.
+
+</details>

src/unsafe-deep-dive/pinning/drop-and-not-unpin-worked-example.md

@@ -0,0 +1,112 @@
+---
+minutes: 5
+---
+
+# Worked example Implementing `Drop` for `!Unpin` types
+
+```rust,editable,ignore
+use std::cell::RefCell;
+use std::marker::PhantomPinned;
+use std::mem;
+use std::pin::Pin;
+
+thread_local! {
+    static BATCH_FOR_PROCESSING: RefCell<Vec<String>> = RefCell::new(Vec::new());
+}
+
+#[derive(Debug)]
+struct CustomString(String);
+
+#[derive(Debug)]
+struct SelfRef {
+    data: CustomString,
+    ptr: *const CustomString,
+    _pin: PhantomPinned,
+}
+
+impl SelfRef {
+    fn new(data: &str) -> Pin<Box<SelfRef>> {
+        let mut boxed = Box::pin(SelfRef {
+            data: CustomString(data.to_owned()),
+            ptr: std::ptr::null(),
+            _pin: PhantomPinned,
+        });
+
+        let ptr: *const CustomString = &boxed.data;
+        unsafe {
+            Pin::get_unchecked_mut(Pin::as_mut(&mut boxed)).ptr = ptr;
+        }
+        boxed
+    }
+}
+
+impl Drop for SelfRef {
+    fn drop(&mut self) {
+        // SAFETY: Safe because we are reading bytes from a String
+        let payload = unsafe { std::ptr::read(&self.data) };
+        BATCH_FOR_PROCESSING.with(|log| log.borrow_mut().push(payload.0));
+    }
+}
+
+fn main() {
+    let pinned = SelfRef::new("Rust 🦀");
+    drop(pinned);
+
+    BATCH_FOR_PROCESSING.with(|batch| {
+        println!("Batch: {:?}", batch.borrow());
+    });
+}
+```
+
+<details>
+
+This example uses the `Drop` trait to add data for some post-processing, such as
+telemetry or logging.
+
+**The Safety comment is incorrect.** `ptr::read` creates a bitwise copy, leaving
+`self.data` in an invalid state. `self.data` will be dropped again at the end of
+the method, which is a double free.
+
+Ask the class for fix the code.
+
+**Suggestion 0: Redesign**
+
+Redesign the post-processing system to work without `Drop`.
+
+**Suggestion 1: Clone**
+
+Using `.clone()` is an obvious first choice, but it allocates memory.
+
+```rust,ignore
+impl Drop for SelfRef {
+    fn drop(&mut self) {
+        let payload = self.data.0.clone();
+        BATCH_FOR_PROCESSING.with(|log| log.borrow_mut().push(payload));
+    }
+}
+```
+
+**Suggestion 2: ManuallyDrop**
+
+Wrapping `CustomString` in `ManuallyDrop` prevents the (second) automatic drop
+at the end of the `Drop` impl.
+
+```rust,ignore
+struct SelfRef {
+    data: ManuallyDrop<CustomString>,
+    ptr: *const CustomString,
+    _pin: PhantomPinned,
+}
+
+// ...
+
+impl Drop for SelfRef {
+    fn drop(&mut self) {
+        // SAFETY: self.data
+        let payload = unsafe { ManuallyDrop::take(&mut self.data) };
+        BATCH_FOR_PROCESSING.with(|log| log.borrow_mut().push(payload.0));
+    }
+}
+```
+
+</details>

src/unsafe-deep-dive/pinning/phantompinned.md

@@ -0,0 +1,36 @@
+---
+minutes: 5
+---
+
+# PhantomPinned
+
+## Definition
+
+```rust,ignore
+pub struct PhantomPinned;
+
+impl !Unpin for PhantomPinned {}
+```
+
+## Usage
+
+```rust,editable
+pub struct DynamicBuffer {
+    data: Vec<u8>,
+    cursor: NonNull<u8>,
+    _pin: std::marker::PhantomPinned,
+}
+```
+
+<details>
+
+`PhantomPinned` is a marker type.
+
+If a type contains a `PhantomPinned`, it will not implement `Unpin` by default.
+
+This has the effect of enforcing pinning when `DynamicBuffer` is wrapped by
+`Pin`.
+
+</details>
+
+<!-- TODO: Monitor issue https://github.com/rust-lang/rust/issues/125735 as this guidance will change at some point and future code will move to UnsafePinned -->