From fc89f1af2c40ee4165e1c3cef68400ea53d28deb Mon Sep 17 00:00:00 2001 From: Manish Goregaokar Date: Thu, 30 Dec 2021 16:37:30 -0800 Subject: [PATCH 1/9] Add design doc --- utils/yoke/design_doc.md | 235 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 235 insertions(+) create mode 100644 utils/yoke/design_doc.md diff --git a/utils/yoke/design_doc.md b/utils/yoke/design_doc.md new file mode 100644 index 00000000000..b406c4b7cde --- /dev/null +++ b/utils/yoke/design_doc.md @@ -0,0 +1,235 @@ +# Yoke: Lifetime Erasure for Rust + +## Problem statement + +Zero-copy deserialization is a very effective way to speed up programs and avoid , however can lead to lifetimes pervasively spreading throughout the codebase, and prevents using diverse memory management techniques like caching. + +It would be nice if it were possible to "erase" lifetimes and turn them into dynamically managed lifetimes (similar to type erasure with `dyn`) to allow for more flexible memory management. + +## Background + +[ICU4X](https://github.com/unicode-org/icu4x) is an internationalization library that has pluggable data loading as a core value proposition. Internationalization often needs a lot of data, and we want to make sure data loading can be fast and efficient. Zero-copy deserialization is quite attractive as a way to reduce this load. + +Unfortunately, zero-copy deserialization leads to pervasive lifetimes in anything that consumes this data. The user has to hold on to the data source for as long as the deserialized data is needed, which can be a pain. More sophisticated memory management strategies, like using `Rc` to dynamically cache the source of the data for as long as it's needed, cannot work since lifetimes are purely static constructs. + +It would be nice if it were possible to "erase" lifetimes and allow for memory management of zero-copy deserialized data. + + +## Requirements + + - It should be possible to use zero-copy deserialization without storage of the deserialized data introducing a lifetime (**required**) + - It should be possible to zero-copy deserialize data from some source and have that source data kept alive until the deserialized data is no longer needed, perhaps using things like `Rc`, with dynamically-known lifetimes (**required**) + - It should be possible to manipulate the zero-copy deserialized data, not just read it (**preferred**) + - It should be possible to conveniently use this abstraction without needing to write any unsafe code (**preferred**) + - It would be nice if complicated lifetimes were not introduced at _any_ point (**optional**) + + +## High level design + +The `yoke` crate provides the [`Yoke`][`Yoke`] and [`Yokeable<'a>`][`Yokeable`] traits, which form its core value proposition. + +`Yoke` allows one to "yoke" a zero-copy deserialized object (say, a `Cow<'a, str>`) to the source it was deserialized from, (say, an `Rc<[u8]>`), known as a "cart", producing a type that looks like `Yoke, Rc<[u8]>>` and can be moved around with impunity. + +The `'static` is somewhat of a lie, the lifetime of the data the `Cow` borrows from is the lifetime of the `Yoke`, however since this `Cow` cannot be normally extracted from the `Yoke` the lifetime does not matter. + +Most of the time the yokeable `Y` type will be some kind of zero-copy deserializable abstraction, potentially with an owned variant (like `Cow`, [`ZeroVec`](https://docs.rs/zerovec), or an aggregate containing such types), and the cart `C` will be some smart pointer like `Box`, `Rc`, or `Arc`, potentially wrapped in an `Option`. + +### Basic functionality + +The `Yokeable<'a>` trait is implemented on the `'static` version of any zero-copy type, e.g. `Cow<'static, T>` implements `Yokeable<'a>` (for all `'a`). One can use `Yokeable::Output` on this trait to obtain the "lifetime'd" value of the `Cow<'static, T>`, e.g. ` as Yokeable<'a>'>::Output` is `Cow<'a, T>`. + +The key behind this crate is [`Yoke::get()`][get], with a signature as follows: + +```rust +impl Yoke where Y: for<'a> Yokeable<'a> { + pub fn get<'a>(&'a self) -> &'a >::Output { + // ... + } +} +``` + +Essentially, calling [`.get()`][get] on a type like `Yoke, _>` will get you a short-lived `&'a Cow<'a, str>`, restricted to the lifetime of the borrow used during `.get()`. This is entirely safe since the `Cow` borrows from the cart type, which cannot be interfered with as long as the `Yoke` is borrowed by `.get()`. `.get()` protects access by essentially reifying the erased lifetime to a safe local one when necessary. + + +Constructing a `Yoke` can be done using one of the [`attach_to_cart()`][attach] methods with a signature like so: + + +```rust +impl Yoke where Y: for<'a> Yokeable<'a>, C: StableDeref { + pub fn attach_to_cart(cart: C, f: F) -> Self + where F: for<'de> FnOnce(&'de ::Target) -> >::Output { + // .. + } + +} +``` + +which can be used as follows: + +```rust +let data: Rc<[u8]> = load_data_from_file(); + +// constructs a Yoke by zero-copy deserializing from data +let yoke: Yoke, Rc<[u8]>> = Yoke::attach_to_cart(data.clone(), |data| deserialize_cow_from_data(data)); + +// prints the data out +println!("{:?}", yoke.get()); + +// you can move `yoke` without needing to move data +``` + +`Yokeable` is implemented on all basic borrowed types `Cow<'static, T>`, `&'static T`, etc, and can be implemented on complex types using [`#[derive(Yokeable)]`][yokeable-derive]: + +```rust +#[derive(Yokeable)] +struct AggregateZeroCopy<'a> { + field1: Cow<'a, str>, + field2: Cow<'a, [u32]>, +} +``` + +As usual with zero-copy deserialization, it is good to use types that have both owned and borrowed variants, like `Cow` and [`ZeroVec`](https://docs.rs/zerovec) + +### Mutation + +It is possible to mutate `Yoke`s using the [`.with_mut()`](with_mut) method. This cannot be used to introduce _new_ borrowed data, however one can freely mutate values (or parts of values) into their owned variants: + +```rust +// yoke is a Yoke, _> +yoke.with_mut(|cow| { + let mut_str = cow.to_mut(); + mut_str.clear(); + mut_str.push_str("bar"); +}); +``` + +### Projection + +There are various [`.project()`][project] methods that allow turning a `Yoke` into another `Yoke` containing a different type that may contain elements of the original yoked value. For example, one can turn a `Yoke<&str, _>` into a `Yoke<&[u8]>` like so: + +```rust +yoke.project(move |yk, _| yoke.as_bytes()); +``` + + +This can also be used to, for example, focus a `Yoke` onto containing a subfield of the type it uses, e.g. we can extract `field1` of `AggregateZeroCopy` above with: + +```rust +yoke.project(move |yk, _| yoke.field1); +``` + + [get]: https://docs.rs/yoke/latest/yoke/struct.Yoke.html#method.get + [attach]: https://docs.rs/yoke/latest/yoke/struct.Yoke.html#method.attach + [with_mut]: https://docs.rs/yoke/latest/yoke/struct.Yoke.html#method.with_mut + [project]: https://docs.rs/yoke/latest/yoke/struct.Yoke.html#method.project + +## Detailed design + +### Yokeable + +The core design of [`Yoke`] relies on the [`Yokeable`] trait, with the following signature: + +```rust +pub unsafe trait Yokeable<'a>: 'static { + type Output: 'a; + fn transform(&'a self) -> &'a Self::Output; + fn transform_owned(self) -> Self::Output; + unsafe fn make(from: Self::Output) -> Self; + fn transform_mut(&'a mut self, f: F) + where F: 'static + for<'b> FnOnce(&'b mut Self::Output); +} +``` + +This trait should be implemented for all `'a` on the `'static` version of any type with [covariant] lifetimes, with the `Output` type having the lifetime swapped with `'a`. In essence, a [covariant] lifetime is one where replacing the lifetime with a shorter lifetime is always safe. This is true for, e.g. `'a` in `&'a T` and `Cow<'a, T>`, but it is not true in `Cell<&'a T>` or `fn(&'a u8)`. The covariance is crucial here, because `Yoke` uses this trait as a utility for replacing lifetimes, and the trait asserts that replacing lifetimes this way is safe. + +A way of looking at `Self::Output` is `Self<'a>` if we look at `Self` as a higher kinded type here: the shorthand `Self<'a>` is equivalent to `>::Output`). In this explanation we can use `Self<'static>` and `Self<'a>` (etc) to be clearer about what's happening. + +The first two methods on the trait are basically an in-code representation of covariance: it should always be possible to replace `Self<'static>` with `Self<'a>`. In fact, in most cases these methods can be implemented with the body being just `self`. + +`make()` is used when _constructing_ `Yoke`s and enables us to construct the "fake" `'static` lifetime by converting `Self<'a>` to `Self<'static>`. This is very unsafe, and is only called when `Yoke`s are being constructed since such an operation is basically handing over responsibility for respecting the lifetime to the `Yoke`. + +`transform_mut()` is interesting: it's what enables [`Yoke::with_mut()`][with_mut], and it works by enforcing extremely strict constraints on what data can be smuggled in or out of the closure passed to it. + +The bound is `F: 'static + for<'b> FnOnce(&'b mut Self::Output)`: the `'static` ensures that no borrowed data can be pulled from the outside, and the `for<'b>` ensures that data from the inside cannot leak since the closure has to be valid for _all_ lifetimes, not just the lifetime of `self`, which means that it needs to theoretically be able to handle data with much shorter lifetimes even if it's not actually going to be passed such data. + +### Safety of various methods + +The main things that need to be taken care of are: + + 1. Borrowed data from within the `Yoke` may only escape with short-lived lifetimes, if at all + 2. When constructing or modifying a `Yoke` no borrowed data from the outside may sneak in, it must all come from the Cart. + + +Most of the methods derive safety from careful choices of bounds in their signatures. + + +[`.get()`][get] has the following signature: + +```rust +impl Yoke where Y: for<'a> Yokeable<'a> { + pub fn get<'a>(&'a self) -> &'a >::Output { + // ... + } +} +``` + +by returning `&'a Y<'a>`, `.get()` enforces that the returned type cannot live longer than the method borrow, satisfying safety point 1. Point 2 is irrelevant since this is not mutating or constructing `Yoke`s + +[`Yoke::attach_to_cart()`][attach] and friends have signatures like the following: + +```rust +impl Yoke where Y: for<'a> Yokeable<'a>, C: StableDeref { + pub fn attach_to_cart(cart: C, f: F) -> Self + where F: for<'de> FnOnce(&'de ::Target) -> >::Output { + // .. + } +} +``` + +The `StableDeref` requirement on the `Cart` type ensures that moving the cart does not move any of the data found by `Deref`ing the cart, which is crucial to ensure that the closure is only operating on data already pinned by the allocation of the cart. The `for<'de>` on `F` pins down the lifetime flow by ensuring that any borrowed data in the output (`Y<'de>`) _must_ have been borrowed from the input (`&'de C::Target`). It can't be borrowed from anywhere else because the function must be written to work for all lifetimes `'de`. This satisfies safety point 2, and point 1 is irrelevant since the `Yoke` hasn't been constructed yet. + +[`.with_mut()`][with_mut] just proxies to `Yokeable::transform_mut` and satisfies point 2 for the reasons stated above when describing that method. It also satisfies point 1 due to the `for<'a>` lifetime in the bound; if the function is allowed to be called with any lifetime, it cannot export that data since it will not statically know what the lifetime will be. + +[`.project()`] has the following signature (the other project methods are a little bit more complicated to allow for captures, but they work off the same principles): + +```rust +impl Yoke where Y: for<'a> Yokeable<'a> { + pub fn project

( + self, + f: for<'a> fn(_: >::Output, _: PhantomData<&'a ()>) ->

>::Output + ) -> Yoke + where P: for<'a> Yokeable<'a> + // ... + } +} +``` + +This is a fair bit more complicated. First off the bat, the `PhantomData` can be ignored completely, it exists to satisfy the compiler (which needs the lifetime `'a` to be used in a more concrete place). + +What this function does is take a closure that, for all `'a`, can convert `Y<'a>` to `P<'a>`. The `for<'a>` here has the same effect as in `Yoke::attach_to_cart()`: it pins down a lifetime flow such that all borrowed data in `P<'a>` _must_ have come from `Y<'a>` and nowhere else, satisfying safety point 2. There's no chance for `f` to smuggle any borrowed out so safety point 1 is also satisfied. The `with_capture` variants of this are able to enforce that no data of the wrong lifetime is smuggled out by relying on the `for<'a>` again: any data being smuggled out would not have a statically known lifetime and cannot be stuffed into the capture since that would give the capture a statically unknown lifetime. + + +### ZeroCopyFrom + +The `yoke` crate comes with an additional trait, [`ZeroCopyFrom`], which can be used to reborrow cow-like zero-copy types. It is not directly used by [`Yoke`], however it is a useful utility to pair with it. + +Using this trait, for example, one can generically talk about taking a `Cow<'a, T>` and borrowing it to produce a `Cow<'b, T>` that is `Cow::Borrowed`, borrowing from the original `Cow`, regardless of whether or not the original `Cow` was owned or borrowed. + +It has the following signature and also has a custom derive: + +```rust +pub trait ZeroCopyFrom: for<'a> Yokeable<'a> { + fn zero_copy_from<'b>(cart: &'b C) -> >::Output; +} +``` + + + [covariant]: https://doc.rust-lang.org/nomicon/subtyping.html + [`Yoke`]: https://docs.rs/yoke/latest/yoke/struct.Yoke.html + [`Yokeable`]: https://docs.rs/yoke/latest/yoke/trait.Yokeable.html + [yokeable-derive]: https://docs.rs/yoke/latest/yoke/derive.Yokeable.html + [`ZeroCopyFrom`]: https://docs.rs/yoke/latest/yoke/trait.ZeroCopyFrom.html + [get]: https://docs.rs/yoke/latest/yoke/struct.Yoke.html#method.get + [attach]: https://docs.rs/yoke/latest/yoke/struct.Yoke.html#method.attach + [with_mut]: https://docs.rs/yoke/latest/yoke/struct.Yoke.html#method.with_mut + [project]: https://docs.rs/yoke/latest/yoke/struct.Yoke.html#method.project From daf266e4e6e60a62a71d4951b9d29ae91691df2a Mon Sep 17 00:00:00 2001 From: Manish Goregaokar Date: Thu, 30 Dec 2021 17:12:06 -0800 Subject: [PATCH 2/9] fix --- utils/yoke/design_doc.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/utils/yoke/design_doc.md b/utils/yoke/design_doc.md index b406c4b7cde..f6270f37a91 100644 --- a/utils/yoke/design_doc.md +++ b/utils/yoke/design_doc.md @@ -2,7 +2,7 @@ ## Problem statement -Zero-copy deserialization is a very effective way to speed up programs and avoid , however can lead to lifetimes pervasively spreading throughout the codebase, and prevents using diverse memory management techniques like caching. +Zero-copy deserialization is a very effective way to speed up programs and avoid allocations, however can lead to lifetimes pervasively spreading throughout the codebase, and prevents using diverse memory management techniques like caching. It would be nice if it were possible to "erase" lifetimes and turn them into dynamically managed lifetimes (similar to type erasure with `dyn`) to allow for more flexible memory management. From e4fe4986136b5b80e2ca8c32052df0b357e424e3 Mon Sep 17 00:00:00 2001 From: Manish Goregaokar Date: Thu, 30 Dec 2021 17:12:45 -0800 Subject: [PATCH 3/9] fix --- utils/yoke/design_doc.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/utils/yoke/design_doc.md b/utils/yoke/design_doc.md index f6270f37a91..3ec3bed2286 100644 --- a/utils/yoke/design_doc.md +++ b/utils/yoke/design_doc.md @@ -190,7 +190,7 @@ The `StableDeref` requirement on the `Cart` type ensures that moving the cart do [`.with_mut()`][with_mut] just proxies to `Yokeable::transform_mut` and satisfies point 2 for the reasons stated above when describing that method. It also satisfies point 1 due to the `for<'a>` lifetime in the bound; if the function is allowed to be called with any lifetime, it cannot export that data since it will not statically know what the lifetime will be. -[`.project()`] has the following signature (the other project methods are a little bit more complicated to allow for captures, but they work off the same principles): +[`.project()`][project] has the following signature (the other project methods are a little bit more complicated to allow for captures, but they work off the same principles): ```rust impl Yoke where Y: for<'a> Yokeable<'a> { From 86382a3fd26b5de41c7a46546a4b65a8d9424f5a Mon Sep 17 00:00:00 2001 From: "Shane F. Carr" Date: Fri, 7 Jan 2022 05:53:51 -0600 Subject: [PATCH 4/9] Manually apply a few changes --- utils/yoke/design_doc.md | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/utils/yoke/design_doc.md b/utils/yoke/design_doc.md index 3ec3bed2286..a8392ae4958 100644 --- a/utils/yoke/design_doc.md +++ b/utils/yoke/design_doc.md @@ -1,20 +1,18 @@ -# Yoke: Lifetime Erasure for Rust +# Yoke: Self-Referential Borrowing for Rust ## Problem statement -Zero-copy deserialization is a very effective way to speed up programs and avoid allocations, however can lead to lifetimes pervasively spreading throughout the codebase, and prevents using diverse memory management techniques like caching. +Zero-copy deserialization is a very effective way to speed up programs and avoid allocations. However, the requirement that data is borrowed from somewhere else means that: -It would be nice if it were possible to "erase" lifetimes and turn them into dynamically managed lifetimes (similar to type erasure with `dyn`) to allow for more flexible memory management. +1. All data types that contain zero-copy data, even indirectly, need to carry a lifetime parameter +2. Certain memory management techniques are hampered, like caching. + +The goal of Yoke is to allow the borrowing of data from self, so that we don't need lifetime parameters to track data ownership, and to enable reference-counted data that can be safely dropped from a cache. ## Background [ICU4X](https://github.com/unicode-org/icu4x) is an internationalization library that has pluggable data loading as a core value proposition. Internationalization often needs a lot of data, and we want to make sure data loading can be fast and efficient. Zero-copy deserialization is quite attractive as a way to reduce this load. -Unfortunately, zero-copy deserialization leads to pervasive lifetimes in anything that consumes this data. The user has to hold on to the data source for as long as the deserialized data is needed, which can be a pain. More sophisticated memory management strategies, like using `Rc` to dynamically cache the source of the data for as long as it's needed, cannot work since lifetimes are purely static constructs. - -It would be nice if it were possible to "erase" lifetimes and allow for memory management of zero-copy deserialized data. - - ## Requirements - It should be possible to use zero-copy deserialization without storage of the deserialized data introducing a lifetime (**required**) @@ -30,13 +28,13 @@ The `yoke` crate provides the [`Yoke`][`Yoke`] and [`Yokeable<'a>`][`Yokea `Yoke` allows one to "yoke" a zero-copy deserialized object (say, a `Cow<'a, str>`) to the source it was deserialized from, (say, an `Rc<[u8]>`), known as a "cart", producing a type that looks like `Yoke, Rc<[u8]>>` and can be moved around with impunity. -The `'static` is somewhat of a lie, the lifetime of the data the `Cow` borrows from is the lifetime of the `Yoke`, however since this `Cow` cannot be normally extracted from the `Yoke` the lifetime does not matter. +The `'static` is somewhat of a lie: it is actually a self-referential lifetime. The `Cow` is allowed to borrow data from the cart (the `Rc<[u8]>`), but the Rust compiler does not allow this, so we use `'static`. Since this `Cow` cannot be normally extracted from the `Yoke`, the lifetime is considered an implementation detail. Most of the time the yokeable `Y` type will be some kind of zero-copy deserializable abstraction, potentially with an owned variant (like `Cow`, [`ZeroVec`](https://docs.rs/zerovec), or an aggregate containing such types), and the cart `C` will be some smart pointer like `Box`, `Rc`, or `Arc`, potentially wrapped in an `Option`. ### Basic functionality -The `Yokeable<'a>` trait is implemented on the `'static` version of any zero-copy type, e.g. `Cow<'static, T>` implements `Yokeable<'a>` (for all `'a`). One can use `Yokeable::Output` on this trait to obtain the "lifetime'd" value of the `Cow<'static, T>`, e.g. ` as Yokeable<'a>'>::Output` is `Cow<'a, T>`. +The `Yokeable<'a>` trait is implemented on the `'static` version of any zero-copy type; for example, `Cow<'static, T>` implements `Yokeable<'a>` (for all `'a`). One can use `Yokeable::Output` on this trait to obtain the "lifetime'd" value of the `Cow<'static, T>`, e.g. ` as Yokeable<'a>'>::Output` is `Cow<'a, T>`. The key behind this crate is [`Yoke::get()`][get], with a signature as follows: From 2f2df9a2f57a65c75a6f1222ed38cb5719b65684 Mon Sep 17 00:00:00 2001 From: "Shane F. Carr" Date: Fri, 7 Jan 2022 13:28:16 -0600 Subject: [PATCH 5/9] More feedback edited directly into the doc --- utils/yoke/design_doc.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/utils/yoke/design_doc.md b/utils/yoke/design_doc.md index a8392ae4958..fb44d0e7966 100644 --- a/utils/yoke/design_doc.md +++ b/utils/yoke/design_doc.md @@ -90,7 +90,7 @@ As usual with zero-copy deserialization, it is good to use types that have both ### Mutation -It is possible to mutate `Yoke`s using the [`.with_mut()`](with_mut) method. This cannot be used to introduce _new_ borrowed data, however one can freely mutate values (or parts of values) into their owned variants: +It is possible to mutate `Yoke`s using the [`.with_mut()`](with_mut) method. This cannot be used to introduce _new_ borrowed data, but one can freely mutate values (or parts of values) into their owned variants: ```rust // yoke is a Yoke, _> @@ -202,14 +202,14 @@ impl Yoke where Y: for<'a> Yokeable<'a> { } ``` -This is a fair bit more complicated. First off the bat, the `PhantomData` can be ignored completely, it exists to satisfy the compiler (which needs the lifetime `'a` to be used in a more concrete place). +This is a fair bit more complicated. First off the bat, the `PhantomData` can be ignored completely; it exists to satisfy the compiler, which needs the lifetime `'a` to be used in a more concrete place. What this function does is take a closure that, for all `'a`, can convert `Y<'a>` to `P<'a>`. The `for<'a>` here has the same effect as in `Yoke::attach_to_cart()`: it pins down a lifetime flow such that all borrowed data in `P<'a>` _must_ have come from `Y<'a>` and nowhere else, satisfying safety point 2. There's no chance for `f` to smuggle any borrowed out so safety point 1 is also satisfied. The `with_capture` variants of this are able to enforce that no data of the wrong lifetime is smuggled out by relying on the `for<'a>` again: any data being smuggled out would not have a statically known lifetime and cannot be stuffed into the capture since that would give the capture a statically unknown lifetime. ### ZeroCopyFrom -The `yoke` crate comes with an additional trait, [`ZeroCopyFrom`], which can be used to reborrow cow-like zero-copy types. It is not directly used by [`Yoke`], however it is a useful utility to pair with it. +The `yoke` crate comes with an additional trait, [`ZeroCopyFrom`]. Implementing this trait allows one to define a canonical, infallible implementation of Yoke's `attach_to_cart` function, enabling various additional constructors on Yoke for convenience, including `Yoke::attach_to_borrowed_cart()`, `Yoke::attach_to_box_cart()`, and `Yoke::attach_to_rc_cart()`. Using this trait, for example, one can generically talk about taking a `Cow<'a, T>` and borrowing it to produce a `Cow<'b, T>` that is `Cow::Borrowed`, borrowing from the original `Cow`, regardless of whether or not the original `Cow` was owned or borrowed. From 33bb5380c357b1fd51e4bfd277342987b291abc6 Mon Sep 17 00:00:00 2001 From: "Shane F. Carr" Date: Fri, 7 Jan 2022 13:31:41 -0600 Subject: [PATCH 6/9] Feedback --- utils/yoke/design_doc.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/utils/yoke/design_doc.md b/utils/yoke/design_doc.md index fb44d0e7966..22b37102fc9 100644 --- a/utils/yoke/design_doc.md +++ b/utils/yoke/design_doc.md @@ -1,4 +1,4 @@ -# Yoke: Self-Referential Borrowing for Rust +# Yoke: Enabling Self-Referential Borrowing via Lifetime Erasure in Rust ## Problem statement @@ -7,7 +7,7 @@ Zero-copy deserialization is a very effective way to speed up programs and avoid 1. All data types that contain zero-copy data, even indirectly, need to carry a lifetime parameter 2. Certain memory management techniques are hampered, like caching. -The goal of Yoke is to allow the borrowing of data from self, so that we don't need lifetime parameters to track data ownership, and to enable reference-counted data that can be safely dropped from a cache. +Yoke utilizes *lifetime erasure* to allow the borrowing of data from self. This means we don't need lifetime parameters to track data ownership, and that we can carry reference-counted data that can be safely dropped from a cache. ## Background From 81c932fa58f3625435e7da5631f036f684c029dc Mon Sep 17 00:00:00 2001 From: Manish Goregaokar Date: Fri, 7 Jan 2022 16:24:49 -0800 Subject: [PATCH 7/9] clarify lifetime erasure --- utils/yoke/design_doc.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/utils/yoke/design_doc.md b/utils/yoke/design_doc.md index 22b37102fc9..95a2767788a 100644 --- a/utils/yoke/design_doc.md +++ b/utils/yoke/design_doc.md @@ -1,4 +1,4 @@ -# Yoke: Enabling Self-Referential Borrowing via Lifetime Erasure in Rust +# Yoke: Lifetime Erasure in Rust ## Problem statement @@ -7,7 +7,7 @@ Zero-copy deserialization is a very effective way to speed up programs and avoid 1. All data types that contain zero-copy data, even indirectly, need to carry a lifetime parameter 2. Certain memory management techniques are hampered, like caching. -Yoke utilizes *lifetime erasure* to allow the borrowing of data from self. This means we don't need lifetime parameters to track data ownership, and that we can carry reference-counted data that can be safely dropped from a cache. +Similar to how `dyn` enables Rust programs to perform "type erasure": turning compile-time types into "erased" runtime ones, `Yoke` enables Rust programs to perform the analogous "lifetime erasure": turning compile-time lifetimes into "erased" runtime ones. This means we don't need lifetime parameters to track data ownership, and that we can carry reference-counted data that can be safely dropped from a cache. ## Background From 60ec140403d6490538507d8eac5e0e1b9047b732 Mon Sep 17 00:00:00 2001 From: Manish Goregaokar Date: Fri, 7 Jan 2022 16:27:40 -0800 Subject: [PATCH 8/9] quotes --- utils/yoke/design_doc.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/utils/yoke/design_doc.md b/utils/yoke/design_doc.md index 95a2767788a..0f8497f414e 100644 --- a/utils/yoke/design_doc.md +++ b/utils/yoke/design_doc.md @@ -1,4 +1,4 @@ -# Yoke: Lifetime Erasure in Rust +# Yoke: "Lifetime Erasure" in Rust ## Problem statement From 17576104e5cd2c0ed2edeb6e86080a64d1727836 Mon Sep 17 00:00:00 2001 From: Manish Goregaokar Date: Fri, 7 Jan 2022 16:35:53 -0800 Subject: [PATCH 9/9] mention targeting --- utils/yoke/design_doc.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/utils/yoke/design_doc.md b/utils/yoke/design_doc.md index 0f8497f414e..47ea9ec0abe 100644 --- a/utils/yoke/design_doc.md +++ b/utils/yoke/design_doc.md @@ -1,4 +1,4 @@ -# Yoke: "Lifetime Erasure" in Rust +# Yoke: Targeted Lifetime Erasure in Rust ## Problem statement @@ -7,7 +7,7 @@ Zero-copy deserialization is a very effective way to speed up programs and avoid 1. All data types that contain zero-copy data, even indirectly, need to carry a lifetime parameter 2. Certain memory management techniques are hampered, like caching. -Similar to how `dyn` enables Rust programs to perform "type erasure": turning compile-time types into "erased" runtime ones, `Yoke` enables Rust programs to perform the analogous "lifetime erasure": turning compile-time lifetimes into "erased" runtime ones. This means we don't need lifetime parameters to track data ownership, and that we can carry reference-counted data that can be safely dropped from a cache. +Similar to how `dyn` enables Rust programs to perform "type erasure": turning compile-time types into "erased" runtime ones, `Yoke` enables Rust programs to perform the analogous "lifetime erasure": turning specific compile-time lifetimes into "erased" runtime ones. This means we don't need lifetime parameters to track data ownership, and that we can carry reference-counted data that can be safely dropped from a cache. ## Background