Overhaul internals to support faster execution (#111)

The internals are overhauled to avoid as much allocation as possible.
This is done by leveraging the object store as much as possible and hard
coding settings such as max threads. Additionally, the history tracked
by atomic cells is limited. These changes enable tracking state without
the use of vecs and other allocated collections.

The atomic abstraction now has improved coherence tracking.

The "backtrace" feature is removed in favor of the nightly `Location`
API. To enable this, a nightly Rust must be used and `--cfg
loom_nightly` specified at compile time.

This patch comes with a number of breaking changes, including:

* `CausalCell` deferred checks is removed as doing so is undefined
  behavior.
* `CausalCell` is renamed to `UnsafeCell`.

Cargo.toml

@@ -9,7 +9,7 @@ name = "loom"
 #   - Cargo.toml
 #   - README.md
 # - Create git tag
-version = "0.2.15"
+version = "0.3.0"
 edition = "2018"
 license = "MIT"
 authors = ["Carl Lerche <me@carllerche.com>"]
@@ -38,4 +38,3 @@ serde = { version = "1.0.92", features = ["derive"], optional = true }
 serde_json = { version = "1.0.33", optional = true }
 
 futures-util = { version = "0.3.0", optional = true }
-backtrace = { version = "0.3.44", optional = true }

README.md

@@ -8,6 +8,16 @@ model. It uses state reduction techniques to avoid combinatorial explosion.
 
 [Documentation](https://docs.rs/loom)
 
+## Overview
+
+Loom is an implementation of techniques described in [CDSChecker: Checking
+Concurrent Data Structures Written with C/C++ Atomics][cdschecker]. It is a
+library for writing unit tests where all possible thread interleavings are
+checked. It also is check all possible atomic cell behaviors and validate
+correct access to `UnsafeCell`.
+
+[cdschecker]: http://demsky.eecs.uci.edu/publications/c11modelcheck.pdf
+
 ## Getting started
 
 To use `loom`, first add this to your `Cargo.toml`:
@@ -50,32 +60,164 @@ fn buggy_concurrent_inc() {
 }
 ```
 
-## Overview
+## Usage
 
-Loom is an implementation of techniques described in [CDSChecker: Checking
-Concurrent Data Structures Written with C/C++ Atomics][cdschecker].
+Currently, using Loom comes with a bit of friction. Libraries must be written to
+be Loom-aware, and doing so comes with some boilerplate. Over time, the friction
+will be removed.
 
-[cdschecker]: http://demsky.eecs.uci.edu/publications/c11modelcheck.pdf
+The following provides a brief overview of how to usage Loom as part of the
+testing workflow of a Rust crate.
+
+### Structuring tests
+
+When running Loom tests, the Loom concurrency types must be used in place of the
+`std` types. However, when **not** running loom tests, the `std` should be used.
+This means that library code will need to use conditional compilation to decide
+which types to use.
+
+It is recommended to use a `loom` cfg flag to signal using the Loom types. Then,
+when running Loom tests, include `RUSTFLAGS="--cfg loom"` as part of the
+command.
+
+One strategy is to create module in your crate named `sync` or any other name of
+your choosing. In this module, list out the types that need to be toggled
+between Loom and `std`:
+
+```rust
+#[cfg(loom)]
+pub(crate) use loom::sync::atomic::AtomicUsize;
+
+#[cfg(not(loom))]
+pub(crate) use std::sync::atomic::AtomicUsize;
+```
+
+Then, elsewhere in the library:
+
+```rust
+use crate::sync::AtomicUsize;
+```
+
+### Handling Loom API differences.
+
+If your library must use Loom APIs that differ from `std` types, then the
+library will be required to implement those APIs for `std`. For example, for
+`UnsafeCell`, in the library's source, add the following:
+
+```rust
+#![cfg(not(loom))]
+
+#[derive(Debug)]
+pub(crate) struct UnsafeCell<T>(std::cell::UnsafeCell<T>);
+
+impl<T> UnsafeCell<T> {
+    pub(crate) fn new(data: T) -> UnsafeCell<T> {
+        UnsafeCell(std::cell::UnsafeCell::new(data))
+    }
+
+    pub(crate) fn with<R>(&self, f: impl FnOnce(*const T) -> R) -> R {
+        f(self.0.get())
+    }
+
+    pub(crate) fn with_mut<R>(&self, f: impl FnOnce(*mut T) -> R) -> R {
+        f(self.0.get())
+    }
+}
+```
+
+### Running Loom tests
+
+Loom tests must be run separately, with `RUSTFLAGS="--cfg loom"` specified. For
+example, if the library includes a test file: `tests/loom_my_struct.rs` that
+includes tests with `loom::model`, then run the following command:
 
-### Thread ordering
+```
+RUSTFLAGS="--cfg loom" cargo test --test loom_my_struct
+```
 
-TODO
+#### Handling large models
 
-### Atomics
+By default, Loom runs an **exhaustive** model. All possible execution paths are
+checked. Loom's state reduction algorithms significantly reduce the state space
+that must be explored, however, complex models can still take **significant**
+time. There are two strategies to deal with this.
 
-In the C++11 memory model, stores to a single atomic cell are totally ordered.
-This is the modification order. Loom permutes the modification order of each
-atomic cell within the bounds of the coherence rules.
+The first strategy is to run loom tests with `--release`. This will greatly
+speed up execution time.
 
+The second strategy is to **not** run an exhaustive check. Loom is able to set a
+thread pre-emption bound. This means that Loom will check all possible
+executions that include **at most** `n` thread pre-emptions. In practice,
+setting the thread pre-emption bound to 2 or 3 is enough to catch most bugs.
 
-## Limitations
+To set the thread pre-emption bound, set the `LOOM_MAX_PREEMPTIONS` environment
+variable when running tests. For example:
 
-While already very useful, loom is in its early stages and has a number of
-limitations.
+```
+LOOM_MAX_PREEMPTIONS=3 RUSTFLAGS="--cfg loom" cargo test --test loom_my_struct
+```
 
-* Execution is slow (#5).
-* The full C11 memory model is not implemented (#6).
-* No fence support (#7).
+### Debugging failed tests
+
+Loom's deterministic execution helps with debugging. The specific chain of
+events leading to a test failure can be isolated.
+
+When a loom test fails, the first step is to isolate the exact execution path
+that resulted in the failure. To do this, Loom is able to output the execution
+path to a file. Two environment variables are useful for this process:
+
+- `LOOM_CHECKPOINT_FILE`
+- `LOOM_CHECKPOINT_INTERVAL`
+
+The first specifies the file to write to and read from. The second specifies how
+often to write to the file. If the execution fails on the 10,000,000th
+permutation, it is faster to write to a file every 10,0000 iterations instead of
+every single one.
+
+To isolate the exact failing path, run the following commands:
+
+```
+LOOM_CHECKPOINT_FILE=my_test.json [other env vars] \
+    cargo test --test loom_my_struct [failing test]
+```
+
+Then, the following:
+
+```
+LOOM_CHECKPOINT_INTERVAL=1 LOOM_CHECKPOINT_FILE=my_test.json [other env vars] \
+    cargo test --test loom_my_struct [failing test]
+```
+
+The test should fail on the first permutation, effectively isolating the failure
+scenario.
+
+The next step is to enable additional log output. Again, there are some
+environment variables for this:
+
+- `LOOM_LOG`
+- `LOOM_LOCATION` (nightly Rust only)
+
+The first environment variable, `LOOM_LOG`, outputs a marker on every thread switch.
+This helps with tracing the exact steps in a threaded environment that results
+in the test failure.
+
+The second, `LOOM_LOCATION`, enables location tracking. This includes additional
+information in panic messages that helps identify which specific field resulted
+in the error. To enable this, `RUSTFLAGS="--cfg loom_nightly"` must also be
+specified.
+
+Put together, the command becomes (yes, we know this is not great... but it
+works):
+
+```
+LOOM_LOG=1 \
+    LOOM_LOCATION=1 \
+    LOOM_CHECKPOINT_INTERVAL=1 \
+    LOOM_CHECKPOINT_FILE=my_test.json \
+    RUSTFLAGS="--cfg loom --cfg loom_nightly" \
+    [other env vars] \
+    cargo test --test loom_my_struct [failing test]
+```
 
 ## License