Skip to content

Commit

Permalink
docs(s2n-quic): update CI docs (#2297)
Browse files Browse the repository at this point in the history
* docs(s2n-quic): update CI docs

* update heading

* update cloudfront links to use new latest alias
  • Loading branch information
WesleyRosenblum authored Aug 22, 2024
1 parent 3148138 commit a753efb
Show file tree
Hide file tree
Showing 2 changed files with 58 additions and 7 deletions.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

- a simple, easy-to-use API. See [an example](https://github.com/aws/s2n-quic/blob/main/examples/echo/src/bin/quic_echo_server.rs) of an s2n-quic echo server built with just a few API calls
- high configurability using [providers](https://docs.rs/s2n-quic/latest/s2n_quic/provider/index.html) for granular control of functionality
- extensive automated testing, including fuzz testing, integration testing, unit testing, snapshot testing, efficiency testing, performance benchmarking, interoperability testing and [more](https://github.com/aws/s2n-quic/blob/main/docs/ci.md)
- extensive automated testing, including fuzz testing, integration testing, unit testing, snapshot testing, efficiency testing, performance benchmarking, interoperability testing and [more](https://github.com/aws/s2n-quic/blob/main/docs/dev-guide/ci.md)
- integration with [s2n-tls](https://github.com/aws/s2n-tls), AWS's simple, small, fast and secure TLS implementation, as well as [rustls](https://crates.io/crates/rustls)
- thorough [compliance coverage tracking](https://github.com/aws/s2n-quic/blob/main/docs/ci.md#compliance) of normative language in relevant standards
- and much more, including [CUBIC congestion controller](https://www.rfc-editor.org/rfc/rfc8312.html) support, [packet pacing](https://www.rfc-editor.org/rfc/rfc9002.html#name-pacing), [Generic Segmentation Offload](https://lwn.net/Articles/188489/) support, [Path MTU discovery](https://www.rfc-editor.org/rfc/rfc8899.html), and unique [connection identifiers](https://www.rfc-editor.org/rfc/rfc9000.html#name-connection-id) detached from the address
Expand Down
63 changes: 57 additions & 6 deletions docs/dev-guide/ci.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,61 @@
# s2n-quic Continuous Integration

`s2n-quic` runs many tests on [each pull request](https://github.com/aws/s2n-quic/actions/workflows/ci.yml?query=event%3Apull_request) and merge into [main](https://github.com/aws/s2n-quic/actions/workflows/ci.yml?query=branch%3Amain). This ensures each change is thoroughly tested.
`s2n-quic` executes a comprehensive suite of tests, linters, benchmarks, simulations and other checks on [each pull request](https://github.com/aws/s2n-quic/actions/workflows/ci.yml?query=event%3Apull_request) and merge into [main](https://github.com/aws/s2n-quic/actions/workflows/ci.yml?query=branch%3Amain). Information about these checks is provided below.

## Tests

`s2n-quic` defines many tests that can be executed with `cargo test`. These tests include unit, integration, snapshot, property, and fuzz tests.
`s2n-quic` defines many tests that can be executed with `cargo test`. These tests, described below, are executed across a variety of operating systems, architectures, and Rust versions.

#### Unit Tests

Unit tests validate the expected behavior of individual components of `s2n-quic`. Typically, unit tests will be located in a `tests` module at the end of the file being tested. When there are a significant number of unit test functions for given component, the `tests` module may be located in a separate file.

#### Integration Tests

`s2n-quic` integration tests use the public API to validate the end-to-end behavior of the library under specific scenarios and configurations. Integration tests are located in the top-level `s2n-quic` crate, in the [tests module](https://github.com/aws/s2n-quic/tree/main/quic/s2n-quic/src/tests).

#### Snapshot Tests

Snapshot tests use [insta](https://crates.io/crates/insta) to assert complex output remains consistent with expected references values. In `s2n-quic`, snapshot tests are typically constructed in one of two ways:
* using the `insta::assert_debug_snapshot` macro to compare the `Debug` representation of an instance to the snapshot
* using the `event::testing::Publisher::snapshot()` event publisher to assert against a snapshot of events emitted by the `s2n-quic` [event framework](https://docs.rs/s2n-quic/latest/s2n_quic/provider/event/trait.Event.html)

#### Property Tests

Property tests assert that specific properties of the output of functions and components are upheld under a variety of inputs. `s2n-quic` uses the [Bolero property-testing framework](https://camshaft.github.io/bolero/introduction.html) to execute property-testing with multiple fuzzing engines as well as the [Kani Rust Verifier](https://model-checking.github.io/kani/). For more details on how Bolero and Kani are used together in `s2n-quic`, see the following blog posts from the [Kani Rust Verifier Blog](https://model-checking.github.io/kani-verifier-blog/):
* [From Fuzzing to Proof: Using Kani with the Bolero Property-Testing Framework](https://model-checking.github.io/kani-verifier-blog/2022/10/27/using-kani-with-the-bolero-property-testing-framework.html)
* [How s2n-quic uses Kani to inspire confidence](https://model-checking.github.io/kani-verifier-blog/2023/05/30/how-s2n-quic-uses-kani-to-inspire-confidence.html)

#### Fuzz Tests

Fuzz tests provide large amounts of varied input data to assert `s2n-quic` behaves as expected regardless of the input. Fuzz testing in `s2n-quic` comes in three flavors:
* Component-level fuzz testing using the [Bolero property-testing framework](https://camshaft.github.io/bolero/introduction.html). These tests generate a corpus of inputs that is included in the `s2n-quic` repository to allow the fuzz tests to be replayed when executing `cargo test`.
* End-to-end QUIC protocol-level fuzzing using [quic-attack](https://github.com/aws/s2n-quic/blob/main/scripts/quic-attack/README.md). `quic-attack` is a collection of features that collectively turn `s2n-quic` into an online QUIC protocol fuzzer. It allows incoming and outgoing datagrams and packets, as well as the port number on incoming datagrams, to be intercepted and manipulated.
* UDP protocol-level fuzzing using [udp-attack](https://github.com/aws/s2n-quic/tree/main/tools/udp-attack). `udp-attack` generates random UDP packets and transmits them to `s2n-quic` to catch issues with packet handling.

#### Concurrency Permutation Tests

[loom](https://crates.io/crates/loom) is used to validate the behavior of concurrent code in `s2n-quic`. `loom` executes concurrent code using a simulation of the operating system scheduler and Rust memory model to evaluate concurrent code under all possible thread interleavings.

## Interoperability

The [quic-interop-runner](https://github.com/marten-seemann/quic-interop-runner) defines a suite of test cases that ensure compatibility between QUIC implementations. `s2n-quic` publishes [a report](https://dnglbrstg7yg.cloudfront.net/latest/interop/index.html) with the results.

## Compliance

`s2n-quic` annotates source code with inline references to requirements in [IETF RFC](https://www.ietf.org/process/rfcs/) specifications. [Duvet](https://github.com/awslabs/duvet) is used to generate [a report](https://dnglbrstg7yg.cloudfront.net/latest/interop/index.html), which makes it easy to track compliance with each requirement.

## Simulations

A Monte Carlo simulation tool is used to execute thousands of randomized simulations of `s2n-quic` that vary one or more network variables, such as bandwidth, jitter, and round trip time. The [report output](https://dnglbrstg7yg.cloudfront.net/latest/sim/index.html) provides a visual representation of the relationship between the input variables and overall performance.

A loss recovery simulation tool plots the growth of the congestion window over time under various simulated loss scenarios and publishes the results in [a report](https://dnglbrstg7yg.cloudfront.net/latest/recovery-simulations/index.html) to visualize changes to the congestion control algorithm.

## Performance & Efficiency Profiling

[Flame graphs](https://www.brendangregg.com/flamegraphs.html) are generated and published in a [report](https://dnglbrstg7yg.cloudfront.net/latest/perf/index.html) to visualize stack traces produced under a variety of data transfer scenarios.

[dhat](https://crates.io/crates/dhat) performs heap profiling and publishes the results in [a report](https://dnglbrstg7yg.cloudfront.net/dhat/dh_view.html?url=/latest/dhat/dhat-heap.json).

## Clippy

Expand All @@ -14,10 +65,10 @@

[rustfmt](https://github.com/rust-lang/rustfmt) ensures code is consistently formatted.

## Interop
## Miri

The [quic-interop-runner](https://github.com/marten-seemann/quic-interop-runner) defines many test cases that ensure many of the 3rd party QUIC implementations are compatible. `s2n-quic` publishes [a report](https://dnglbrstg7yg.cloudfront.net/08c33571ee8679775e810303f65c96c1d48e270d/interop/index.html) with the results.
[Miri](https://github.com/rust-lang/miri) detects [undefined behavior](https://doc.rust-lang.org/reference/behavior-considered-undefined.html) and memory leaks.

## Compliance
## Code Coverage

`s2n-quic` annotates source code with inline references to requirements in design documents and RFCs. [A report](https://dnglbrstg7yg.cloudfront.net/08c33571ee8679775e810303f65c96c1d48e270d/compliance.html#/) is then generated, which makes it easy to track compliance with each requirement.
[LLVM source-based code coverage](https://llvm.org/docs/CommandGuide/llvm-cov.html) measures how much of `s2n-quic` code is executed by tests. `s2n-quic` publishes [a report](https://dnglbrstg7yg.cloudfront.net/latest/coverage/index.html) with the results.

0 comments on commit a753efb

Please sign in to comment.