Skip to content

Commit

Permalink
feat: Procedural overhaul
Browse files Browse the repository at this point in the history 
Signed-off-by: Jonathan Woollett-Light <jcawl@amazon.co.uk>
  • Loading branch information
Jonathan Woollett-Light committed Jul 14, 2023
1 parent 6970de9 commit 61e3b3d
Show file tree
Hide file tree
Showing 23 changed files with 626 additions and 2,670 deletions.
24 changes: 0 additions & 24 deletions .buildkite/custom-tests.json
View file
Original file line number Diff line number Diff line change
@@ -1,29 +1,5 @@
{
"tests": [
{
"test_name": "build-gnu-remote_endpoint",
"command": "cargo build --release --features=remote_endpoint",
"platform": [
"x86_64",
"aarch64"
]
},
{
"test_name": "build-musl-remote_endpoint",
"command": "cargo build --release --features=remote_endpoint --target {target_platform}-unknown-linux-musl",
"platform": [
"x86_64",
"aarch64"
]
},
{
"test_name": "check-warnings-remote_endpoint",
"command": "RUSTFLAGS=\"-D warnings\" cargo check --features=remote_endpoint",
"platform": [
"x86_64",
"aarch64"
]
},
{
"test_name": "performance",
"command": "pytest -s rust-vmm-ci/integration_tests/test_benchmark.py",
Expand Down
18 changes: 4 additions & 14 deletions Cargo.toml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,22 +11,12 @@ edition = "2021"

[dependencies]
vmm-sys-util = "0.11.0"
libc = "0.2.39"
libc = { version = "0.2.39", features = ["extra_traits"] }

[dev-dependencies]
criterion = "0.3.5"

[features]
remote_endpoint = []
test_utilities = []
criterion = "0.5.1"
rand = "0.8.5"

[[bench]]
name = "main"
harness = false

[lib]
bench = false # https://bheisler.github.io/criterion.rs/book/faq.html#cargo-bench-gives-unrecognized-option-errors-for-valid-command-line-options

[profile.bench]
lto = true
codegen-units = 1
harness = false
179 changes: 31 additions & 148 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,160 +10,43 @@ mechanism for handling I/O notifications.

## Design

This crate is built around two abstractions:
- Event Manager
- Event Subscriber

The subscriber defines and registers an interest list with the event manager.
The interest list represents the events that the subscriber wants to monitor.

The Event Manager allows adding and removing subscribers, and provides
APIs through which the subscribers can be updated in terms of events in their
interest list. These actions are abstracted through the `SubscriberOps` trait.

To interface with the Event Manager, the Event Subscribers need to provide an
initialization function, and a callback for when events in the
interest list become ready. The subscribers can update their interest list
when handling ready events. These actions are abstracted through the
`EventSubscriber` and `MutEventSubscriber` traits. They contain the same
methods, but the former only requires immutable `self` borrows, whereas the
latter requires mutable borrows. Any type implementing `EventSubscriber`
automatically implements `MutEventSubscriber` as well.

A typical event-based application creates the event manager, registers
subscribers, and then calls into the event manager's `run` function in a loop.
Behind the scenes, the event manager calls into `epoll::wait` and maps the file
descriptors in the ready list to the subscribers it manages. The event manager
calls the subscriber's `process` function (its registered callback). When
dispatching the events, the event manager creates a specialized object and
passes it in the callback function so that the subscribers can use it to alter
their interest list.

![](docs/event-manager.png)
This crate offers an abstraction (`EventManager`) over [epoll](https://man7.org/linux/man-pages/man7/epoll.7.html) that
allows for more ergonomic usage with many file descriptors.

The `EventManager` allows adding and removing file descriptors with a callback closure. The
`EventManager` interest list can also be modified within these callback closures.

A typical event-based application:

1. Creates the `EventManager` (`EventManager::default()`).
2. Registers file descriptors with (`EventManager::add`).
3. Calls `EventManager::wait` in a loop.

Read more in the [design document](docs/DESIGN.md).

## Implementing an Event Subscriber

The event subscriber has full control over the events that it monitors.
The events need to be added to the event manager's loop as part of the
`init` function. Adding events to the loop can return errors, and it is
the responsibility of the subscriber to handle them.

Similarly, the event subscriber is in full control of the ready events.
When an event becomes ready, the event manager will call into the subscriber
`process` function. The subscriber SHOULD handle the following events which
are always returned when they occur (they don't need to be registered):
- `EventSet::ERROR` - an error occurred on the monitor file descriptor.
- `EventSet::HANG_UP` - hang up happened on the associated fd.
- `EventSet::READ_HANG_UP` - hang up when the registered event is edge
triggered.

For more details about the error cases, you can check the
[`epoll_ctl documentation`](https://www.man7.org/linux/man-pages/man2/epoll_ctl.2.html).


## Initializing the Event Manager

The `EventManager` uses a generic type parameter which represents the
subscriber type. The crate provides automatic implementations of
`EventSubscriber` for `Arc<T>` and `Rc<T>` (for any `T: EventSubscriber +?Sized`),
together with automatic implementations of `MutEventSubscriber` for `Mutex<T>`
and `RefCell<T>` (for any `T: MutEventSubscriber + ?Sized`). The generic type
parameter enables either static or dynamic dispatch.

This crate has no default features. The optional `remote_endpoint`
feature enables interactions with the `EventManager` from different threads
without the need of more intrusive synchronization.

## Examples

For closer to real life use cases, please check the examples in
[tests](tests).

### Basic Single Thread Subscriber

#### Implementing a Basic Subscriber

```rust
use event_manager::{EventOps, Events, MutEventSubscriber};
use vmm_sys_util::{eventfd::EventFd, epoll::EventSet};

use std::os::unix::io::AsRawFd;
use std::fmt::{Display, Formatter, Result};

pub struct CounterSubscriber {
event_fd: EventFd,
counter: u64,
}

impl CounterSubscriber {
pub fn new() -> Self {
Self {
event_fd: EventFd::new(0).unwrap(),
counter: 0,
}
}
}

impl MutEventSubscriber for CounterSubscriber {
fn process(&mut self, events: Events, event_ops: &mut EventOps) {
match events.event_set() {
EventSet::IN => {
self.counter += 1;
}
EventSet::ERROR => {
eprintln!("Got error on the monitored event.");
}
EventSet::HANG_UP => {
event_ops.remove(events).unwrap_or(
eprintln!("Encountered error during cleanup")
);
panic!("Cannot continue execution. Associated fd was closed.");
}
_ => {}
}
}

fn init(&mut self, ops: &mut EventOps) {
ops.add(Events::new(&self.event_fd, EventSet::IN)).expect("Cannot register event.");
}
}
```

#### Adding Subscribers to the Event Manager

```rust
struct App {
event_manager: EventManager<CounterSubscriber>,
subscribers_id: Vec<SubscriberId>,
}

impl App {
fn new() -> Self {
Self {
event_manager: EventManager::<CounterSubscriber>::new().unwrap(),
subscribers_id: vec![]
}
}

fn add_subscriber(&mut self) {
let counter_subscriber = CounterSubscriber::default();
let id = self.event_manager.add_subscriber(counter_subscriber);
self.subscribers_id.push(id);
}

fn run(&mut self) {
let _ = self.event_manager.run_with_timeout(100);
}
}
```
## Implementing an event

Like `epoll` a file descriptor only monitors specific events.

The events ars specified when calling `EventManager::add` with `vmm_sys_util::epoll::EventSet`.

When an event becomes ready, the event manager will call the file descriptors callback closure.

The `epoll` events `EPOLLRDHUP`, `EPOLLERR` and `EPOLLHUP` (which correspond to
`EventSet::READ_HANG_UP`, `EventSet::ERROR` and `EventSet::HANG_UP` respectively) are documented to
always report, even when not specified by the user.

> epoll_wait(2) will always report for this event; it is not
> necessary to set it in events when calling epoll_ctl().
*https://man7.org/linux/man-pages/man2/epoll_ctl.2.html*

As such it is best practice to always handle the cases where the `EventSet` passed to the file
descriptor callback closure is `EventSet::READ_HANG_UP`, `EventSet::ERROR` or `EventSet::HANG_UP`.

## Development and Testing

The `event-manager` is tested using unit tests, Rust integration tests and
performance benchmarks. It leverages
[`rust-vmm-ci`](https://github.com/rust-vmm/rust-vmm-ci) for continuous
`event-manager` uses [`rust-vmm-ci`](https://github.com/rust-vmm/rust-vmm-ci) for continuous
testing. All tests are run in the `rustvmm/dev` container.

More details on running the tests can be found in the
Expand Down
Loading

0 comments on commit 61e3b3d

Please sign in to comment.