Thoughts on user provided transforms

[rukai]

There has been a lot of work/discussion towards scriptable or plugin-driven transforms.
But this sounds like it could be the wrong approach.
The transforms have the same requirements as shotover itself which needs to be:

• extremely fast
• correct
• cannot crash or otherwise drop data

The perfect candidate for this is rust, we obviously believe that because shotover itself is written in rust.

Even so we could still use rust compiled to wasm or rust dylib plugins.
That way developers can use rust or if they really really want they can use another language.

But doing so has several disadvantages:

• poor versioning - by using a dylib or wasm we step outside of cargo's robust semantic versioning and dependency management.
• adds complexity to the implementation resulting in a large maintenance burden and surface area for bugs
• when writing plugins in another language:
  • mismatch between the provided API and the way code is written in the plugin language - can be reduced by bindings that provide abstractions suitable for the plugin language - but again, requires effort to implement + maintain
• when writing plugins in rust:
  • for dylib we lose our type safety because we have to expose an FFI API
  • for wasm we have to serialize and deserialize our types into bincode as we can only communicate over an array of bytes between the host and wasm

So how can we give shotover the flexibility it needs without plugins/wasm?

We can expose shotover as a library crate exposing a builder pattern API.
Then the user of shotover implements their own binary crate using this shotover library.

e.g. They could run vanilla shotover with just the following

fn main() {
    Shotover::new().run();
}

This would be a very batteries included approach. run would be responsible not just for running shotover but it would also start the async runtime, read the config from disk, start the logger etc.

We could still provide a binary release of shotover by compiling this simple example. This binary release could do everything except implement custom transforms.

Then to extend shotover with their own transforms, developers implement a Transform trait and register it like this:

struct MyTransform { }

impl Transform for MyTransform {
    ...
}

fn main() {
    Shotover::new()
        .register_transform(MyTransform::new())
        .run();
}

As a bonus we could allow bundling config into the binary like this:

fn main() {
    Shotover::new()
        // config guaranteed to exist at compile time
        .set_config(include_str!("config.yaml"));
        .run();
}

or we could even go so far as to have the config constructed in our rust builder pattern instead of yaml:

fn main() {
    Shotover::new()
        // config structure gauranteed to be correct at compile time
        .add_source(Source { name: "foo", batch_size_hint: 100, listen_addr: "127.0.0.1:6379");
        .add_chain(Config {
            name: "bar",
            sources: vec!("foo"),
            transforms: vec!(
                MyTransform::new(),
                StandardTransform::new(TransformConfig { contact_points: vec!("127.0.0.1:2920", "127.0.0.1:2921") })
            )
        });
        .run();
}

This particular idea is a little out there. I think just having a yaml file is more readable.

An example of this sort of "rust API instead of a binary" approach can be seen in a personal project I built, which provides a rust API for compiling gameboy asm: https://github.com/rukai/ggbasm
And of course rust webservers all use a similar approach. e.g. https://github.com/SergioBenitez/Rocket