New wasm transform

[binarylogic]

A transform that would support the web assembly runtime allowing users to program transforms in a variety of languages.

This issue definitely needs more discussion before beginning work, but I wanted to get something in place.

[Hoverbear]

I feel like this is more than a transform.

If we're going to go through the effort of including and defining bindings to a WASM runtime, we should consider making it a full blown 'plugin system'.

[lukesteensen]

Agreed, I think there's a whole spectrum of abilities we could unlock with wasm. That being said, transforms are likely the simplest to get started with.

[Hoverbear]

Since I'm exploring the WASM related topic I'll assign this to me.

[yarncraft]

Working at an IoT company, I really endorse the idea of being able to write transformers in a wide spectrum of languages! Leveraging WASM this way would certainly increase Vector's adoption rate 👍

[Hoverbear]

@yarncraft Did you have any use cases we might want to consider in our designs? :)

[yarncraft]

@yarncraft Did you have any use cases we might want to consider in our designs? :)

Being able to write functional stream processing pipelines in languages such as Rust, Go or Javascript/Typescript (https://github.com/cujojs/most) would be great to start with since it gives more flexibility when working with JSON messages. We also need solutions that implement (or give the ability to implement) rolling windows etc.

[yarncraft]

A second use case of WASM would be to have the ability to train ML models on streaming batches and use them for prediction purposes. (https://juggernaut.rs/demo/)