Initial WebAssembly subsystem #12666

rockwotj · 2023-08-08T18:42:35Z

Add the initial Wasm engine to Redpanda.

Right now we are using WasmEdge as the underlying engine but the engine that is used is abstracted away behind some interfaces so that we can easily extend it to other runtimes. I have a branch that shows this interface works fine for wasmtime as well.

Generally this component exists in isolation from the rest of the system, but this new subsystem will eventually be consumed by the transform subsystem.

Major pieces:

ffi: A set of classes/functions that is used to abstract away exposing host functions into a Wasm engine
wasi: A (mostly stub) implementation of the WASI standard. This allows user code to write to the transform subsystem logs, and access environment variables
wasmedge: The actual implementation of our wasm engine abstraction using wasmedge
transform_module: An implementation of the ABI contract, it includes host functions that we expose into the Wasm module as well as helpers for calling into the runtime correctly
schema_registry: A wrapper around the pandaproxy::schema_registry subsystem for usage within our host module
schema_registry_module: A Wasm module that exposes schema registry to user transforms

The guest side of the ABI is implemented in #12322, and there is a video overview of how the ABI contract works here.

Wasm powered Data Transforms will be available in the community edition of Redpanda hence this code all being written under the BSL

NOTE: ~~Right now I am checking in the .wasm test fixtures, but after #12322 goes in I will hook that into the build system and enable more tests.~~ I hooked up the tests into the build system.

Backports Required

Release Notes

none

mmaslankaprv · 2023-08-09T11:05:43Z

src/v/ssx/work_queue.h

+#include "seastarx.h"
+#include "ssx/future-util.h"
+#include "utils/mutex.h"
+
+#include <seastar/core/abort_source.hh>
+#include <seastar/core/future-util.hh>
+#include <seastar/core/future.hh>
+#include <seastar/core/shared_ptr.hh>
+#include <seastar/core/thread.hh>
+#include <seastar/util/noncopyable_function.hh>
+
+#include <exception>
+#include <type_traits>
+#include <utility>


nit: clean up includes

Thanks done.

mmaslankaprv · 2023-08-09T11:13:58Z

src/v/model/transform.h

+    // Each transform revision has a UUID, which is the key for the wasm_binary
+    // topic and can be used to uniquely identify this revision.
+    uuid_t uuid;
+    // The offset of the commmitted WASM source in the wasm_binary topic.


Thanks fixed.

mmaslankaprv · 2023-08-09T11:54:52Z

src/v/wasm/wasmedge.cc

+        vlog(
+          wasm_log.trace,
+          "Transforming batch: {}",
+          batch.header().record_count);


maybe we can log the whole header here ?

Certainly 😄

mmaslankaprv · 2023-08-09T11:58:04Z

src/v/wasm/wasmedge.cc

+        std::vector<ss::sstring> args{_meta.name()};
+        absl::flat_hash_map<ss::sstring, ss::sstring> env = _meta.environment;
+        env.emplace("REDPANDA_INPUT_TOPIC", _meta.input_topic.tp());
+        env.emplace("REDPANDA_OUTPUT_TOPIC", _meta.output_topics.begin()->tp());


do i understand that correctly that for now we support only one output topic ?

Correct. Added a comment here.

mmaslankaprv · 2023-08-09T11:59:43Z

src/v/wasm/wasmedge.cc

+          args, env, _logger);
+        register_wasi_module(wasi_module.get(), wasmedge_wasi_module);
+
+        (void)_worker;


this line seems like not needed ?

thanks removed, we'll add compilation on an alien thread to later.

mmaslankaprv · 2023-08-09T12:05:13Z

src/v/wasm/probe.h

+// Per transform probe
+class transform_probe {
+public:
+    using hist_t = log_hist_public;


is this histogram light enough to be used per transform ?

sizeof(log_hist_public) == 160 so I think so? If we have 1000 different transforms on a shard for 1000 partitions then we get 160kB which feels small enough. Happy to just switch this to a counter and have a stat for CPU utilization instead?

src/v/wasm/api.h

BenPope · 2023-08-09T13:05:22Z

cmake/oss.cmake.in

@@ -324,3 +324,18 @@ ExternalProject_Add(protobuf
    -Dprotobuf_BUILD_TESTS=OFF
    ${common_cmake_args}
 )
+
+ExternalProject_Add(WasmEdge


Could you add an entry to: https://github.com/redpanda-data/redpanda/blob/dev/licenses/third_party.md

👍 done - also removed v8

rockwotj · 2023-08-16T03:31:43Z

Force push: Added named constants to redpanda Wasm module error numbers

dotnwat · 2023-08-16T03:34:56Z

Maybe it's worth building a malformed guest so we can verify the error paths are correctly handled? I think that should come in a followup.

great idea, and follow is reasonable

This will be used in the Wasm engine integration and is a useful abstraction for fibers to interact with a running seastar thread. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

Right now util depends on json, so this prevents models from depending on util Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

rockwotj · 2023-08-16T17:38:00Z

Force push: Added tests for FFI helpers, removed shared library cycle with json -> model -> utils.

This commit defines a ABI for wasm modules to perform transforms within redpanda. There are also helpers for custom host bindings that are defined in an engine agnostic way. See redpanda-data#12322 for the corresponding guest side bindings as well as a video overview of the ABI contract. Additionally, a WASI module is defined, which supports a subset of the WASI standard. Namely, environment variables and writing to std{out,err} is supported. There are currently bindings to both WasmEdge and wasmtime, although the wasmtime integration is mostly experimental and can easily crash the redpanda process (see scylladb/scylladb#14163). Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

tinygo doesn't yet use wasm/wasi1p build tags like upstream go, so just switch to using tinygo completely as the build tag. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

Now that we're building assets as part of CMake Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

rockwotj · 2023-08-17T03:06:54Z

Force pushed to fix a test that changed after the seastar upgrade

These functions where already defined in redpanda-data#12666 and their encoding. We also add a stub version of the ABI for IDEs that don't use the tinygo build tags (and we can build/test this package in standard go). Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

rockwotj requested review from BenPope and a team as code owners August 8, 2023 18:42

rockwotj requested review from andrewhsu and removed request for a team August 8, 2023 18:42

rockwotj mentioned this pull request Aug 8, 2023

Initial Wasm Golang Transform SDK #12322

Merged

7 tasks

github-actions bot added area/build area/redpanda labels Aug 8, 2023

rockwotj mentioned this pull request Aug 8, 2023

Initial Wasm Engine #12427

Closed

7 tasks

rockwotj force-pushed the rockwood/initial-wasm-engine branch from 0689bae to 673bdd0 Compare August 8, 2023 18:49

rockwotj requested review from mmaslankaprv and dotnwat August 8, 2023 18:49

rockwotj force-pushed the rockwood/initial-wasm-engine branch from 673bdd0 to 1abf895 Compare August 8, 2023 19:55

mmaslankaprv reviewed Aug 9, 2023

View reviewed changes

BenPope reviewed Aug 9, 2023

View reviewed changes

src/v/wasm/api.h Show resolved Hide resolved

BenPope self-requested a review August 9, 2023 12:31

BenPope reviewed Aug 9, 2023

View reviewed changes

BenPope self-requested a review August 9, 2023 13:07

rockwotj force-pushed the rockwood/initial-wasm-engine branch from 1abf895 to 1bdd788 Compare August 9, 2023 20:00

rockwotj requested review from emaxerrno and dswang as code owners August 9, 2023 20:00

rockwotj force-pushed the rockwood/initial-wasm-engine branch from 1bdd788 to a910092 Compare August 9, 2023 20:01

rockwotj requested a review from mmaslankaprv August 9, 2023 20:02

rockwotj force-pushed the rockwood/initial-wasm-engine branch 3 times, most recently from 4460b5b to 059366b Compare August 10, 2023 02:54

rockwotj added 4 commits August 16, 2023 09:56

Add a work queue based on ss::thread

2af1c66

This will be used in the Wasm engine integration and is a useful abstraction for fibers to interact with a running seastar thread. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

Remove json library dependencies on model

3d6a4b2

Right now util depends on json, so this prevents models from depending on util Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

model: Add transform metadata to models

fc64e39

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

Add WasmEdge to open source build

cd85aec

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

rockwotj force-pushed the rockwood/initial-wasm-engine branch from d3d4f61 to c9d1d7b Compare August 16, 2023 17:37

rockwotj requested a review from dotnwat August 16, 2023 17:38

This was referenced Aug 16, 2023

wasm: Add tests for wasm transform guest with invalid ABI usage #12849

Open

Extract dependent_false helper #12850

Closed

rockwotj force-pushed the rockwood/initial-wasm-engine branch from c9d1d7b to 6c0c317 Compare August 16, 2023 20:32

dotnwat previously approved these changes Aug 16, 2023

View reviewed changes

rockwotj added 6 commits August 16, 2023 17:35

Gitignore .wasm files

968f82f

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

Add testdata tinygo transforms

55290a2

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

cmake: Add tinygo compiler to 3rd party deps

7913ffc

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

Fix transform ABI build tags

e40e3ca

tinygo doesn't yet use wasm/wasi1p build tags like upstream go, so just switch to using tinygo completely as the build tag. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

Enable all wasm subsystem tests

e44376a

Now that we're building assets as part of CMake Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

rockwotj dismissed dotnwat’s stale review via e44376a August 16, 2023 22:36

rockwotj force-pushed the rockwood/initial-wasm-engine branch from 6c0c317 to e44376a Compare August 16, 2023 22:36

rockwotj requested a review from dotnwat August 17, 2023 03:06

dotnwat approved these changes Aug 17, 2023

View reviewed changes

dotnwat merged commit 01a317b into redpanda-data:dev Aug 17, 2023

rockwotj mentioned this pull request Aug 28, 2023

transform-sdk: Introduce sr package #13049

Merged

7 tasks

rockwotj deleted the rockwood/initial-wasm-engine branch August 28, 2023 19:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial WebAssembly subsystem #12666

Initial WebAssembly subsystem #12666

rockwotj commented Aug 8, 2023 •

edited

Loading

mmaslankaprv Aug 9, 2023

rockwotj Aug 9, 2023

mmaslankaprv Aug 9, 2023

rockwotj Aug 9, 2023

mmaslankaprv Aug 9, 2023

rockwotj Aug 9, 2023

mmaslankaprv Aug 9, 2023

rockwotj Aug 9, 2023

mmaslankaprv Aug 9, 2023

rockwotj Aug 9, 2023

mmaslankaprv Aug 9, 2023

rockwotj Aug 9, 2023

BenPope Aug 9, 2023

rockwotj Aug 9, 2023

rockwotj commented Aug 16, 2023

dotnwat commented Aug 16, 2023

rockwotj commented Aug 16, 2023 •

edited

Loading

rockwotj commented Aug 17, 2023

Initial WebAssembly subsystem #12666

Initial WebAssembly subsystem #12666

Conversation

rockwotj commented Aug 8, 2023 • edited Loading

Backports Required

Release Notes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rockwotj commented Aug 16, 2023

dotnwat commented Aug 16, 2023

rockwotj commented Aug 16, 2023 • edited Loading

rockwotj commented Aug 17, 2023

rockwotj commented Aug 8, 2023 •

edited

Loading

rockwotj commented Aug 16, 2023 •

edited

Loading