General Overview

[afonsolage]

Introduction

In this discussion, I'll write some thoughts on how to create an interaction between app and module.

Note that those thoughts may not be connected to each order in the order they appear, but I'll try my best to keep short and organized.

Note also that I assume that you already know Bevy and it's terminology.

Definitions

• App is where the Bevy App is built and ran;
• Mod is the wasm module which is compiled, loaded and executed by App;
• Host is the wasm runtime. On native builds, it refers to a wasm runtime, like wasmtime or wasmer. On web builds, it refers to JavaScript runtime of the browser, like V8 or SpiderMonkey;
• Guest is any wasm module which is executed by Host. When running native builds, it refers to Mod. When running on web builds, it may refer to App or Mod, depending of the context;
• API is the protocol defined to allow the communication between App and Mod;
• Mod Asset is the raw binary data of a compiled wasm module;
• Mod Instance is an executable instance of a Mod Asset;
• Mod System a system exported from a Mod Instance;
• Mod Run Criteria is a run criteria which may be informed by any Mod System that must be evaluated to true in order for the Mod System to be ran;
• Contex is the data required by any Mod System to run;
• Context Switching means switch execution from Host to Guest or vice-versa, either by calling a function or returning from a previous function call. This is an analogy with Context Switch of threads.

Native and Web targets

One of the biggest challenges of this project is to enable seamless modding in either native or web (and in the future, mobile) platforms. The main reason why it is challenging, is because when targeting web (wasm32-unknown-unkown) the App is a Guest, since it's running inside the browser, and the JS runtime of the browser is the Host. On the other hand, when running on native platform, the App is also the Host and the Mod is also the Guest.

To summarize, always keep in mind that:

• Native: Host App <-> Guest Mod;
• Web: Guest App <-> Host Browser <-> Guest Mod;

App and Mod interaction overview

App will use exclusive_systems to run Mod System, since it's not possible at moment to add or remove dynamic systems.

1. App loads and compile¹ Mod Asset;
2. App instantiate a new Mod Instance;
3. App calls __wabi_alloc on Mod Instance to allocate it's memory buffer;
4. App calls __wabi_init on Mod Instance to initialize any internal state;
5. App calls __wabi_systems on Mod Instance to get all systems which will be run;
6. On each Stage, App:
   1. Checks for any exported Mod System to be ran on current stage;
   2. Prepares a Context for each Mod System;
   3. Run all Mod System with appropriated Context;
   4. Apply any modifications made on Context to the World;

¹ _{Even tho the wasm is already compiled, it is compiled to a bytecode format. Before running it, the Host needs to do some platform specific optimizations.}

App and Mod data sharing

Since Mod runs in a sand boxed environment, it's not possible to share memory from Host to Guest. In order to share data, a WebAssembly.Memory must be used.

When a Mod Instance is instantiated, App asks Mod to allocate a fixed size buffer and returns a pointer for this allocated buffer. This buffer will be allocated for the entire lifetime of the Mod, to avoid memory corruption.

Since App has control over which WebAssembly.Memory was given to each Mod Instance, when App needs to share some data with Mod, it writes the data on WebAssembly.Memory buffer offsetted by the returned pointer address, which is relative to the memory buffer. After writing, App asks for Mod to read the buffer, passing the length of the written data. That way, Mod can safely read the data.

When Mod needs to share some data with App, a similar approach is used, whereas Mod writes the data on allocated buffer and than send the written length to the App, which in turns reads from the memory buffer, using the previous returned pointer offset and the given length.

Note that due to this workflow, the Mod should not relay on the memory buffer to run any logic, since a simple log call may overwrite the buffer content, instead, before running any function call by the App, the Mod must make use of the data, making a copy of the relevant data.

Mod definition

Each Mod must export at least those functions, in order to interact with App:

• __wabi_alloc which will allocate an internal buffer to share data between App and Mod. This will be called before any other Mod function;
• __wabi_init which will initialize any internal state needed by the module. This will be called only once, after __wabi_alloc_;
• __wabi_systems which will describe what systems should be run. This will be called at the beginning of each Bevy event loop, to allow systems being enabled or disabled;

Each Mod must import the following Host functions:

• __wabi_send_action which will request the App to execute some action;

Mod Run Criteria

Since there is some overhead involved when Context Switching, it is mandatory for the App to only run Mod System when it is absolutely needed. To do so, a Mod System can inform one or more Mod Run Criteria s that must evaluate to true in order for the system to be ran.

These are the initial planned _Mod Run Criteria_s:

• ComponentAdded
• ComponentChanged
• ComponentRemoved
• ResourceAdded
• ResourceChanged
• ResourceRemoved
• QueryNotEmpty
• EventNotEmpty

Mod System definition

Each Mod must export at least one Mod System, which will be ran when the all _Mod Run Criteria_s are met, if specified. In order to export a Mod System a Mod must inform one or more:

• Exported system function name;
• Required Context;
• Zero or more _Mod Run Criteria_s;

The exported system function must accept one argument, which is the size of the supplied Context when the function will be ran.

Context

The Context used by Mod System will contains zero or more of the following Bevy types:

• Entity;
• Component;
• Resource;
• Commands;
• EventReader;
• EventWriter;

Types like Component and Resource will have a internal bit flag which will indicate if the component was updated or not, this will be used to avoid triggering change detection when not needed.

After each Mod System execution, the Context will be read by App to check for modifications, as follows:

• Components, if the changed bit flag is set;
• Resources, if the changed bit flag is set;
• EventWriter will have all it's items dispatched;
• Commands will have all it's commands flushed;

Context Switching

Since every Mod should be ran in a single thread, the Context Switching is quite easy to handle, since it'll follow a simple call-n-return function:

1. App calls the Mod System exported function supplying the required Context;
   1. Mod System starts to execute the logic based on given Context;
   2. If needed, Mod System calls __wabi_send_action and wait for return;
   3. App runs the callback function associated with __wabi_send_action and return a value, if needed;
   4. Mod System check the returned value and continue execution;
   5. Repeat 2, 3 and 4 until Mod System finishes it's execution;
2. App continues the execution;

Also there is a little overhead when switching context between App and Mod. Some initial benchmarks lead to the following results on release mode:

Function call

• (native) Host App to Guest Mod: ~1μs
• (web) Guest App to Host to Guest Mod: ~2μs

Share state (12k buffer size reading)

• (native) Host App to Guest Mod: ~2μs
• (web) Guest App to Host to Guest Mod: ~4μs

So we can rough estimate that a Context Switching where Mod System send some data to App and it in turns returns back some data, will have a ~6μs overhead on native and ~12μs on web.

Note that those are very rough estimations, and depending on the machine, platform, browser and data shared, may change A LOT, but for the purpose of this document, it will help to make some initial decisions.

[afonsolage]

Edit: RPC is out-of-scope for now. See #2

I'll keep the bellow text just for future reference.

---

API RPC

Another important aspect of any API for modding is the ability to forward calls from Mod to App, so Mod doesn't have to reimplement all the logic which already exists in App.

When dealing with wasm, the API RPC must be language agnostic, which means any language which is able to target wasm should be able to use the API RPC without any problems.

To understand the challenge here, take for instance an example:

let transform = Transform::from_xyz(10.0, 11.0, 12.0)
    .looking_at(Vec3::ZERO, Vec3::Y)
    .with_scale(Vec3::splat(1.0));

Since App and Mod doesn't share the same stack, one must use the memory buffer to transfer data between App and Mod as explained before. Breaking down the above sample code, this is what it will look like in the Guest:

1. Transform::from_xyz:

   1. Guest writes to the memory buffer a list of floats;
   2. Guest calls __wabi_send_action passing the written buffer length and an action code asking to create a new Transform from xyz;
   3. App reads the memory buffer with the given length;
   4. App dispatches the action code to a TransformProxy which will translate to the actual Transform::from_xyz;
   5. App gets the new crated Transform, serializes it and write to memory buffer, returning the length written on buffer;
   6. Guest reads the memory buffer and deserializes the Transform;

2. .looking_at(Vec3::ZERO, Vec3::Y)

   1. Guest serializes the Transform, Vec3::ZERO and Vec3::Y;
   2. Guest writes the serialized data to memory buffer;
   3. Guest calls __wabi_send_action passing the written buffer length and an action code relative to Transform::looking_at;
   4. App reads the memory buffer with the given length;
   5. App dispatches the action code to a TransformProxy which will translate to the actual Transform::looking_at;
   6. App gets the new returned Transform, serializes it and write to memory buffer, returning the length written on buffer;
   7. Guest reads the memory buffer and deserializes the Transform;

3. Vec3::splat(1.0)

   1. Guest calls a internal function with will create a new Vec3 using the splat function;
   2. Guest writes to the memory buffer a float;
   3. Guest calls __wabi_send_action passing the written buffer length and an action code relative to Vec3::splat;
   4. App reads the memory buffer with the given length;
   5. App dispatches the action code to a Vec3Proxy which will translate to the actual Vec3::splat;
   6. App gets the new returned Vec3, serializes it and write to memory buffer, returning the length written on buffer;
   7. Guest reads the memory buffer and deserializes the Vec3;
   8. Guest returns the new created Vec3;

4. .with_scale(Vec3::splat(1.0))

   1. Guest serializes the Transform and the returned Vec3;
   2. Guest writes the serialized data to memory buffer;
   3. Guest calls __wabi_send_action passing the written buffer length and an action code relative to Transform::with_scale;
   4. App reads the memory buffer with the given length;
   5. App dispatches the action code to a TransformProxy which will translate to the actual Transform::with_scale;
   6. App gets the new returned Transform, serializes it and write to memory buffer, returning the length written on buffer;
   7. Guest reads the memory buffer and deserializes the Transform;

That's a lot of Contex Switching and memory read/writes, but there are a lot of optimizations that can be done also, for instance:

1. Cache objects when applying chained self functions;
2. Use zero-copy ser/de to avoid copying data around;
3. Batch actions, so the chained transformations happens only once on App;
4. Use multiple memory buffers to reduce the amount of copy;

But those optimizations is out of scope of this document, since it's very risk to assume the benefits of those optimizations without proper testing and benchmarking, which will be done after there is a MVP working.

Proxy types

There should be two kinds of proxies:

• ModProxy which will forward the call to the App, providing a proper function code to map to a AppProxy;
• AppProxy which will receive a function code and forward for the actual underlying type function;

ModProxy

A ModProxy is a mapping between functions id and functions parameters and results. When a method is called, ModProxy will expect two types of parameters:

• A primitive type;
• Another ModProxy type;

Then it'll serialize the parameters and forward the call to __wabi_proxy_call passing the serialized data length, ProxyTypeId and ProxyFunctionId`.

AppProxy

Each AppProxy will have a underlying type T: Reflect, a ProxyTypeId and for each public function, a ProxyFunctionId, which is unique within an AppProxy. Both ProxyTypeId and ProxyFunctionId are u32 since it can be passed as function argument, avoiding using the memory buffer when requesting a RPC call;

When a dispatch action is request, AppProxy will deserialize the given Reflect dynamic object, mapping to the correct underlying function arguments and then call it, If the underlying function uses &mut self, serialize the underlying type and the returned value; if any, otherwise, only serialize the returned value.

[linonetwo]

How does this solution different from extism's solution, can this be built on extism?