Experimental Host Function Support (and rc4 release) #39

bhelx · 2024-01-03T19:23:54Z

This builds on my last PR and adds support for Host Functions. Currently the limitation is that host functions must have 1 i64 ptr param and 1 i64 ptr result. We enforce this through validation of the typescript interface file.

To enable host functions you just need to declare another interface extism:host/user:

declare module 'main' {
  export function greet(): I32;
}

declare module 'extism:host' {
  interface user {
    myHostFunction1(ptr: I64): I64;
    myHostFunction2(ptr: i64): i64;
  }
}

These needs to have this signature for now. We will work on making different function arities work next.

To use these you need to use Host.getFunctions():

const { myHostFunction1, myHostFunction2 } = Host.getFunctions()

Calling them is a similar process to other PDKs. You need to manage the memory with the Memory object and pass across an offset as the i64 ptr. Using the return value means dereferencing the returned i64 ptr from Memory.

  let msg = "Hello from js 1"
  let mem = Memory.fromString(msg)
  let offset = myHostFunction1(mem.offset)
  let response = Memory.find(offset).readString()
  if (response != "myHostFunction1: " + msg) {
    throw Error(`wrong message came back from myHostFunction1: ${response}`)
  }

See example in examples/host_funcs

bhelx · 2024-01-03T19:26:07Z

crates/cli/src/ts_parser.rs

+                                    .as_ident()
+                                    .unwrap()
+                                    .sym;
+                                Param::new(vn, typ)


I'm sure there is a nicer way to pull out all these values with some high level rust magic but this was what came naturally to me.

.github/workflows/publish.yml

bhelx · 2024-01-03T19:27:18Z

crates/core/src/globals.rs

+    let mem_obj = context.object_value()?;
+    mem_obj.set_property("fromBuffer", memory_from_buffer)?;
+    mem_obj.set_property("find", memory_find)?;
+    mem_obj.set_property("readBytes", read_bytes)?;


Going tot think about this interface a little bit more

Note: I think we should add a free as well. Need to reference the other PDK APIs and try to match it.

bhelx · 2024-01-03T19:28:13Z

crates/core/src/globals.rs

+            let data = args.get(0).ok_or(anyhow!("Expected data argument"))?;
+            if !data.is_array_buffer() {
+                bail!("Expected data to be an array buffer");
+            }
+            let data = data.as_bytes()?;
+            let m = extism_pdk::Memory::from_bytes(data)?;
+            let mut mem = HashMap::new();
+            let offset = JSValue::Int(m.offset() as i32);
+            let len = JSValue::Int(m.len() as i32);
+            mem.insert("offset".to_string(), offset);
+            mem.insert("len".to_string(), len);
+            Ok(JSValue::Object(mem))


I might move some more of this logic up into the Javascript shim layer. Thinking about it. Won't affect the interface though.

bhelx · 2024-01-03T19:28:47Z

examples/host_funcs/script.d.ts

@@ -0,0 +1,10 @@
+declare module 'main' {
+  export function greet(): I32;


Exports must have this signature

bhelx · 2024-01-03T19:29:02Z

examples/host_funcs/script.d.ts

+    myHostFunction1(ptr: I64): I64;
+    myHostFunction2(ptr: i64): i64;


Imports must have this signature, for now

bhelx · 2024-01-03T19:29:36Z

examples/host_funcs/script.js

+  let msg = "Hello from js 1"
+  let mem = Memory.fromBuffer((new TextEncoder().encode(msg)).buffer)
+  let ptr = myHostFunction1(mem.offset)
+  let response = new TextDecoder().decode(Memory.readBytes(ptr))
+  if (response != "myHostFunction1: " + msg) {
+    throw Error(`wrong message came back from myHostFunction1: ${response}`)
+  }


Going to add some wrappers on Memory to make this a little nicer

bhelx · 2024-01-03T19:40:05Z

crates/cli/src/main.rs

+    // let tmp_dir = "/tmp/derp";
+    // let core_path = PathBuf::from("/tmp/derp/core.wasm");
+    // let export_shim_path = PathBuf::from("/tmp/derp/export-shim.wasm");
+    // let import_shim_path = PathBuf::from("/tmp/derp/import-shim.wasm");
+    // let linked_shim_path = PathBuf::from("/tmp/derp/linked.wasm");


Had this for testing. will clean up or add a command flag for this.

bhelx · 2024-01-03T19:40:38Z

crates/cli/src/main.rs

+        bail!("wasm-merge failed. Couldn't merge export shim");
    }

-    remove_dir_all(tmp_dir)?;


I didn't realize that the temp dir gets deleted when the object gets dropped.

bhelx · 2024-01-03T19:42:05Z

crates/cli/src/main.rs

+        .arg("--enable-reference-types")
+        .arg("--enable-bulk-memory")


TODO: I need to verify this is going to work in non wasmtime runtimes

bhelx · 2024-01-03T19:42:55Z

crates/cli/src/main.rs

-        .arg(&opts.output)
+        .arg(&linked_shim_path)
        .spawn()?;
    let status = command.wait()?;


TODO: need a better error message for when wasm-merge can't be found on the machine

bhelx · 2024-01-04T22:13:25Z

Going to merge and get rc4 out now. Will be back next week with follow ups!

bhelx added 12 commits December 21, 2023 11:32

stash

209e3fd

stash

5dfe75b

works

3cabf94

stash

86c5092

fix host example

8a87715

Add python to CI

0dbe22b

pin to 1.74

0c7d469

pin to 1.73

ab47ab9

why is wasm32 target not here

4d07ec8

try extra params

030fbe7

add validation

f01baf6

init a logger for CLI

32da777

bhelx commented Jan 3, 2024

View reviewed changes

nilslice reviewed Jan 3, 2024

View reviewed changes

.github/workflows/publish.yml Outdated Show resolved Hide resolved

bhelx commented Jan 3, 2024

View reviewed changes

forgot to delete this

b72a1f5

bhelx commented Jan 3, 2024

View reviewed changes

we have to move forward with import shim even if there are none

11c303c

bhelx commented Jan 3, 2024

View reviewed changes

Add some more to the memory API

05a327b

chrisdickinson approved these changes Jan 4, 2024

View reviewed changes

bhelx merged commit 95787ff into main Jan 4, 2024

bhelx deleted the host-fn-support branch January 4, 2024 22:18

		@@ -0,0 +1,10 @@
		declare module 'main' {
		export function greet(): I32;

		myHostFunction1(ptr: I64): I64;
		myHostFunction2(ptr: i64): i64;

		.arg("--enable-reference-types")
		.arg("--enable-bulk-memory")

Experimental Host Function Support (and rc4 release) #39

Experimental Host Function Support (and rc4 release) #39

Uh oh!

Conversation

bhelx commented Jan 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bhelx commented Jan 4, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bhelx commented Jan 3, 2024 •

edited

Loading