Pause/Stop wasm execution

[inkeliz]

I have the following code:

commandOffset, _ := module.ExportedFunction("SetupCommand").Call(nil)

// ...

module.ExportedFunction("Update").Call(nil)

command, _ := module.Memory().Read(uint32(commandOffset[0]), 2048)
copy(commandCache[:], command)

The idea is:

Any arbitrary module/wasm should create one array of 2048bytes. I omit the error handling. My program will call SetupCommand when the wasm initialize, so I can read the information in the future. I will call Update multiple times (say, every 32ms), and the module/wasm should populate the command buffer with some content. Then, my program (which is using wazero) will read the given command (based on the pointer given on SetupCommand).

---

The issue is:

Currently, I need to copy the information, and I'm not sure how safe it's, since the module/wasm can change the memory at anytime, including while I'm copying it.

Also, nothing prevents the wasm to continue to run, in the background, which may consume CPU. For instance, one "malicious wasm" could perform any task outside of the Update function. To make clear, one "malicious" code could use go func(){for {}}, so it will keep running in the background. Or even worse, it could use some go func(){for{rand.Read(commandBuf)}}(), so the wasm is changing the command (which my program is reading).

---

I would like to do something like:

update, _ := module.ExportedFunction("Update")
module.Stop()                                            // << Stop the execution of wasm

for range time.Tick(32 * time.Millisecond) {
	module.Continue()                            // << Continue the execution of wasm
	update.Call(nil)
	module.Stop()                                  // << Stop the execution of wasm
	
	command, _ = module.Memory().Read(uint32(commandOffset[0]), 2048)
}

The Continue and Stop will pause the execution of the module/wasm, preventing it from manipulate the memory and from perform any other task. In order words: the module/wasm will ONLY work while Update doesn't return (and I can implement some "timeout" to kill the wasm if it takes too long, that is not the point here). After calling Stop(), the module/wasm must not be able to perform any task anymore. The module/wasm can continue do whatever it's doing once Continue is called, including change the memory information and read it, it can also reply the next Update call. The honest code should never be affected, since it's design to only perform tasks inside Update.

I'm not sure if that can be done already, the only option seems to use .Close(), but I'm not sure how it would affect the performance, and if that will keep the information from the module (keep the memory).

[codefromthecrypt]

Will take some time to think through this. For starters, you can invoke an unexported function for the setup, if it is in wasm as the "start function"

https://www.w3.org/TR/wasm-core-1/#start-function%E2%91%A4

other modules won't be able to call an unexported function directly

[inkeliz]

Thank you. I'll check the start function, and how I can use that, I'm calling one exported function because seems easier to use. 😅

I don't understand the internals of wasm, so I'm not sure how hard is to pause the execution of the wasm. I guess it have their on stack and so on, and it should recover from that point once Continue is called. 🤔

[codefromthecrypt]

the longer thinking will be around that wasm isn't necessarily executed from the same goroutine, though to pause it would imply it is forked somehow. An interceptor could pause anything, though even without that a host function can. pausing in the middle of wasm of a binary is likely expensive to implement unless the break points are well defined. If you are routinely routing through host functions, that's probably the cheapest pause area.

[inkeliz]

You give me one idea, but I'm not sure if will work. In my case that might work, but not in general case. I would like to support many languages as possible (like Zig, Rust, Go, and may others). My goal is to use wazero similar to an "VM", running some arbitrary code (the compilation will be performed on my end, but the source-code might not be trusted). That is why I'm interested to set some limitation.

I need to test it little bit more, but in general, that is the idea:

env, err := r.NewModuleBuilder("env").
	ExportFunction(`done`, func(ctx wasm.Module) int32 {
		done <- release
		<-release
		return 0
	}).
	Instantiate()

Expose one Done function, which the wasm must call inside Update. So, in theory, the done will never return, and will only return when a new Update is requested:

update := module.ExportedFunction("Update")

var lock *chan struct{}
for range time.Tick(32 * time.Millisecond) {
	t := time.Now()
	if lock != nil {
		*lock <- struct{}{}
	}

	go func() {
		if _, err = update.Call(nil); err != nil {
			panic(err)
		}
	}()

	select {
	case <-time.After(1 * time.Millisecond):
		fmt.Println("bye", time.Since(t), module.Memory().Size())
		module.Close()
		return
	case d := <-done:
		fmt.Println("done", time.Since(t), module.Memory().Size())
		lock = &d
	}
}

---

I test the wasm, using TinyGo, with:

//export done
func done() int32

//export Update
func Update() {
	done()
}

Compiling with tinygo build -target wasm -scheduler none -no-debug -o wasm.wasm ., the -scheduler none seems to enforce that it's not possible to run some go func() { done() }, in away to bypass the "lock". However, I'm not sure if it would be enough for Zig/Rust thread system. My concern is that someone can create "interrupt/switch" in wasm itself, which can call done and while waiting it can perform another task. I'm not sure how efficient it's, when running multiples wasm at same time, and all of them waiting for the "lock" be released.

[codefromthecrypt]

ps I plan to revisit this topic after tomorrow as have some things to chase up. if you have other ideas and context meanwhile, please dump. This could even include links to other runtimes if they have some sort of stop-the-world feature.

[codefromthecrypt]

ps wires crossed as I was typing my response when you sent yours :D

[codefromthecrypt]

Note: we have multiple things that end up needing some sort of interceptor. If the pause feature can be loose to "the next call to an external function" interceptor would work. This same interceptor could implement tracing for each boundary cross (as demarcated by an external (import or export)) function call.

[codefromthecrypt]

PS I am going to focus time next week on the interceptor thing and we can see if it helps enough or not. I have to implement it anyway as debugging arbitrary wasm is too hard (takes too long) without some sort of interceptor, and if I don't I won't have enough time to do other things :D

It is possible an interceptor won't be the right hook, but anyway we can find out and go another route if it doesn't.

[sam-at-luther]

From the go docs, it's not immediately clear if canceling the passed ctx will stop execution. From this discussion I gather that it does not (I have not tried it), and that the ctx is more for tracing purposes--is that right?

[codefromthecrypt]

Going to give some updates

ps @sam-at-luther I answered you on a different issue as this one is on pause/resume which will be different than close (via cancel) #509

@inkeliz so @anuraaga added the first go of listener which has function hooks. This is internal which means to try it requires a fork of wazero. That's intentional because it isn't stable, yet and notably will need to change more as we finish the other two value types in WebAssembly 2.0

https://github.com/tetratelabs/wazero/blob/main/internal/experimental/api/listener.go
https://github.com/tetratelabs/wazero/blob/main/internal/experimental/examples/function-listener/print-trace.go#L43-L44

I've also progressed some other things more direct to your question I think, though it isn't integrated in a way you can use it, yet. For example, your earlier question was about freezing the memory state. api.Memory now has all operations propagating context, which means at least from a trace POV you can know who is trying to modify memory and make a decision. The problem is there are no interceptors based on memory hooks and they may or may not be in the end interceptor or event listener design. As we're trying to get Wasm 2.0 features finished, this isn't likely to resolve in the next couple weeks. However, I wanted to give you an update as this stuff has been on the mind and action, albeit slower.

[evacchi]

some of the use cases described here have now been addressed by #1108

[iameli]

Hi! I have a use case that involves "save states" of a text adventure game, so I was looking for a mechanism for doing that, something like wasmerio/wasmer#489 but for wazero. If I'm understanding this correctly, is it the case that I could pause execution and get a full dump of memory but perhaps not yet of the stack pointer and whatnot?

EDIT: Oh, #1808 seems like what I'm looking for, never mind!

[mathetake]

https://github.com/tetratelabs/wazero/blob/main/experimental/checkpoint.go