AudioWorklet support

[jon-myers]

calling .audioWorklet on an initialized AudioContext returns undefined. Is working with web audio audio worklets supported?

[b-ma]

Hey, no the AudioWorklet API is not yet supported unfortunately, as few other nodes. Hopefully it will be in some future but I can't give any roadmap nor due date.

I will had some notes on that in the README

[jampekka]

I'd be interested in looking into this. The biggest hurdle is probably to get the AudioWorkerProcessor to run in the web-audio-api-rs audio rendering thread. I think this could be possible by spawning a new napi::Env in the rendering thread, call Env::run_script on that and implement AudioWorkletProcessor in the rust side that then calls the JS AudioWorkletProcessor defined in Env::run_script.

I'm very new to rust and node.js internals, so this may not be the way to go. Could somebody more familiar with these comment if this architecture is worth pursuing?

[orottier]

Hi @jampekka that would be very welcome. Your suggestion makes sense and I have looked into this before, but I was unable to get an env set up on the render thread. It is obviously not allowed to use the Env of the main thread, but nowhere in the n-api docs I could find a way to construct a new one.

I will investigate further and report back in this issue. If you do the same, that would be great

[b-ma]

but nowhere in the n-api docs I could find a way to construct a new one.

I have made a bit of research and found this https://docs.rs/napi/latest/napi/attr.module_init.html (examples are a bit weird) which seems to correspond to NAPI_MODULE_INIT in https://nodejs.org/api/n-api.html (you need to manually search for it, no direct link), which seems to be the macro that allows to create a new env

The parameters env and exports are provided to the body of the NAPI_MODULE_INIT macro.

Maybe this can help?

[orottier]

Hey @b-ma, I read those docs but I think that is not related to the actual executor. But more for the module loading (require ...) and running a one-time initialization method on the module. But I may be mistaken.

I'm simultaneously looking into https://nodejs.org/api/vm.html which may be interesting provided it does not share the event loop with the main thread

A common use case is to run the code in a different V8 Context. This means invoked code has a different global object than the invoking code.

Another option is to spawn a worker https://nodejs.org/api/worker_threads.html

The node:worker_threads module enables the use of threads that execute JavaScript in parallel. ...
Workers (threads) are useful for performing CPU-intensive JavaScript operations.

Which would then serve as an extra thread alongside the render thread. We then use message passing to invoke process calls on the worker thread. Not ideal but maybe a good start.

[jampekka]

The more I dig into this, the more confused I get how to do it.

My original idea to create a new Env using napi probably doesn't work: I can't find a way to create a new Env, and my impression from the Node-API docs is that this shouldn't be done.

@orottier: https://nodejs.org/api/vm.html does look promising. I guess to do this we should be able to access the created vm.Script object on the rust side and try to somehow run code in it in the rendering thread?

Yet another possibility would be to create a totally new node.js instance by embedding it in the web-audio-api-rs rendering thread. Embedding a new instance seems quite straightforward (from C++ at least): https://github.com/nodejs/node/blob/main/doc/api/embedding.md

[jampekka]

The node:vm approach looks promising. At least on JS side it seems to work as we'd like. We can hook a "processor" from a new VM and call that from the parent, and we're able to write into parameters passed from the parent. This all also happens synchronously, as we'd like for the processor to be called from the render thread.

So on the rust side this could be just as simple as calling the registered processor in the render thread. This would be along the lines of implementing e.g. JsAudioWorkletProcessor on rust side that stores the callback (or the javascript AudioWorkletProcessor object) and an associated AudioWorkletNode<JsAudioWorkletProcessor>?

const vm = require('node:vm');

const script_code = `
function process(inputs, outputs) {
	// Just to show we can write the value
	outputs[0][0] = inputs[0][0];

	return true;
};

register(process);
`;

var processor = null;
function register(proc) {
	processor = proc;
}

const processor_context = {register};
vm.createContext(processor_context);
const script = new vm.Script(script_code);
script.runInContext(processor_context);

inputs = [new Float32Array([1, 1, 1])];
outputs = [new Float32Array([0, 0, 0])];
console.log(outputs[0]); // Float32Array(3) [ 0, 0, 0 ]

var ret = processor(inputs, outputs);
console.log(ret); // true
console.log(outputs[0]); // Float32Array(3) [ 1, 0, 0 ]

[b-ma]

Looks nice indeed (I need to dig into this vm stuff, looks really interesting),

But I'd just like to add that at some point we will need to cast from rust [f32] to js Float32Array and back, all in the render thread, and I actually don't see how this can be done without an actual Env...

I'm a bit afraid that we might have to deal directly wether with the V8 api actually, cf. https://v8docs.nodesource.com/node-4.8/d5/dda/classv8_1_1_isolate.html, or as you said, with embedding a new node instance in the render thread

[jampekka]

napi-rs seems to have Float32Array built in, and we can mutate them in JS side. At least conversion from Vec<f32> works. I don't know enough Rust to know what's the difference to [f32] in this regard. We of course do have an Env in napi-rs bound functions, although I'm not sure if it's the main script's Env or the VM script's Env!

Here's a simple napi-rs POC that defines a callback in a new VM that is called in Rust to do Float32Array processing. Not sure what will happen if the callback is called from another thread, but if it works, this could work for audio worklets if we can store the callback value that is passed to process_callback?

#![deny(clippy::all)]

use napi_derive::napi;
use napi::bindgen_prelude::*;

#[napi]
pub fn process_callback<T: Fn(Float32Array, &mut Float32Array) -> Result<()>>(callback: T) -> Float32Array {
    let input = Float32Array::new([1.0, 2.0, 3.0].to_vec());
    let mut output = Float32Array::new([0.0, 0.0, 0.0].to_vec());
    let _ = callback(input, &mut output);
    return output;
}

const {processCallback} = require('./index')

const vm = require('node:vm');

// This defines our process function in a different VM
const script_code = `
function process(input, output) {
	// Just to show we can write the value
	output[0] = input[0]*1337;

	return true;
};

register(process);
`;


// The processor in the VM is stored here
var processor = null;
function register(proc) {
	processor = proc;
}

// Run the VM, which populates the processor
const processor_context = {register};
vm.createContext(processor_context);
const script = new vm.Script(script_code);
script.runInContext(processor_context);

// Run the processor in rust side
output = processCallback(processor);
console.log(output); // Float32Array(3) [ 1337, 0, 0 ]

https://github.com/jampekka/napi-rs-testing

[orottier]

@orottier: https://nodejs.org/api/vm.html does look promising. I guess to do this we should be able to access the created vm.Script object on the rust side and try to somehow run code in it in the rendering thread?

That's the big question. Can we detach a pointer to the VM and then work with it independently from the main context/thread. Or is a VM just an isolation mechanism and cannot be run independently from the main Env. I mean, to call runInContext we still need a valid Env handle...

Yet another possibility would be to create a totally new node.js instance by embedding it in the web-audio-api-rs rendering thread. Embedding a new instance seems quite straightforward (from C++ at least): https://github.com/nodejs/node/blob/main/doc/api/embedding.md

This could be interesting because a drawback of using Worker is that we cannot force it to run with high priority/realtime safety and we could end up with priority inversion

I'm wondering if this requires us to include V8 as a dependency on the Rust side, or if we can 'reuse' the fact that we are already running in a NodeJs context. The binary size will increase greatly if we can't, and it does not really feel right to have V8 as a dependency of a NodeJs lib..

-

In any case, I would like to get started with something so I will probably experiment with the Worker setup in the coming weeks. But let's continue with other explorations @b-ma @jampekka, that is very helpful!

[orottier]

Hey @b-ma I remember you had another comment here, did that not work out?

I have another terrible idea:

• spawn a new node process from the rust side
• copy the audio worklet constructor code into stdin and instantiate it
• on each render quantum, call into the node process and copy the stdout back into the rust audio buffer

So that looks like the Worker setup (with the same drawbacks regarding real-time safety) but this has the benefits that I can write more Rust and less NodeJs, so it's easier for me to tinker with.

I will start with it and keep you posted!

[b-ma]

Hey @b-ma I remember you had another comment here, did that not work out?

No, after more checks this was a dead end, everything was launched in same thread...

For info, also had a look at https://nodejs.org/api/worker_threads.html (which would be kind of ideal asSharedArrayBuffer and messagePorts would be free...) but unfortunately I don't see any way to launch the worker into a given thread

I have another terrible idea:

Hehe quite brute force indeed...

I was thinking about launching a new node process with another napi entry point, which looks quite similar finally... So... Let's see where it goes!

[orottier]

Alright. I have my terrible experiment running at orottier/web-audio-api-rs#503