[audioworklet] Bring Your Own Buffer style of memory management

[joeberkovitz]

Discussing with WebAssembly group, we've been talking about having an alternate approach to managing the buffers used to communicate data between an AudioWorkletProcessor and the rendering thread environment (input, output and parameters buffers). It would be better for WASM in many ways if we allowed the AWP code to respond to the needs of the environment by providing its own buffers, rather than having to work with JS buffers or views provided from outside. This would eliminate the number of copies and make for a very direct WebAssembly/Web Audio connection.

[hoch]

The conversation from TPAC:
We need to come up with the API design. For example, AudioWorkletProcessor's constructor can expose methods like registerInputBuffer() and registerOutputBuffer() so the user code pass the piece of WASM heap memory.

[svgeesus]

related AudioWorklets integration with WebAssembly

[rtoy]

Teleconf: Probably a duplicate of #2427, which has more detail on AudioWorklet and WASM.

[hoch]

The outcome from TPAC 2020:

This is not an exact duplicate of #2427. This can be applied to non-WASM use cases as well. With both improvements, the performance of Audio Worklet can be certainly improved.

What's being proposed is to call something like registerBuffer(wasmHeap) or registerBuffer(typedArray) at the node/processor registration procedure.

Also a callback function like ontopologychange is necessary, so the user code can reallocate memory when the incoming AudioNode configuration (e.g. channel count) is changed.

[padenot]

Discussed during the AudioWG online F2F, some pseudo-code that show how this could work (just a proposal, nothing concrete, drafted while talking during the call):

partial interface AudioWorkletProcessor {
    enum BufferType {
      INPUT = 1,
      OUTPUT = 2
    };
    // indicates that this memory will be used by the AudioWorklet -- don't touch it yourself
    // variant: register just the output, just the input.
    // This can be called in the ctor, or while process is running to provide the output from something already computed
    registerBuffers(ArrayBuffer memory, uint64_t offset, uint64_t maxLength, type = INPUT|OUTPUT);
 
    // Called when the input or output channel count changes for whatever reason
    // you can accept or deny the request. In case of error, `process` isn't called anymore
    channelTopologyChange = function(ArrayBuffer memory, uint64_t offset, uint64_t newMaxLength) {
        // return false; // reject the change request: audio will be down/up mixed to the previous topology
 
       // find some memory in already allocated space/allocate new memory
        
        return {
            memory: memory,
            offset: new_offset,
            maxLength: new_max_length
            type: INPUT
         };
    }
}

then the memory backing the arrays in process would be the memory registered above.

Being able to call registerBuffers from process will be very nice for be for developers using this pattern described by @hoch: buffers of the right size are prepared in advance (with a safety margin), and the worklet just uses one of them as output. The buffer could then be safely recycled in the next process call.

[mdjp]

This would be a great issue to discuss across groups as TPAC 2021

[juj]

I did recently write an implementation that enables multithreaded Wasm code to define AudioWorkletProcessors and instantiate AudioWorkletNodes from Wasm. The branch can be found here:

https://github.com/juj/emscripten/compare/wasm_workers...juj:audio_worklets?expand=1

The intent is to bring the support in to core Emscripten in the near future.

The benefit of that branch with respect to this github issue is probably to concretely show what the JS<->Wasm marshalling copying overhead currently looks like, see the process function in src/library_webaudio.js.

In the implementation explicit care was taken to make the marshalling as "tight" as possible with minimized JS overhead (use Wasm thread stack space to avoid mallocs, minimize excess computation or abstractions in JS). So far we don't unfortunately have any data to know how much that marshalling performance overhead is in general.

For a small tone generator C program example of how one would use that API, see the end of the PR, tests/webaudio/tone_generator.c.

[Jonathhhan]

I use Open Frameworks with Emscripten, and try to replace scriptprocessornode with audioworklet.
With sharedarraybuffer I can access the wasm memory from the main thread in the audioworklet. My problem is, that I cannot import a function (the audio callback) from the wasm Module in the main thread (it also needs to be in the main thread because it generates graphics too).

This is how I import the memory:

var outbufferArray = Module.HEAPF32.subarray(outbuffer >> 2, (outbuffer >> 2) + bufferSize * outputChannels); AUDIO.contexts[context_id].audioWorklet.addModule("bypass-processor.js").then(() => { const bypasser = new AudioWorkletNode(AUDIO.contexts[context_id], "bypass-processor", [1, 1, 2]); var worker = new Worker("wasm_worker.js"); bypasser.port.postMessage(outbufferArray) bypasser.port.onmessage = (e) => { dynCall("viiii", callback, [bufferSize, inputChannels, outputChannels, userData]); } bypasser.connect(AUDIO.contexts[context_id].destination) } );

This is the Audioworklet:

`
var counter = 0
class BypassProcessor extends AudioWorkletProcessor {
constructor() {
super()
this.port.onmessage = (e) => {
console.log(e.data, sampleRate)
this.data = e.data;
}
}

process(inputs, outputs) {

    counter = currentFrame / 128 % 64;
    if (counter == 0) {
        this.port.postMessage('pong')
    }
    const input = inputs[0];
    const output = outputs[0];
    for (let channel = 0; channel < 1; ++channel) {
        const outputChannel = output[channel];
            outputChannel.set(this.data.slice(counter * 128, counter * 128 + 128));
    }
    return true
}

}
registerProcessor('bypass-processor', BypassProcessor);
`

I think, my problem would be solved, if I could call dynCall("viiii", callback, [bufferSize, inputChannels, outputChannels, userData]); from the audioworklet. Calling the callback with port.postMessage is less reliable than audioscriptprocessor (but the sharedarraybuffer works well).

[juj]

@Jonathhhan check out the audio_worklets branch in my Emscripten Github repo.

In particular, the commit juj/emscripten@4aa53f4 . That achieves such a shared setup where the main wasm application can run on the main browser thread, and create wasm-based audioworklet nodes that call to a custom wasm function entry point for audio processing.

[Jonathhhan]

@juj thank you. I will try that soon. Yesterday I had success with the solution from @tklajnscek emscripten-core/emscripten#12502 would be really great, if audioworklets will become a standard feature of Emscripten.