Alternative stream-based API (for mediacapture-transform)

[jan-ivar]

From https://github.com/alvestrand/mediacapture-transform/issues/59 (see link for earlier discussion):

As requested, this is to discuss our upcoming proposal, which I've written up as a standards document, with 3 examples.

This brief explainer isn't a substitute for that doc or the slides, but walks through a 41-line fiddle:

Since tracks are transferable, instead of of creating all tracks ahead of time and transferring their streams, we simply transfer the camera track to the worker:

const stream = await navigator.mediaDevices.getUserMedia({video: {width: 1280, height: 720}});
video1.srcObject = stream.clone();
const [track] = stream.getVideoTracks();
const worker = new Worker(`worker.js`);
worker.postMessage({track}, [track]);

...and receive a processed track in return:

  const {data} = await new Promise(r => worker.onmessage);
  video2.srcObject = new MediaStream([data.track]);
};

The worker pipes the camera track.readable through a video processing step into a writable VideoTrackSource source, whose resulting source.track it transfers back with postMessage.

// worker.js
onmessage = async ({data: {track}}) => {
  const source = new VideoTrackSource();
  parent.postMessage({track: source.track}, [source.track]);

  await track.readable.pipeThrough(new TransformStream({transform: crop})).pipeTo(source.writable);
};

This avoids exposing data on main thread by default. The source (and the real-time media pipeline) stays in the worker, while its source.track (a control surface) can be transferred to the main thread. track.clone() inherits the same source.

This aligns with mediacapture-main's sources and sinks model, which separates a source from its track, and makes it easy to extend the source interface later.

The slides go on to show how clone() and applyConstraints() used in the worker help avoid needing the tee() function.

Please conduct further WG discussion here, so that discussion is tracked.

[dontcallmedom]

There is now a base WHATWG-streams API adopted by the WG, closing this issue