Alternative stream-based API

[jan-ivar]

As requested, this is to discuss our upcoming proposal, which I've written up as a standards document, with 3 examples.

This brief explainer isn't a substitute for that doc or the slides, but walks through a 41-line fiddle:

Since tracks are transferable, instead of of creating all tracks ahead of time and transferring their streams, we simply transfer the camera track to the worker:

const stream = await navigator.mediaDevices.getUserMedia({video: {width: 1280, height: 720}});
video1.srcObject = stream.clone();
const [track] = stream.getVideoTracks();
const worker = new Worker(`worker.js`);
worker.postMessage({track}, [track]);

...and receive a processed track in return:

  const {data} = await new Promise(r => worker.onmessage);
  video2.srcObject = new MediaStream([data.track]);
};

The worker pipes the camera track.readable through a video processing step into a writable VideoTrackSource source, whose resulting source.track it transfers back with postMessage.

// worker.js
onmessage = async ({data: {track}}) => {
  const source = new VideoTrackSource();
  parent.postMessage({track: source.track}, [source.track]);

  await track.readable.pipeThrough(new TransformStream({transform: crop})).pipeTo(source.writable);
};

This avoids exposing data on main thread by default. The source (and the real-time media pipeline) stays in the worker, while its source.track (a control surface) can be transferred to the main thread. track.clone() inherits the same source.

This aligns with mediacapture-main's sources and sinks model, which separates a source from its track, and makes it easy to extend the source interface later.

The slides go on to show how clone() and applyConstraints() used in the worker help avoid needing the tee() function.

[aboba]

This illustrates use cases for transferrable MediaStreamTrack. But can't MST transfer also be used with MSTP/MSTG? Trying to understand the differences unique to the alternative stream-based API.

[jan-ivar]

can't MST transfer also be used with MSTP/MSTG? Trying to understand the differences unique to the alternative stream-based API.

MSTP/MSTG is "not an adopted working group document" because it

1. Exposes the realtime media pipeline on main thread by default
2. Fails to encourage use in workers (users & UAs)

Those are the two main blockers this proposal aims to address. I've added some intro slides on the importance of this.

MSTP/MSTG was also written before transferable MST, making it awkwardly backwards and unfriendly to workers: A worker shouldn't have to ask main thread to access a track it already has, or to create a new one from a source of frames it has.

The benefits slide now solely covers benefits over MSTP/MSTG:

Benefits

1. Simpler API taking advantage of transferable MST
2. Fewer new API objects to learn. Friendly to workers
3. Satisfies core principles from a worker POV, without main-thread entangled APIs (“worker see, worker do”)
4. Doesn’t block realtime media pipeline on main thread by default
5. Source stays in worker, separate from its track(s). Clean xfer
6. applyConstraints available in the worker
7. Parity+ with MSTP/MSTG features & brevity (e.g. 41 lines)
8. Doesn’t need transferable streams UA “optimizations”
9. Source.muted attribute

[jan-ivar]

@alvestrand can you transfer this to mediacapture-extensions? I seem unable to.

[alvestrand]

I don't seem to have that power either, despite having admin rights on the webrtc-extensions repo.

[alvestrand]

https://docs.github.com/en/issues/tracking-your-work-with-issues/transferring-an-issue-to-another-repository says that you can't transfer issues between user repos and org repos (or between orgs), so I guess you'll just have to file a new one and point back to this discussion.

[jan-ivar]

Closed in favor of w3c/mediacapture-extensions#38.