define behaviors of the common ConstrainablePattern Interfaces

[ShijunS]

The spec currently doesn't cover behaviors for the ConstrainablePattern Interfaces. As example, here are some questions with applyConstraints()

• Whether the "frameRate" constraint can be applied to video track captured from video element and canvas (might be relevant to issue <canvas>.captureStream() framerate parameter is too restrictive #43)
• Whether apps want to capture a lower resolution video from an UHD/4k-video source, and whether the "width" and "height" constraints should be honored in that case
• Whether the "channelCount" constraint should be applied for captured audio tracks. As an potential use case, we might need to convert multi-channel audio to stereo before we send it out with peerconnection.
• How about other audio constraints: volume, sampleRate, and sampleSize - although it might require audio DSP operations so leading to more complexity in implementations.

In case any of the common ConstrainablePattern Interfaces is irrelevant, the spec should call that out explicitly.

[martinthomson]

Yes, I think that this is important. I think that the only things we want to honor here is the frameRate constraint. The rest can remain unsupported.

[alvestrand]

If one desires to make a 640x480 recording of an UHD video element (

(It's OK to say "can't do this", but I don't like the idea of making the spec "can't ask for this").

[Pehrsons]

In my opinion you cannot constrain the decoder because it will output UHD frames always - as they were encoded. It would become a processing step rather than a configuration (by constraining the source) step.

I think MediaRecorder should have an API for this instead.

[alvestrand]

In the scenario I sketched, there might not be a decoder (the UHD-sized video element might be showing animation from a Canvas). Or there might be one.
The video element what it produces into a MediaStreamTrack. A MediaStreamTrack has a constrainable interface.
Why would we specify two APIs for doing the same thing?

In all cases, the UHD frames will be fed to a rescaler somewhere in the system. The question is just about how we specify the API whereby the user can say "I want this to be rescaled".

[yell0wd0g]

To bring the Constrainable Pattern to the CanvasCaptureMediaStreamTrack and drawing from my experience with ImageCapture w3c/mediacapture-image#146, I think roughly here we'd need to define a supported constraints dictionary as sth like:

partial dictionary MediaTrackSupportedConstraints {
 boolean width = false;
 boolean height = false;
 boolean aspectRatio = false;
 boolean frameRate = true;
 boolean facingMode = false;
 boolean volume = false;
 boolean sampleRate = false;
 boolean sampleSize = false;
 boolean echoCancellation = false;
 boolean latency = false;
 boolean channelCount = false;
 boolean deviceId = false;
 boolean groupId = false;
};

and also the behaviour of the rest of the Constrainable Pattern methods that are needed as well, i.e. these guys:

MediaTrackCapabilities getCapabilities ();
MediaTrackConstraints getConstraints ();
MediaTrackSettings getSettings ();
Promise<void> applyConstraints (optional MediaTrackConstraints constraints);

And to round the change, captureStream() should accept a MediaTrackConstraintSet a la getUserMedia().

We should also have a similar analysis for the "standard" MediaStreamTrack produced by captureStream() on an HTMLMediaElement.

ImageCapture has a rendering of my proposed PR in https://rawgit.com/Miguelao/mediacapture-image/pr001_constrainable_pattern/index.html

[martinthomson]

Maybe width and height are OK to change. I'd have to double check with @jan-ivar though on the latest viewpoint on those regarding processing in the track (as opposed to having the source just change its capture size).

[martinthomson]

If we only have one constrainable value, MediaTrackConstraintSet is overkill, but if the intent is to make things consistent, then fine. Every time that pattern spreads its shadow the web dies a little more, but it was already pestilent when we got here I guess.

[alvestrand]

as to the mediaTrackSupportedConstraints thing - I think you would want to return this in Capabilities (give no alternatives for things you can't alter) rather than supportedConstraints (which ar esupposed to list things where the name is understood by the platform).

given that CanvasCaptureMediaStreamTrack is a subclass of MediaStreamTrack, it already inherits the constrainable-pattern functions. You don't get to remove features when inheriting.

[jan-ivar]

The Constrainable pattern was designed for device discovery and sharing. It's complex, because A) people's devices may not suit your needs, and B) devices you get may already be in use, effectively shared with another party (another tab) - without any knowledge of other parties - which puts a cramp on available settings. It's an API designed around compromise and negotiation.

A rescaler is the opposite of that. If you ask for 800x600, you'll always get it. Every time. Exactly.

This makes the Constrainable pattern overkill and a terrible API for a rescaler.

It's also a weak API, because the pattern allows huge variance in browser implementations. Maybe you'll get OverConstrainedError if you used exact, maybe it'll succeed with different values than you asked for if you used ideal. Instead, you just want all browsers to rescale. Every time. Exactly.

In case any of the common ConstrainablePattern Interfaces is irrelevant, the spec should call that out explicitly.

I'd say on the contrary, unless a source explicitly defines constrainable properties, then there are none implied by simply pulling in a pattern. E.g. None of the properties width, height or frameRate come with the ConstrainablePattern.

Also, IMHO we should start with needs, not go looking for functionality to implement out of some need for symmetry.

From the needs I've heard, something straightforward like what @Pehrsons mentions sounds better to me.

[dontcallmedom-bot]

This issue was mentioned in WEBRTCWG-2024-03-26 (Page 29)