Test video resizing during playback

[foolip]

Fixes #17821.

New media files are from that issue.

[foolip]

Results at https://wpt.fyi/results/html/semantics/embedded-content/the-video-element/resize-during-playback.html?label=pr_head&max-count=1&pr=19030.

@zcorpan feel like reviewing this?

[foolip]

Still timeout in Safari. I tried playing the file locally and the video is corrupted. Added 'ended' to the event list to fail if it's fired.

[foolip]

Good, test results are what I wanted now.

@eric-carlson @jernoble FYI that this could be an incoming Safari-specific failures. If it's a problem with the media file and Safari should support this sort of this, that'd be great to know.

[guest271314]

Instead of using static files one variation of the test is to use the code which produces the file at each browser with MediaRecorder set the src to an object URL then run the test. Perhaps that might be a test for MediaRecorder and HTML <video> element.

[guest271314]

Does Safari support MediaRecorder?

[foolip]

I dunno, I'd prefer to separate the two, this test is revealing a lack of interop on this specific input that using MediaRecorder might mask.

[guest271314]

Have not compiled a full list, though there are certainly more than one interoperability issues re MediaRecorder, MediaStream and MediaStreamTrack implementations between Firefox and Chrome.

[guest271314]

This issue describes some of the interoperability issues re the previous comment w3c/mediacapture-fromelement#78.

[guest271314]

@foolip This demonstrates that the test can fail at Chromium, Chrome when a WebM file produced by Chromium, Chrome implementation of MediaRecorder is used https://plnkr.co/edit/AovnkZ?p=preview.

[chcunningham]

Conceptually this test LGTM. I've updated chrome to ensure dynamic framesizes are supported for all vp8 decoders. Verified this test still passes when libvpx is used.

Re: media files, at a glance the webm file looks sane. I'm not savvy enough to validate the mp4.

[zcorpan]

The test LGTM. I haven't reviewed the media files themselves.

The expected events are in this order:

1. resize https://html.spec.whatwg.org/multipage/media.html#loading-the-media-resource:event-media-resize
2. playing https://html.spec.whatwg.org/multipage/media.html#ready-states:notify-about-playing
3. resize https://html.spec.whatwg.org/multipage/media.html#the-video-element:event-media-resize
4. probably the test finishes before ended event.

(I recall a Presto test for this from an eon ago, nice to see this finally come to wpt!)

[foolip]

wpt-chrome-dev-stability is failing on Taskcluster like this:

Unstable results

Test	Subtest	Results	Messages
/html/semantics/embedded-content/the-video-element/resize-during-playback.html	mp4 video	FAIL: 2/10, PASS: 8/10	assert_equals: Expected resize event, but got playing event instead expected "resize" but got "playing"
/html/semantics/embedded-content/the-video-element/resize-during-playback.html	webm video	FAIL: 1/10, PASS: 9/10	assert_equals: Expected resize event, but got playing event instead expected "resize" but got "playing"

My interpretation is that the event order isn't reliably in the expected order, but I think this is a case where flaky fail is the best case.

Still, I'll rebase to see if the problem has gone away with a newer version of Chrome. Not reproducing isn't evidence that it's really fixed, of course.

[guest271314]

@foolip

1s_red_400x300_1s_green_200x150.mp4.zip (Does not resize at all media players, use the file at #19030 (comment))

[guest271314]

@fooip Can you confirm the .mp4 video resizes at Safari?

[foolip]

@guest271314, no resize event in Safari:
https://wpt.fyi/results/html/semantics/embedded-content/the-video-element/resize-during-playback.html?label=pr_head&max-count=1&pr=19030

[foolip]

@guest271314 are you entirely sure the mp4 file does correctly resize half way through? Is there a browser or a player where it does resize? I ask because it doesn't seem to resize in Firefox, but WebM does in Firefox.

[guest271314]

@foolip No. Am at a different OS. Erroneously just used ffmpeg without appropriate patches https://bugs.chromium.org/p/webm/issues/detail?id=1642#c17 then uploaded the MP4 file here. Today have been working on compiling FFmpeg from the linked branch, applying patches, then upload here file, again, here.

[guest271314]

@foolip
1s_red_400x300_1s_green_200x150.mp4.zip

Tested at Chromium 80 and Firefox 69.

Was not able to build FFmpeg and apply the patches, again, yet.

The procedure that used was 1) Use <canvas> to draw frames with mimeType set to "video/x-matroska;codecs=h264" (interestingly when canvas.captureStream() to draw frames is not used video does not resize even with H264 codec set https://plnkr.co/edit/uCnx1R?p=info); 2) Copy the H264 encoded video from Matroska to MPEG container, without re-encoding $ /usr/bin/ffmpeg -i 1s_red_400x300_1s_green_200x150.mkv -c:v copy 1s_red_400x300_1s_green_200x150.mp4.

[foolip]

@guest271314 I've applied the PR you sent against this branch and rebased it, let's see what happens now.

[guest271314]

@foolip Do not currently have access to Safari to test. Tried both WebM and MP4 videos at Epiphany (3.28.5) to simulate Safari. Neither video resized. Tried to install Epiphany Technology Preview got error

$ flatpak install https://nightly.gnome.org/repo/appstream/org.gnome.Epiphany.Devel.flatpakref

error: No such ref 'app/org.gnome.Epiphany.Devel/i386/master' in remote gnome-nightly

Therefore currently do not have a means to verify Safari output, which is currently for MP4

FAIL message: assert_equals: Expected resize event, but got ended event instead expected "resize" but got "ended"

and for WebM

PRECONDITION_FAILED message: webm supported

Am not sure what the message

PRECONDITION_FAILED message: mp4 supported

indicates for Firefox at https://wpt.fyi/results/html/semantics/embedded-content/the-video-element/resize-during-playback.html?label=pr_head&max-count=1&pr=19030 ?

[foolip]

https://wpt.fyi/results/html/semantics/embedded-content/the-video-element/resize-during-playback.html?label=pr_head&max-count=1&pr=19030 looks good, the files must be OK because both mp4 and webm pass in Chrome. Looks like Safari doesn't handle resizing mp4 files.

So I'll go ahead and merge this.

@stephenmcgruer FYI, this won't show up as a browser-specific failure because of codec support differing. I don't have any ideas for how to fix this other than changing the query.

[stephenmcgruer]

@stephenmcgruer FYI, this won't show up as a browser-specific failure because of codec support differing. I don't have any ideas for how to fix this other than changing the query.

To be clear, the case you are worried about is that:

i. codec support is not relevant for this test (unknown to me offhand if it's spec'd at all), and
ii. Firefox supports webm but not mp4, and
iii. Safari supports mp4 but not webm, and
iv. Safari does not get the right event sequence for the test.

I think this is probably rare enough to just leave as is, we don't want to start including precondition_failed in browser-specific failure stats. Though I am quite concerned that people are going to use it incorrectly (specifically in cases where a browser doesn't support something that is spec'd and does matter for the test), but time will tell...

[guest271314]

@stephenmcgruer

ii. Firefox supports webm but not mp4, and

Note, Firefox does support mp4 playback.

[stephenmcgruer]

Note, Firefox does support mp4 playback.

Oh, my bad. It seemed to be failing the precondition based on https://wpt.fyi/results/html/semantics/embedded-content/the-video-element/resize-during-playback.html (the precondition is https://github.com/web-platform-tests/wpt/pull/19030/files#diff-785bb71c11bc7b9e3c0fb147340ea8b9R15)

However trying https://output.jsbin.com/vequduj/quiet (which just checks video.canPlayType('video/mp4')) out on Firefox on my Linux desktop I get 'maybe' though (which is acceptable), so seems like something might be weird with the way Firefox is configured in the Taskcluster setup... maybe @gsnedders knows something, or knows who we could ask further?

[guest271314]

"maybe" is not technically equalivalent to failure.

What does

assert_precondition(video.canPlayType(`video/${format}`), `${format} supported`);

expect the output of

video.canPlayType(`video/${format}`)

and

`${format} supported`

to be?

Why is the term "supported" printed in the template literal?

Even Chromium outputs "maybe" for "video/webm" and an empty string "" for "video/x-matroska", although there is a means to both create and playback Matroska videos with H264 and AVC1 codecs at Chromium.

"supported" does not appear to be correct string to test for in any event. Another means can be used to determine if the video can play.

[zcorpan]

canPlayType only gives at best "maybe" when asking for a container format where the codecs are unknown. When codecs parameter is used, and they are also supported, the return value can be "probably".

[guest271314]

Why is the precodition

    function assert_precondition(precondition, description) {
        if (!precondition) {
            throw new PreconditionFailedError(description);
        }
    }

test failing where Firefox 70 and Nightly 72 both output "maybe" for video.canPlayType("video/mp4")?

[zcorpan]

I think @stephenmcgruer is right in

so seems like something might be weird with the way Firefox is configured in the Taskcluster setup...

that is, Firefox as it's run for wpt.fyi doesn't seem to support it. I'm not sure why, or if this is a known thing.
cc @jgraham

[guest271314]

As described at web-platform-tests/wpt.fyi#1648 (comment) there needs to be a means to set and get preferences/flags for browsers run at wpt.fyi. Instead of guessing why certain very basic or experimental features of a given browser are either defined and implemented or not (e.g., OffscreenCanvas tests), after the fact of composing tests for the intended API. Browser settings at wpt.fyi, etc. should be immediately clear before and after the fact.

In the meantime, checking video.canPlayType(<MIME Type>) for an empty string does not necessarily provide accurate results, for example, the case of Matroska support at Chromium, where it is not immediately obvious precisely how to play an .mkv file at Chromium, Chrome.

Am not certain why the assert_precondition function is outputting results that appear to be failure for mp4 at Firefox.

What we do know is that currently Firefox does not support playback of Matroska (.mkv) container (https://bugzilla.mozilla.org/show_bug.cgi?id=1422891; https://bugzilla.mozilla.org/show_bug.cgi?id=1562862) which means we can rely on Firefox throwing a DOMException

    fetch('data:video/x-matroska;base64,<data>')
      .then(r => r.blob())
      .then(blob => {
        const video = document.createElement('video');
        video.controls = true;
        video.src = URL.createObjectURL(blob);
        return video.play()
                  .then(_ => `can play ${blob.type}`)
                  .catch(_ => `cannot play ${blob.type}` /* or empty string to use precondition_assert */);
      })
      .then(console.log, console.error);

DOMException: "The media resource indicated by the src attribute or assigned media provider object was not suitable."

when an .mkv file is played at Firefox or Nightly.

Using that fact we can use the same pattern to test if the video plays by executing play() and returning a value ("can play ") from the new Promise from a chained then(), or catch() the DOMException and return a value ("can not play "), or to continue to use assert_precondition, "+" or "" which should output consistent results at Chrome, Firefox (am not able to verify for Safari) https://plnkr.co/edit/g1rqBtcLwgEC5R1JLQxP?p=preview.

[guest271314]

Note, Firefox will throw an error at play() for .mp4 videos if autoplay for audio and video is not un-blocked.

[guest271314]

Firefox 70 and Nightly 73 pass Run in your browser on w3c-test.org. What does Safari output when the test is run in the browser?

[stephenmcgruer]

There seems to be a lot of cross-talk here. I'm going to try and address what I consider the most pertinent points.

As described at web-platform-tests/wpt.fyi#1648 (comment) there needs to be a means to set and get preferences/flags for browsers run at wpt.fyi. Instead of guessing why certain very basic or experimental features of a given browser are either defined and implemented or not (e.g., OffscreenCanvas tests), after the fact of composing tests for the intended API. Browser settings at wpt.fyi, etc. should be immediately clear before and after the fact.

I do not disagree with your statement. It should be easy for people to understand what flags runs on wpt.fyi were run with. That said, tests should not be written assuming anything about wpt.fyi runs - they should be written to the specs. It is also infeasible to fully describe the environment that leads to a certain result in wpt.fyi; for example in this case I could believe that the behavior we are seeing (Firefox reporting the empty string for canPlayType) is due to some check of the underlying OS or similar rather than an explicit preference given to Firefox.

I have opened #20581 to look into this.

In the meantime, checking video.canPlayType() for an empty string does not necessarily provide accurate results, for example, the case of Matroska support at Chromium, where it is not immediately obvious precisely how to play an .mkv file at Chromium, Chrome.

If Chromium is returning an empty string for mkv, but supports mkv in <video> elements, that sounds like a browser bug to me. The spec is clear about canPlayType:

i. canPlayType is specified to return an empty string if "type is a type that the user agent knows it cannot render"
ii. "a type that the user agent knows it cannot render" is specified as a "type that the user agent knows it cannot render is one that describes a resource that the user agent definitely does not support".

If browsers are widely violating the spec, and we can get agreement that the violation is reasonable, we change the spec. Otherwise we treat deviations from specs as browser bugs.

Using that fact we can use the same pattern to test if the video plays...

Given that canPlayType('video/mp4') is falsey for Firefox on our Taskcluster-based runs, I would suspect that this more complicated approach to detecting support is also going to fail. You could always give it a go though; uploading a pull request with the change will cause a Taskcluster run which should give you an answer.

What does Safari output when the test is run in the browser?

Based on a browserstack run, Safari 13 on Catalina reports "http://w3c-test.org/html/semantics/embedded-content/the-video-element/resize-during-playback.html" for mp4 and PRECONDITION_FAILED for webm. That is the same result as we get from our Azure Pipelines runs of Safari that are shown on wpt.fyi.