Skip to content

Commit

Permalink
Specify range fetching for media elements
Browse files Browse the repository at this point in the history 
See whatwg#2814, especially whatwg#2814 (comment), for some background.

The strategy taken here is that multiple opaque range responses are OK as long as they're from the same origin. (They don't have to be the same URL.) Programmatic service worker responses are considered a null origin for these purposes.

This matches implemented behaviors.
  • Loading branch information
@noamr
noamr authored and Mason Freed committed Jun 3, 2022
1 parent 6c11304 commit e1572fb
Showing 1 changed file with 156 additions and 37 deletions.
193 changes: 156 additions & 37 deletions source
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2539,6 +2539,7 @@ a.setAttribute('href', 'https://example.com/'); // change the content attribute
<li><dfn data-x-href="https://fetch.spec.whatwg.org/#finalize-and-report-timing">finalize and report timing</dfn></li>
<li><dfn data-x="serialize-a-response-url-for-reporting" data-x-href="https://fetch.spec.whatwg.org/#serialize-a-response-url-for-reporting">serialize a response URL for reporting</dfn></li>
<li><dfn data-x="body safely extract" data-x-href="https://fetch.spec.whatwg.org/#bodyinit-safely-extract">safely extracting a body</dfn></li>
<li><dfn data-x="body-incrementally-read" data-x-href="https://fetch.spec.whatwg.org/#body-incrementally-read">incrementally reading a body</dfn></li>
<li><dfn data-x-href="https://fetch.spec.whatwg.org/#process-response-end-of-body">processResponseConsumeBody</dfn></li>
<li><dfn data-x-href="https://fetch.spec.whatwg.org/#fetch-processresponseendofbody">processResponseEndOfBody</dfn></li>
<li>
Expand All @@ -2556,6 +2557,10 @@ a.setAttribute('href', 'https://example.com/'); // change the content attribute
<li><dfn data-x="concept-response-location-url" data-x-href="https://fetch.spec.whatwg.org/#concept-response-location-url">location URL</dfn></li>
<li><dfn data-x="concept-response-timing-info" data-x-href="https://fetch.spec.whatwg.org/#concept-response-timing-info">timing info</dfn></li>
<li><dfn data-x="concept-response-service-worker-timing-info" data-x-href="https://fetch.spec.whatwg.org/#response-service-worker-timing-info">service worker timing info</dfn></li>
<li>
<dfn data-x-href="https://wicg.github.io/background-fetch/#extract-content-range-values">extract content-range values</dfn>
<!-- TODO: move this to FETCH -->
</li>
</ul>
</li>
<li>
Expand Down Expand Up @@ -2595,6 +2600,7 @@ a.setAttribute('href', 'https://example.com/'); // change the content attribute
<li><dfn data-x="concept-request-reload-navigation-flag" data-x-href="https://fetch.spec.whatwg.org/#concept-request-reload-navigation-flag">reload-navigation flag</dfn></li>
<li><dfn data-x="concept-request-history-navigation-flag" data-x-href="https://fetch.spec.whatwg.org/#concept-request-history-navigation-flag">history-navigation flag</dfn></li>
<li><dfn data-x="concept-request-user-activation" data-x-href="https://fetch.spec.whatwg.org/#request-user-activation">user-activation</dfn></li>
<li><dfn data-x-href="https://fetch.spec.whatwg.org/#concept-request-add-range-header">add a range header</dfn></li>
</ul>
</li>
</ul>
Expand Down Expand Up @@ -33860,6 +33866,11 @@ interface <dfn interface>HTMLMediaElement</dfn> : <span>HTMLElement</span> {

</p>

<p>A <span>media resource</span> has an associated
<dfn data-x="media-resource-origin">origin</dfn>, which is either "<code data-x="">none</code>",
"<code data-x="">multiple</code>", "<code data-x="">rewritten</code>", or an
<span>origin</span>. It is initially set to "<code data-x="">none</code>".</p>

<p>A <span>media resource</span> can have multiple audio and video tracks. For the purposes of a
<span>media element</span>, the video data of the <span>media resource</span> is only that of the
currently selected track (if any) as given by the element's <code
Expand Down Expand Up @@ -34676,6 +34687,66 @@ interface <dfn interface>MediaError</dfn> {
</li>
</ol>

<p>To <dfn>verify a media response</dfn> given a <span data-x="concept-response">response</span>
<var>response</var>, a <span>media resource</span> <var>resource</var>, and
"<code data-x="">entire resource</code>" or a
(number, number or "<code data-x="">until end</code>") tuple <var>byteRange</var>:</p>

<ol>
<li><p>If <var>response</var> is a <span>network error</span>, then return false.</p></li>

<li><p>If <var>byteRange</var> is "<code data-x="">entire resource</code>", then return
true.</p></li>

<li><p>Let <var>internalResponse</var> be <var>response</var>'s
<span>unsafe response</span>.</p></li>

<li><p>If <var>internalResponse</var>'s <span data-x="concept-response-status">status</span> is
not 206, then return false.</p></li>

<li><p>Let <var>origin</var> be "<code data-x="">rewritten</code>" if
<var>internalResponse</var>'s <span data-x="concept-response-url">URL</span> is null; otherwise
<var>internalResponse</var>'s <span data-x="concept-response-url">URL</span>'s
<span data-x="concept-url-origin">origin</span>.</p></li>

<li><p>Let <var>previousOrigin</var> be <var>resource</var>'s
<span data-x="media-resource-origin">origin</span>.</p></li>

<li>
<p>If any of the following conditions are true:</p>
<ul>
<li><var>previousOrigin</var> is "<code data-x="">none</code>";</li>

<li><var>origin</var> and <var>previousOrigin</var> are "<code data-x="">rewritten</code>";
or</li>

<li><var>origin</var> and <var>previousOrigin</var> are <span data-x="origin">origins</span>,
and <var>origin</var> is <span>same origin</span> with <var>previousOrigin</var></li>
</ul>

<p>then set <var>resource</var>'s <span data-x="media-resource-origin">origin</span> to
<var>origin</var>.</p>

<p>Otherwise, if <var>response</var> is <span>CORS-cross-origin</span>, then return false.</p>

<p>Otherwise, set <var>resource</var>'s <span data-x="media-resource-origin">origin</span> to
"<code data-x="">multiple</code>".</p>

<p class="note">This ensures that opaque responses with range headers do not leak information
by being patched together with other responses from different origins.</p>
</li>

<li><p>Let (<var>start</var>, <var>end</var>) be the result of
<span data-x="extract content-range values">extracting content-range values</span> from
<var>internalResponse</var>.</p></li>

<li><p>If <var>start</var> is not <var>byteRange</var>[0], or if
<var>byteRange</var>[1] is neither "<code data-x="">until end</code>" or <var>end</var>,
return false.</p></li>

<li><p>Return true.</p></li>
</ol>

<p>The <dfn data-x="concept-media-load-resource" export>resource fetch algorithm</dfn> for a
<span>media element</span> and a given <span>URL record</span> or <span>media provider
object</span> is as follows:</p>
Expand Down Expand Up @@ -34739,28 +34810,91 @@ interface <dfn interface>MediaError</dfn> {
</ol>
</li>

<li><p>Let <var>destination</var> be "<code data-x="">audio</code>" if the <span>media
element</span> is an <code>audio</code> element, or "<code data-x="">video</code>"
otherwise.</p>

<li><p>Let <var>request</var> be the result of <span
data-x="create a potential-CORS request">creating a potential-CORS request</span> given
<var>current media resource</var>'s <span>URL record</span>, <var>destination</var>, and the
current state of <span>media element</span>'s
<code data-x="attr-script-crossorigin">crossorigin</code> content attribute.</p></li>

<li><p>Set <var>request</var>'s <span data-x="concept-request-client">client</span> to the
<span>media element</span>'s <span>node document</span>'s <span>relevant settings
object</span>.</p></li>

<li><p>Let <var>byteRange</var>, which is "<code data-x="">entire resource</code>" or a
(number, number or "<code data-x="">until end</code>") tuple, be the byte range required to satisfy
missing data in <span>media data</span>. This value is <span>implementation-defined</span>
and may rely on codec, network conditions or other heuristics. The user-agent may determine
to fetch the resource in full, in which case <var>byteRange</var> would be
"<code data-x="">entire resource</code>", to fetch from a byte offset until the end,
in which case <var>byteRange</var> would be
(number, "<code data-x="">until end</code>"), or to fetch a range between two byte offsets,
im which case <var>byteRange</var> would be a (number, number) tuple representing the two
offsets.</p></li>

<li>
<p>Let <var>destination</var> be "<code data-x="">audio</code>" if the <span>media
element</span> is an <code>audio</code> element and to "<code data-x="">video</code>"
otherwise.</p>
<p>If <var>byteRange</var> is not "<code data-x="">entire resource</code>", then:</p>
<ol>
<li><p>If <var>byteRange</var>[1] is "<code data-x="">until end</code>" then
<span>add a range header</span> to <var>request</var> given
<var>byteRange</var>[0].</p></li>

<p>Let <var>request</var> be the result of <span
data-x="create a potential-CORS request">creating a potential-CORS request</span> given
<var>current media resource</var>'s <span>URL record</span>, <var>destination</var>, and the
<span>media element</span>'s <code data-x="attr-media-crossorigin">crossorigin</code>
content attribute value.
<li><p>Otherwise, <span>add a range header</span> to <var>request</var> given
<var>byteRange</var>[0] and <var>byteRange</var>[1].</p></li>
</ol>
</li>

<p>Set <var>request</var>'s <span data-x="concept-request-client">client</span> to the
<span>media element</span>'s <span>node document</span>'s <span>relevant settings
object</span>.</p>
<li>
<!--FETCH--><p><span data-x="concept-fetch">Fetch</span> <var>request</var>, with
<i data-x="process response">processResponse</i> set to the following steps given
<span data-x="concept-response">response</span> <var>response</var>:</p>

<!--FETCH--><p><span data-x="concept-fetch">Fetch</span> <var>request</var>.</p>
<ol>
<li><p>Let <var>global</var> be the <span>media element</span>'s
<span>node document</span>'s <span>relevant global object</span>.</p></li>

<li><p>Let <var>updateMedia</var> be to <span>queue a media element task</span> given
the <span>media element</span> to run the first appropriate steps from the
<span>media data processing steps list</span> below. (A new task is used for
this so that the work described below occurs relative to the appropriate
<span>media element event task source</span> rather than using the
<span>networking task source</span>.)</p>

<li><p>Let <var>finalize</var> be to <span>finalize and report timing</span> with
<var>response</var>, <var>global</var>, and <var>destination</var>, and call
<var>updateMedia</var>.</p></li>

<li>
<p>Let <var>processEndOfMedia</var> be the following steps:</p>

<ol>
<li><p>Call <var>finalize</var>.</p>

<li><p>If the fetching process has completes without errors, including decoding the
media data, and if all of the data is available to the user agent without network
access, then, the user agent must move on to the <i>final step</i> below.
This might never happen, e.g. when streaming an infinite resource such as web radio, or
if the resource is longer than the user agent's ability to cache data.</p>
</ol>
</li>

<p>The <var>response</var>'s <span>unsafe response</span> obtained in this fashion, if any,
contains the <span>media data</span>. It can be <span>CORS-same-origin</span> or
<span>CORS-cross-origin</span>; this affects whether subtitles referenced in the <span>media
data</span> are exposed in the API and, for <code>video</code> elements, whether a
<code>canvas</code> gets tainted when the video is drawn on it.</p>
<li><p>If the result of <span data-x="verify a media response">verifying</span>
<var>response</var> given the <var>current media resource</var> and
<var>byteRange</var> is false, then call <var>finalize</var>. Otherwise,
<span data-x="body-incrementally-read">incrementally read</span> <var>response</var>'s
<span data-x="concept-response-body">body</span> given <var>updateMedia</var>,
<var>processEndOfMedia</var>, <var>finalize</var>, and <var>global</var>.</p></li>

<li><p>Update the <span>media data</span> with the contents of <var>response</var>'s
<span>unsafe response</span> obtained in this fashion. <var>response</var> can be
<span>CORS-same-origin</span> or <span>CORS-cross-origin</span>; this affects whether
subtitles referenced in the <span>media data</span> are exposed in the API and, for
<code>video</code> elements, whether a<code>canvas</code> gets tainted when the video is
drawn on it.</p></li>
</ol>

<p>The <dfn export id="stall-timeout">media element stall timeout</dfn> is an
<span>implementation-defined</span> length of time, which should be about three seconds.
Expand Down Expand Up @@ -34803,11 +34937,11 @@ interface <dfn interface>MediaError</dfn> {
element's <span>delaying-the-load-event flag</span> to false. This stops <span data-x="delay
the load event">delaying the load event</span>.</p>

<p>The user agent may use whatever means necessary to fetch the resource (within the constraints
put forward by this and other specifications); for example, reconnecting to the server in the
face of network errors, using HTTP range retrieval requests, or switching to a streaming
protocol. The user agent must consider a resource erroneous only if it has given up trying to
fetch it.</p>
<p>Although the above steps give an algorithm for issuing requests, the user agent may use
other means besides those exact ones, especially in the face of error conditions. For
example, the user agent may reconnect to the server or switch to a streaming protocol. The
user agent must only consider the resource erroneous, and proceed into the error branches
of the above steps, if the user agent has given up trying to fetch the resource.</p>

<p>To determine the format of the <span>media resource</span>, the user agent must use the
<span data-x="Content-Type sniffing: video">rules for sniffing audio and video specifically</span>.</p>
Expand All @@ -34817,21 +34951,6 @@ interface <dfn interface>MediaError</dfn> {
given the <span>media element</span> to <span data-x="concept-event-fire">fire an
event</span> named <code data-x="event-media-progress">progress</code> at the element.</p>

<p>The <span>networking task source</span> <span data-x="concept-task">tasks</span> to
process the data as it is being fetched must each <span>immediately</span> <span>queue a
media element task</span> given the <span>media element</span> to run the first appropriate
steps from the <span>media data processing steps list</span> below. (A new task is used for
this so that the work described below occurs relative to the appropriate <span>media element
event task source</span> rather than using the <span>networking task source</span>.)</p>

<p>When the <span>networking task source</span> has <span data-x="queue a
task">queued</span> the last <span data-x="concept-task">task</span> as part of fetching the
<span>media resource</span> (i.e. once the download has completed), if the fetching process
completes without errors, including decoding the media data, and if all of the data is available
to the user agent without network access, then, the user agent must move on to the <i>final step</i> below.
This might never happen, e.g. when streaming an infinite resource such as web radio, or if the
resource is longer than the user agent's ability to cache data.</p>

<p>While the user agent might still need network access to obtain parts of the <span>media
resource</span>, the user agent must remain on this step.</p>

Expand Down

0 comments on commit e1572fb

Please sign in to comment.