Support reconstructing the length and cues in a webm

[yell0wd0g]

Hi! After w3c/mediacapture-record#119 (comment) I have been playing with emscripten-compiling libwebm, but I found this repo that might be able to produce the same result using more succinct code. The problem is essentially that MediaRecorder produces files that are not seekable, one because they have no Duration and second, perhaps, because they have no Cues. Is it something you could help with?

[legokichi]

Please use the EBMLReader class and tools.putRefinedMetaData function. I wrote this to solve the problem.

Here is the example code to insert Duration and SeekHead element.

example: https://github.com/legokichi/ts-ebml/blob/88beee06d5dad0e36c9b082d7da2d7eb227ca3b1/src/seekable_example.ts
demo: https://jsfiddle.net/1nrx33gd/

[legokichi]

related issues:
https://bugzilla.mozilla.org/show_bug.cgi?id=969290
https://bugs.chromium.org/p/chromium/issues/detail?id=569840
https://bugs.chromium.org/p/chromium/issues/detail?id=606000

[legokichi]

https://www.matroska.org/technical/order/index.html

It is much easier to put the Cues at the "end" of the segment, once all the Clusters have been written, otherwise it's hard to predict beforehand the place to reserve at the beginning of the segment. It is also not a big deal if Cues are at the end, given when the user wants to seek in a Matroska stream, it is going to jump somewhere, so it can seek to the Cues entry, read it and then seek to the best position according to the Cues entries. So the Cues should always be written after the Clusters. However the Cues could also appear at the front. In this case the size of the Cues is usually very small compared to the video+audio bitrate of the stream. Even on small bitrate for a full length movie, the whole cue size represents only 0.01 to 0.1 second of download. Which is smaller than the time needed to seek on the network.

Is it not necessary to write the position of Cue in SeekHead at the top of the file?

[yell0wd0g]

Matroska might allow Cues and SeekHead at the end, but WebM doesn't, I believe: https://www.webmproject.org/docs/container/#muxer-guidelines.

[yell0wd0g]

The most interesting Cr bug is https://crbug.com/642012, but IMHO we should not try to complicate the c++ code and instead should try to do this as a polyfill, as I argued in w3c/mediacapture-record#119.

That aside, seekable_examples.js is awesome!! I wonder if we could publish it as a node package? The current implementation is thought of as a wrapper around MediaRecorder by instrumenting ondataavailable, but would it be possible to operate on a single recording Blob? Like a post-processing step, essentially.

[legokichi]

I published in @ 1.4.0.

usage

import EBMLReader from 'ts-ebml/lib/EBMLReader';
import {Decoder, Encoder, tools} from "ts-ebml";
import * as EBML from "ts-ebml";

main("foo.webm");

async function main(file: string){
  const res = await fetch(file);
  const webm_buf = await res.arrayBuffer();
  const elms = new Decoder().decode(webm_buf);
  
  let metadataElms: EBML.EBMLElementDetail[] = [];
  let metadataSize = 0;
  let last_duration = 0;
  const cluster_ptrs: number[] = [];
  const reader = new EBMLReader();
  reader.logging = true;

  reader.addListener("metadata", ({data, metadataSize: size})=>{
    metadataElms = data;
    metadataSize = size;
  });

  reader.addListener("cluster_ptr", (ptr)=>{
    cluster_ptrs.push(ptr);
  });

  reader.addListener("duration", ({timecodeScale, duration})=>{
    last_duration = duration;
  });
  
  elms.forEach((elm)=>{ reader.read(elm); });
  reader.stop();

  const refinedMetadataElms = tools.putRefinedMetaData(metadataElms, cluster_ptrs, last_duration);
  const refinedMetadataBuf = new Encoder().encode(refinedMetadataElms);
  const body = webm_buf.slice(metadataSize);

  const raw_webM = new Blob([webm_buf], {type: "video/webm"});
  const refinedWebM = new Blob([refinedMetadataBuf, body], {type: "video/webm"});

  const raw_video = await fetchVideo(URL.createObjectURL(raw_webM));
  const refined_video = await fetchVideo(URL.createObjectURL(refinedWebM));
  
  document.body.appendChild(raw_video);
  document.body.appendChild(refined_video);
}


function fetchVideo(src: string): Promise<HTMLVideoElement>{
  return new Promise((resolve, reject)=>{
    const video = document.createElement("video");
    video.src = src;
    video.controls = true;
    video.onloadeddata = ()=>{
      video.onloadeddata = <any>null;
      resolve(video);
    };
    video.onerror = (err)=>{
      video.onerror = <any>null;
      reject(err);
    };
  });
}

But this method of inserting SeekHead does not work well with Firefox.
The reason is unknown.
Please try playing this file in Firefox.

https://github.com/legokichi/ts-ebml/blob/f5c7633a8de2a31a4e981edea983abd5f3e9e980/test/firefox55nightly.refined.seekhead.webm

[legokichi]

https://bugzilla.mozilla.org/show_bug.cgi?id=1356833

After all it was my mistake, the file was broken.

here is the new demo: https://jsfiddle.net/1nrx33gd/1/
and new example code: https://github.com/legokichi/ts-ebml/blob/07e7ffa517018e511d95dc54dbf3526e9002e0c6/src/example_seekable.ts

[yell0wd0g]

This is awesome! Thanks!
I'm working on updating my demo code with the example in this comment and the ts-ebml 1.4.0 version.

[vhmth]

@miguelao keep in mind that the WebM spec says that it does not recommend <Cue>s living at the end of a file. The truth is that most byte-range-request-aware players will attempt to grab the last few bytes of a file if it can't find enough metadata. This is the case for all major browsers that I know of.

Additionally, if you run a non-seekable WebM file through and ffmpeg copy op, it will place the <SeekHead> at the front and <CueList> at the end. Not trying to influence the implementation you wish to go ahead with. I definitely think that putting this information before all clusters is the way to go for players that do not have the intelligence to seek to the end of a file.

[legokichi]

update: I stopped adding Cluster to SeekHead and added it to Cues.

• demo: https://jsfiddle.net/ub2jej7c/
• source: https://github.com/legokichi/ts-ebml/blob/54d4917b66936c2370bafd5ae796373f302b9ae8/src/example_seekable.ts

see also

• Update the handling of metadata refinement #3
• 4b81ec6