video.currentTime is not precise enough to indentify a frame

[mzur]

Apparently, video.currentTime can be modified/rounded by the browser for security reasons. This means that the value returned by currentTime is not precise enough to identify the video frame. The following observation confirms this issue: If the video is played and paused and then video.currentTime = video.currentTime is set, the displayed video frame is updated most of the time! Try it in this fiddle.

The issue that frame-accurate handling of videos is not really possible in the browser is discussed for a long time. I don't know how we can handle this, yet, but it might become complicated.

[mzur]

See this discussion for how a switch to frame numbers instead of times could be accomplished. However, it's probably hard to keep an independent timer to track the video progress synchronized with the video (e.g. if it buffers, is paused, the playback rate changes etc.).

[mzur]

Also related: w3c/media-and-entertainment#4

[mzur]

PR #444 implements a workaround where the video frame is updated whenever the video is paused. The updated frame exactly shows the frame corresponding to currentTime so annotations placed at this time will always resolve back to the correct frame. The downside is that the video jumps one frame forward or backward whenever the video is paused. Considering the very complex alternatives, I can live with that. I'll leave this open since this issue should be fixed eventually (maybe with updated web APIs?) but I'll lower the priority, as it is no longer pressing.

[mzur]

Tator implements an elaborate video playback mechanism (see discussion linked above) but I believe it relies on custom video re-encoding (with fixed and known parameters). This could be one thing we could copy to get accurate annotation times. Another tool that seems to be quite sophisticated in this regard is DIVE, which we also could take a look at.

[mzur]

Here is an example how a video could be rendered frame by frame using the Web Codecs API: https://stackoverflow.com/a/32708998/1796523

[mzur]

Here is a project that renders videos using the WebCodecs API: https://github.com/aegir-assembly/omni-clip
It uses FFMpeg (WASM) to demux the video files. This could be a solution for our support of many different containers/codecs. WHile the whole of FFMpeg is very large, maybe we can only compile a subset of it that is required for demuxing.

Another thing to consider is that with WebCodecs we can only support remote files with CORS enabled (similar to the image annotation tool).