standardise HTMLMediaElement.seekToNextFrame() and add a new HTMLMediaElement.seekToFrame(index)

[therealadityashankar]

Investigation

I apologize if this issue may not be up to the required standards, it'd be great if someone could give me any pointers on how I could improve this issue

*previous work
Firefox currently has a method HTMLMediaElement.seekToNextFrame which is amazing but not standardized

• brief abstract:
  Problem : try playing a video on the HTML canvas, try to maintain animation synchronization with an animation playing in the HTML5 canvas element

  the current standardized method the render videos in the html canvas, and control the video's current position has certain issues.

  1. the standard rendering method via directly playing a video with the await HTMLVideoElement.play() is not reliable, since the video may buffer in the middle and fail to synchronize with the wished playing state via requestAnimationFrame() in an html canvas, which will end up causing the browser to lose video - canvas animation synchronicity

  2. setting the HTMLMediaElement.currentTime = sometime , is again not reliable since browsers take a longer amount of time to set the current time, than the "appeared" synchronous nature, the "proper" way to check this is via the HTMLMediaElement's "timeupdate" event

  3. offsetting animation to a video element from the canvas can remove the CPU strain required in creating canvas animations and offset it to the GPU, which is especially useful with games using the canvas.getContext("2d") context in the HTMLCanvasElement

• what would this work do if successful?

  • reduce CPU strain in browsers while running video games
  • allow canvas-video synchronization, which again helps video game developmers

• who would be interested/supportive (types of people&orgs, not individuals)?
  video game developers, video streaming companies, people looking to create an inbrowser video editor

• what should be the next steps.
  I think I'd like to know about issues people have faced while rendering videos in an HTMLCanvasElement and about the issues they've faced while trying to maintain timeline synchronization

[tidoust]

Thanks for the input, @therealadityashankar!

A few thoughts:

• I would address "seeking to a specific frame" and "synchronize an overlay with a media frame" separately.
• Some time ago, the Media & Entertainment Interest Group had a rather lengthy discussion on Frame accurate seeking that discussed some of this.
• The same group is currently looking into Media Production use cases, see for instance the summary of February call. W3C may organize a workshop on the topic in a not-so-distant future.
• The WebCodecs proposal is still at early stages but should allow applications to access and manipulate individual video frames, and render them to canvas.
• I note the HTMLVideoElement.requestVideoFrameCallback proposal that could be more useful than requestAnimationFrame for synchronization purpose.
• Efficient use of video frames and canvas contents within the GPU is also under discussion in the GPU for the Web WG, e.g. see Proposal for a GPUVideoTexture object. Some game engines indeed raised that need.

The repository of the Media & Entertainment Interest Group would be a better place to get the attention of "media companies".

[therealadityashankar]

Hi @tidoust, thank you so much for taking the time to reply to my post (I genuinely appreciate it !), there seems to be some standardization on requestAnimationFrameCallback, and I think that standard seeks to do implement all the specifications I seek.

Mozilla said that they're positive to implement this and I've asked for an update on the linked channel,

Thank you so so much Mr François Daoust, I appreciate the time you've taken to respond !