VideoTexture causes texture upload even when videos are paused

[gfodor]

Description of the problem

It looks like the update routine of VideoTexture sets this.needsUpdate = true even when a video is paused. This results in a continual texture transfer to or on the GPU even if the video is not changing.

I quickly patched the function as such:

THREE.VideoTexture.prototype.update = function() {
  const video = this.image;
  const paused = video.paused;

  // Don't transfer textures from paused videos.
  if (paused && this.wasPaused) return;

  if (video.readyState >= video.HAVE_CURRENT_DATA) {
    if (paused) {
      this.wasPaused = true;
    } else if (this.wasPaused) {
      this.wasPaused = false;
    }

    this.needsUpdate = true;
  }
};

And performance significantly improved when videos were paused (particularly when viewing vp8 videos.) Previously, pausing them had no effect. I'm happy to open a PR if the approach above is reasonable.

Three.js version

r103

Browser

• All of them
• Chrome
• Firefox
• Internet Explorer

OS

• All of them
• Windows
• macOS
• Linux
• Android
• iOS

[Mugen87]

If we support HTMLMediaElement.paused, we probably also need to support HTMLMediaElement.ended. Updating the video texture is not necessary if the video has finished playing.

However, all these changes make the update() method more complex although not all users will require these checks. Right now, I tend to not introduce such modifications to the core and let the user monkey-patch the method if required.

[gfodor]

OK -- for those who do end up pausing the videos, the impact is significant -- particularly with vp8/9 encoded videos. Is the complexity concern here around performance? I believe it'd be possible to set up an event driven approach to capture state changes to the video to result in less branching if so.

[Mugen87]

I believe it'd be possible to set up an event driven approach to capture state changes to the video to result in less branching if so.

How would this look like?

[gfodor]

I was just referring to the fact that the media elements have events you can attach to be notified of pause/resume/etc, so (probably with a decent amount of incidental complexity) you could remove the net impact to the runtime of update by using event handlers to manage the state needed

[gfodor]

if having three.js manage a bunch of event handlers on video elements associated with video textures seem worth the benefits of reducing texture transfers for paused/ended videos then I'm happy to dig into it -- but not sure if that would pass muster.

[gfodor]

also part of the complexity here is that you want to draw the texture at least once after the video is paused or ends, because for videos that start out that way you end up never copying anything

[Mugen87]

Since #19906 (which landed in r119), requestVideoFrameCallback() is used by VideoTexture. The good thing is that the callback does not fire when the video is paused. Meaning no unnecessary assignments to needsUpdate happen and thus no respective texture uploads anymore.

[bryantcodesart]

For those getting here via Google...
While this fix is definitely the correct forward-looking solution, the browser support for requestVideoFrameCallback is not perfect. Works amazing in Chrome, but no support in Firefox, and Safari just arrived to the party--so there's plenty of people running Safari browsers without this feature at time of writing (including me before I just updated). https://caniuse.com/?search=requestVideoFrameCallback. The VideoTexture class appears to fall back to updating every frame if requestVideoFrameCallback is not supported. (Which is a totally reasonable and fine choice, gotta draw the line somewhere.)

BUT

I was pulling my hair out trying to figure out why I was having performance issues on Safari with many paused VideoTextures and incorrectly thinking that it couldn't be the issue mentioned here, because this thread makes it seem totally fixed.

Unless Im mistaken (super possible, correct me!):
If you need this performance boost with widespread browser support in late 2022, I think you still need to hack the VideoTexture class to not update when your vid is paused... for now! (Even with latest three version, I saw massive performance gains on these browsers doing basically what @gfodor did.)