@react-spring/three useSpring stops updating in XR session

[ffdead]

🐛 Bug Report

Springs stop updating when starting an active XR session.

Not sure if this a bug related to @react-spring/three, @react-three/fiber, @react-three/xr or a combination of all three.

To Reproduce

1. Load repro link below on an Oculus Quest 2
2. The mesh changes scale on a timer ever 1s - this is animated with a spring
3. Start a VR session
4. The mesh stops animating scale and stays at the scale it was when the xr session was started

Expected behavior

The mesh should continue to animate scale inside the XR session.

Link to repro (highly encouraged)

https://codesandbox.io/s/react-xr-react-spring-bug-fkzt3?file=/src/index.tsx

Environment

• @react-spring/three >= v9.0.0
• @react-three/fiber >= v6.0.3
• @react-three/xr >= v2.1.0
• react v17.x

Works as expected with the following old versions:

• @react-spring/three v9.0.0-rc.3
• react-three-fiber v5.x.x
• @react-three/xr v2.0.1

[joshuaellis]

How do you 'start a VR session' without a VR Kit? 😕

[ffdead]

How do you 'start a VR session' without a VR Kit? 😕

I'm afraid you don't. Maybe this issue is better handled in https://github.com/pmndrs/react-xr?

(Side note, if you are interested, consider getting a https://www.oculus.com/quest-2/ - they are really cheap and great for WebXR)

[joshuaellis]

Side note: a friend has one I was supposed to borrow a while back! annoying I never got it.

I'm really sorry, I simply can't help actually look into this issue. I would be surprised if this were an issue with react-spring because it works fine with regular WebGL, i'm also happy to be proven completely wrong.

You could open an issue in react-xr sure! Equally, if you were interested in looking into I'm happy to support you however I can // answer questions etc. 👍🏼

[CodyJasonBennett]

There are various WebXR polyfills and browser extensions that you can use to test this behavior.

A notable restriction of a session would be the lack of browser APIs (see WebGLRenderer#setAnimationLoop) which would be my immediate suspicion, assuming this is isolated to VR.

I don't mind taking a look as well and/or verifying behaviors if that helps, but the soonest I'll be able to pickup a headset will be late Thursday (US central).

[ffdead]

Thanks for your input @CodyJasonBennett. I tried the browser extension emulator and it does not show the issue, unfortunately.

I agree that it must have something to do with the animation loop, however, what is weird is that in my test I subscribe to r3f addEffect in a similar way that react-spring runs its frame loop. And the box keeps moving from side to side in VR which means at least the r3f frame loop is still running in VR.

I have been trying to pinpoint where the issue started to happen as I used this for project earlier this year. This is the exact combination of library version I used back then and the spring is working in VR:
react-spring@9.0.0-rc.3 https://codesandbox.io/s/react-xr-react-springthree-v9-rc3-wx24v
What is noticeable is that the r3f frame loop seems to run twice as fast, at 120 FPS (two instances of the loop running?)
If I remove the react-spring import from this sandbox, the FPS drops to a normal 60fps again.

The version before this, however, did not work in VR:
react-spring@9.0.0-rc.2 https://codesandbox.io/s/react-xr-react-springthree-v9-rc2-hzj80
This runs at a normal 60fps.

Could this mean that the only reason it ever worked in 9.0.0-rc3 is because of the double frame loop bug? And this was fixed in 9.0.0 and later. In any case, it's not working in all the latest versions and that is what counts 🤷

I should mention I have only tried this on an Oculus Quest 2 so far, as I'm not in the office this week.

[joshuaellis]

IIRC, we let r3f drive the render loop for animation in spring, it could be something to do with that.... 🤔

[driescroons]

I ran into the same issue for my WebXR game on my Quest 2. I don't seem to be having the problem om my valve index, using steamVR and chrome

I've recorded a video with both my valve index and my oculus quest 2.
https://imgur.com/lqhnkG5

As you can see, the buttons on my valve index have a hover effect and the level loads in as expected, but when doing the same thing on my quest the useSpring effects do not seem to start. When I quit the application and reopen the WebXR experience again, the state has been updated and is showing the to effect.

I recently migrated to all newest versions and noticed the issue. I've updated your codesandbox with the packages I was using before and can confirm that these versions do work in the oculus browser:
https://codesandbox.io/s/react-xr-react-spring-bug-forked-98v38?file=/src/index.tsx

@react-spring/three@9.1.2 => @react-spring/three@9.0.0-rc3
@react-three/fiber@6.1.5 => react-three-fiber@6.0.13

I did not have to change the react-xr version because that did not seem to cause the issue.

Could this mean that the only reason it ever worked in 9.0.0-rc3 is because of the double frame loop bug? And this was fixed in 9.0.0 and later. In any case, it's not working in all the latest versions and that is what counts 🤷

Would be super weird that we both used the same version, got it to work, and stuck with it @ffdead 😃 I also wasn't able to get it working in other versions.

If you need me to test something on oculus / valve index; let me know!

[ffdead]

I ran into the same issue on my Quest 2. I don't seem to be having the problem om my valve index, using steamVR and chrome

Great that you were able to replicate it! Sounds like the issue could be with the Oculus Quest 2 browser in that case which is super weird since R3F and XR seems to work fine in general.

Would be super weird that we both used the same version, got it to work, and stuck with it @ffdead 😃 I also wasn't able to get it working in other versions.

Yes, it's a bit weird. I've also mostly been using HTC Vive / Oculus Rift S in the past which makes it even more baffling that I just happened to pick the exact versions that also worked on Oculus Quest 🤯

@joshuaellis do you know if @react-spring/three only relies on the r3f addEffect here to run the frameloop or if something else could cause it to be blocked?
I have attached an addEffect as a test in my sandbox that moves the box horizontally, and it keeps running fine in XR even though the spring animation is dead.

[ffdead]

I spent a few hours remote debugging my Oculus Quest 2 browser. This is what I could see:

No active XR session

package	function	is executed?
react-spring/three	FrameLoop.advance	YES
react-spring/shared	FrameLoop.advance	YES
react-spring/core	SpringValue.advance	YES

In active XR session

package	function	is executed?
react-spring/three	FrameLoop.advance	YES
react-spring/shared	FrameLoop.advance	NO
react-spring/core	SpringValue.advance	NO

I'm not familiar with the internals of react-spring so I'm not sure what that means. I was a bit surprised to see separate frame loop logic for each target tbh. And I'm not sure I understand where the shared FrameLoop gets called from the three FrameLoop.

Hope this helps nail down the issue. I will wait for someone who actually knows the internals of react-spring to comment 😅

[joshuaellis]

Yeah when we released v9 there was some form of issue with three frame looping that we monkey patched with the old FrameLoop system...

SpringValue.advance is triggered by the render loop. So what we're identifying is an issue in react-spring/shared. the fact it stops running means that advance function is not returning true. We use rafz internally and if you return true it loops.

I think what we should focus on first is getting three using the shared loop. It will possibly break further but it'd potentially be easier to figure out the issue from there.

[joshuaellis]

Also, great investigation so far! ⭐

[ffdead]

Ok, here's what I think is going on:

The main issue

• window.requestAnimationFrame doesn't run in an XR session on the Oculus Quest browser. This is expected as we need to use WebGLRenderer.setAnimationLoop to run animations in XR.

• The three target's FrameLoop is not being used at all currently.
  The solution is to let r3f drive the frame loop which uses WebGLRenderer.setAnimationLoop internally. The three target's FrameLoop is hooked up to r3f via addEffect properly, but the FrameLoop itself is not being used to drive the springs.

• Globals.assign({ frameLoop }) is a noop because:

  1. shared Globals don't actually contain an assign or export for frameLoop
  2. All components that use the FrameLoop import it from import { frameLoop } from '@react-spring/shared' instead of something like Globals.frameLoop
  3. Some components like withAnimated use import { raf } from '@react-spring/shared' directly without using the FrameLoop to schedule the frames.

Temporary workaround

My temporary workaround is to monkey patch the three target to run rafz frames via r3f addEffect. I didn't find a way to request only 1 frame from r3f so I cache the current frame callback in a temporary variable.

// YourApp.js
import { addEffect } from '@react-three/fiber'
import { Globals } from '@react-spring/shared'
let nextFrame: Function | undefined = undefined
addEffect(() => {
  if (nextFrame) nextFrame()
  return true
})
Globals.assign({
  requestAnimationFrame: (cb) => (nextFrame = cb)
})

Workaround sandbox

Problems with the workaround

• Only one frame callback is saved. This might cause issues if we have multiple rafz frames scheduled for each r3f frame, but hopefully, that's not happening.

• r3f invalidate needs to be called manually while springs are active. It's important that apps can stop the r3f render loop when not needed to preserve battery / performance by passing <Canvas frameloop="demand"/>

• the rafz internal loop seem to run all the time which would cause r3f to invalidate continuously if we did something like this:

requestAnimationFrame: cb => {
 nextFrame = cb;
 invalidate();
},

Conclusions

1. The three target's FrameLoop is not being used currently so it could be removed. The issue mentioned with three frame loop should not have been solved by adding the current FrameLoop, is the issue still there?
2. All components that need a frame should go via the FrameLoop - never use rafz directly since it relies on window.requestAnimationFrame.

This feels like fundamental change so I suspect a core maintainer would be better suited to make the necessary changes.

[joshuaellis]

Thanks for the research! It's very helpful ⭐ it's also not easy to resolve (but when is it?).

So it feels like step1 is getting everything to use a set global AnimationLoop function e.g in the context of web this would be rafz. Step 2 would be to be able to swap this default global AnimationLoop function out for a func that can run as an effect in r3f...

[joshuaellis]

Cool, so after some digging this weekend, it turns out that r3f was not driving the animation of react-spring when using @react-spring/three. So that made it easier to remove the FrameLoop class from @react-spring/three.

I then looked at the code for rafz and there's only two places we call the native RequestAnimationFrame in the start function and the loop. Taking inspiration from the work I did with react-three-fiber around frameLoop and when to call it, i've decided to pull this into rafz so that @react-spring/three will change rafz.frameLoop to demand, this will mean nothing using rafz will run unless we manually advance the timer, we can do this by adding an effect function to r3f, since r3f chooses the correct loop function depending on if its xr or not, this should work. Keep eyes pealed for the PR into rafz and the fix for react-spring!

Edit: I want to say a real thank you to @ffdead, great research helped isolate the issue and wrote up incredibly clearly so I could think about how to solve it 🙌🏼

[joshuaellis]

this has been released on v9.2.2 in theory it should all be working, but we can reopen if it's not 🙏🏼

[ffdead]

I just tried it out and it seems to run great inside XR on the Oculus Quest. Awesome job, big thanks!