Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

react-native gesture responders can't be nested #3332

Open
ashkalor opened this issue Aug 11, 2024 · 11 comments
Open

react-native gesture responders can't be nested #3332

ashkalor opened this issue Aug 11, 2024 · 11 comments

Comments

@ashkalor
Copy link

Problem

In react native once a view becomes a Gesture Responder, canvas elements stop gettings events. We can use events on canvas elements as long as no other view encompassing the canvas becomes a Gesture Responder.

What would really help in this situation.

Prioritize events for canvas elements and only set onStartShouldSetResponder and onMoveShouldSetResponder when meshes dont have a hit i.e onPointerMissed, this way we can interact with the 3d scene easily and only when pointers are missed we can choose to use libraries like
r3f-orbit-controls

Current Work around

So right now I am able to get some semblance of this working by doing what I have shared below, however this has a major drawback
that one touch event is lost in trying to set GestureResponder to start.

const startShouldSetResponder = useRef<boolean>(false);
  const moveShouldSetResponder = useRef<boolean>(false);
  const [OrbitControls, events] = useControls();
  return (
    <Canvas
      style={{
        flex: 1,
      }}
      onLayout={events.onLayout}
      onPointerMissed={(event) => {
        console.log("Pointer Missed");
        startShouldSetResponder.current = true;
        moveShouldSetResponder.current = true;
      }}
      onResponderRelease={() => {
        console.log("Canvas click");

        events.onResponderRelease();
        startShouldSetResponder.current = false;
        moveShouldSetResponder.current = false;
      }}
      onResponderMove={(event) => {
        console.log("Canvas move");
        events.onResponderMove(event);
      }}
      onStartShouldSetResponder={(event) => {
        events.onStartShouldSetResponder(event);
        return startShouldSetResponder.current;
      }}
      onMoveShouldSetResponder={(event) => {
        events.onMoveShouldSetResponder(event);
        return moveShouldSetResponder.current;
      }}
    >
      <OrbitControls />
      <ambientLight intensity={0.1} />
      <directionalLight color="red" position={[0, 0, 5]} />
      <mesh
        onClick={() => {
          console.log("Mesh Click");
        }}
      >
        <boxGeometry />
        <meshStandardMaterial />
      </mesh>
    </Canvas>
  );

Possible Solutions

  1. Get GestureResponder event in onPointerMissed so that we can use that event and call the different callbacks without losing that one touch event so that we can start using orbit controls directly.
  2. Create a fake GestureResponderEvent from MouseEvent and do what i mentioned above however I had limited success in getting that to work (Maybe I have knowledge gaps that is causing me to miss something very obvious)
  3. Use an Event State Machine for Native Events as trying to handle propagation and bubbling effectively in native might not be the best idea, inspired from this video by Andy Matuschak. For this i think integrating with React native gesture handler might be extremely fruitful
    as they seem to handle these cases very well. (Currently using GestureDetector from this library inside canvas causes the app to crash)

And Finally I would really like to thank @CodyJasonBennett, @TiagoCavalcante and other contributors in this ecosystem for all the work these guys put in for making something like this even remotely possible for react native.

Packages used

 "@react-three/drei": "^9.109.5",
 "@react-three/fiber": "^8.17.5",
 "r3f-native-orbitcontrols": "^1.0.12",
 "react-native-gesture-handler": "~2.16.1",
@ashkalor ashkalor changed the title React Native Events System with Orbit Controls react-native Events System with Orbit Controls Aug 11, 2024
@CodyJasonBennett
Copy link
Member

CodyJasonBennett commented Aug 11, 2024

In react native once a view becomes a Gesture Responder, canvas elements stop gettings events. We can use events on canvas elements as long as no other view encompassing the canvas becomes a Gesture Responder.

I'm not observing this on the latest version, but note #3252 was just released in 8.17 which has OrbitControls from Drei work OOTB, and allows you to spread r3f-native-orbitcontrols directly over the R3F Canvas. We still don't implement pointer capture right (#3315), so you may want to keep with that library until we reach compliance with web events.

@ashkalor
Copy link
Author

ashkalor commented Aug 11, 2024

Hey
All my observations were made on the latest version. I remember trying orbit controls yesterday and I tried it again right now, only rotation seems to be working any other gesture makes the cube disappear, below is my implementation for reference. If it is of any help this is the same behaviour i noticed when i tried emulating a gesture responder event from onPointerMissed mouse event.

<Canvas
      style={{
        flex: 1,
      }}
      onPointerMissed={() => {
        console.log("Pointer Missed");
      }}
      onPointerEnter={() => {
        console.log("Pointer Enter");
      }}
      onPointerLeave={() => {
        console.log("Pointer Leave");
      }}

    >
      <OrbitControls />
      <ambientLight intensity={0.1} />
      <directionalLight color="red" position={[0, 0, 5]} />
      <mesh
        onClick={() => {
          console.log("Mesh Click");
        }}
      >
        <boxGeometry />
        <meshStandardMaterial />
      </mesh>
</Canvas>

Is this because we haven't implemented pointer capture?

The issues i mentioned are still true otherwise, If we implement any kind of gesture responder on an encompassing view canvas elements don't receive events.

There is also this recurring error in the latest version

Screenshot_20240811_182528_sample

@CodyJasonBennett
Copy link
Member

All my observations were made on the latest version. I remember trying orbit controls yesterday and I tried it again right now, only rotation seems to be working any other gesture makes the cube disappear, below is my implementation for reference.

Why would the cube disappear? Is that with code that is actually effectful and doesn't just print a log as you shared?

Is this because we haven't implemented pointer capture?

No, that's a separate issue where R3F doesn't implement pointer capture on native.

The issues i mentioned are still true otherwise, If we implement any kind of gesture responder on an encompassing view canvas elements don't receive events.

I see, it's as you said, where gesture responders don't support nesting of "interactive" views or surfaces. Is this a known issue with react-native? I'm not happy to hack around such an invasive issue as our options are naturally very limited since now we're sensitive to user-land component structure.

There is also this recurring error in the latest version

This is very likely unrelated to R3F, but I'd need to see an example. I incidentally maintain the renderer in three also, so I can guess there is a critical crash during material compilation which is either a bug in expo-gl or something more critical with the device driver.

@ashkalor
Copy link
Author

ashkalor commented Aug 11, 2024

Why would the cube disappear? Is that with code that is actually effectful and doesn't just print a log as you shared?

Not really, it is exactly same as the code I shared.
I am thinking it has something to do with handling multiple touches but I am really not sure. I would have to hack orbit controls to see what is happening under the hood.

I see, it's as you said, where gesture responders don't support nesting of "interactive" views or surfaces. Is this a known issue with react-native? I'm not happy to hack around such an invasive issue as our options are naturally very limited since now we're sensitive to user-land component structure.

They do support nesting of gesture responders but only one gesture responder can be active at a time. In our case i believe since canvas elements don't support gesture responder props all the touches are blocked at canvas level.

This is very likely unrelated to R3F, but I'd need to see an example. I incidentally maintain the renderer in three also, so I can guess there is a critical crash during material compilation which is either a bug in expo-gl or something more critical with the device driver.

This is an example repo with code.

Let me know if you need any additional help.

What are your thoughts on react native gesture handler they have a very declarative syntax and they seemed to have solved these kind of problems before. If we could provide support for this native canvas it would make so much more things possible.

@CodyJasonBennett
Copy link
Member

This is an example repo with code.
Let me know if you need any additional help.

This is likely very device-dependent. Can you create another issue regarding this and note which device/OS you're seeing this on?

I can create a vanilla example if this is indeed unrelated to R3F or events. I'm still worried that it is somehow.

What are your thoughts on react native gesture handler they have a very declarative syntax and they seemed to have solved these kind of problems before. If we could provide support for this native canvas it would make so much more things possible.

I'm wary of bringing in native dependencies if they complicate the install process. Still, I only see fail cases if we only stick with react-native, so I'm eager to try it anyway.

We used to bring in Pressability from react-native internals prior to #2985. Maybe that would have equal effect, just breaking support for react-native-web.

@ashkalor
Copy link
Author

This is likely very device-dependent. Can you create another issue regarding this and note which device/OS you're seeing this on?

Done in #3333

I'm wary of bringing in native dependencies if they complicate the install process. Still, I only see fail cases if we only stick with react-native, so I'm eager to try it anyway. We used to bring in Pressability from react-native internals prior to #2985. Maybe that would have equal effect, just breaking support for react-native-web.

They seem to have good support for web OOTB. On the whole if both canvas and canvas elements can use GestureDetector from react-native-gesture-handler to handle gestures then I believe your initial goal to support a library like use-gesture with react native will be satisfied.

@CodyJasonBennett CodyJasonBennett changed the title react-native Events System with Orbit Controls react-native gesture responders can't be nested Aug 26, 2024
@CodyJasonBennett
Copy link
Member

Just put out a patch for that crash on Android #3341.

In react native once a view becomes a Gesture Responder, canvas elements stop gettings events. We can use events on canvas elements as long as no other view encompassing the canvas becomes a Gesture Responder.

I'm not sure if this should be true after reading https://reactnative.dev/docs/gesture-responder-system#capture-shouldset-handlers. If onStartShouldSetResponderCapture or onMoveShouldSetResponderCapture are implemented in a parent view, then it blocks interaction from the R3F canvas. Omitting those fields from parents should have nested responders play nice with each other. Is this consistent with your observations? This might be working as intended from react-native's side.

@CodyJasonBennett
Copy link
Member

All my observations were made on the latest version. I remember trying orbit controls yesterday and I tried it again right now, only rotation seems to be working any other gesture makes the cube disappear, below is my implementation for reference.

pmndrs/drei#2067 is possibly related where controls integrates undefined/NaN from an unimplemented property.

@zxffffffff
Copy link

I got the same problem, this is my code(create-expo-app@latest + basic example 2 box) , npm run web

import { useRef, useState } from 'react'
import { Canvas, MeshProps, useFrame } from '@react-three/fiber/native'
import { OrbitControls } from '@react-three/drei/native'
import { Mesh } from 'three';

    <ParallaxScrollView ...>
      ...
      <ThemedView style={{ height: 300 }}>
        <Canvas >
          <ambientLight intensity={Math.PI / 2} />
          <spotLight position={[10, 10, 10]} angle={0.15} penumbra={1} decay={0} intensity={Math.PI} />
          <pointLight position={[-10, -10, -10]} decay={0} intensity={Math.PI} />
          <Box position={[-1.2, 0, 0]} />
          <Box position={[1.2, 0, 0]} />
          <OrbitControls />
        </Canvas>
      </ThemedView>
      ...
    </ParallaxScrollView>

console log (mac m2 chrome)

...
onPointerDown
onPointerOver
onPointerMove
onPointerOut
# the cube disappear

the version

"expo": "~51.0.28",
"expo-gl": "~14.0.2",
"react": "18.2.0",
"react-native": "0.74.5",
"react-native-web": "~0.19.10",
"three": "^0.168.0"
"@react-three/fiber": "^8.17.6",

@CodyJasonBennett
Copy link
Member

Probably related to pmndrs/drei#2067. Can you try disabling zoom?

@zxffffffff
Copy link

Probably related to pmndrs/drei#2067. Can you try disabling zoom?

It works :)

<OrbitControls enableZoom={false} />

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants