Cursor pointer API #750

okwasniewski · 2024-01-08T10:41:10Z

This PR describes a new cursor: "pointer" API for iPad and Apple Vision Pro.

I've created this proposal together with @Saadnajmi

A rendered version of the proposal can be read here

NickGerleman · 2024-01-08T15:26:19Z

proposals/0000-hover-style-ipad-api.md

+
+Additionally, We have lots of React Native experiences in brownfield contexts at Microsoft. The value add of React Native over something like a webview is that we can access the system APIs, and provide the best UX for the platform. This includes iPadOS / visionOS, where hover events are now a part of the UX language and built into UIKit controls. In order for React Native experiences to thrive living side by side with native experiences, we would live to have as close UX as possible.
+
+This API can work side by side with the Pointer Events API. Users can choose to either spin up their own custom interactions with the pointer events API, or use Apple OS default. Adding a hover effect shouldn’t affect pointer events being fired or not. https://reactnative.dev/blog/2022/12/13/pointer-events-in-react-native


@javache shared that Vision Pro SDK does not currently allow tracking gaze, beyond declarative hover events, to protect privacy of what the user is looking at. Under that model, pointer events don't seem possible.

To clarify, are you saying the existing pointer events API in React Native won't work in visionOS? The new hover effects that visionOS added seemed to me something separate from the existing APIs, which I thought should still work without issue?

Vincent left more comprehensive information about this below.

NickGerleman · 2024-01-08T15:29:42Z

proposals/0000-hover-style-ipad-api.md

+<Pressable hoverEffect=”lift” onPress={onPressFunction}>
+  <Text>I'm pressable!</Text>
+</Pressable>


Having unprefixed, single platform props for shared components like Pressable is a little bit smelly.

There are various schemes for prefixing and suffixing platform specific props in RN, and also ones we can do that we haven't tried yet (<Pressable apple:hoverEffect="lift">)

This might actually belong on the Platform's View (or VisionOSView instead), if we would ever want to support hover style outside Pressables.

In my head, something like hoverEffectIOS or hoverEffectApple is fine, though I feel it has been somewhat inconsistent in the past ( elevation isn't elevationAndroid, etc..). As implemented in the visionOS fork right now, it is a prop on the base <View>, and <Pressable> just sets a default that isn't none.

I’m curious, what do you call these events for Windows Ink (Surface or Asus Pen)? Or for hand selection tracking in MRTK/HoloLens? I think there’s both prior art and probably a way to (finally) pull some of these offshoots forward in a unified way.

I have no idea what windows would call it 😅

RN Windows solution for this is pretty crummy.

The event shape for custom onMouseEnter, onMouseLeave is effectively exact copy of PointerPoint

So, API is not coherent with other platforms. IIRC it may not even be typed.

Pointer events unifies all of that, for I guess everything but Vision OS requiring declarative handling.

NickGerleman · 2024-01-08T15:33:37Z

proposals/0000-hover-style-ipad-api.md

+
+## Alternatives
+
+- This API might be also implemented using Pointer Events API by detecting hover and applying some custom styling to the view but this would be less performant than using native solution.


RN on Quest ends up doing something like:

Set declarative information on view

In native, on pointer move, show underlay, vibrate controllers, etc on native hit test, before ever dispatching to JS

Some of this is exposed via Animated.event, which is a facility to animate native props when firing en event, before delivered to JS.

vincentriemer · 2024-01-10T21:59:46Z

Thank you so much for putting together this RFC! I’m more or less going to take this opportunity to elaborate on my own alternative to this proposal that I’ve been thinking about for a while now, but didn’t have the bandwidth/priority to put one together myself.

The summary/motivation you’ve described in this RFC is extremely similar to what I was thinking but there are big differences in how we approach the API design. As additional context I am part of the XPlat React/Convergence workstreams focusing on interoperability between platforms including web, and that greatly informs the lens I look through when approaching these APIs.

As the person who is working on implementing Pointer Events, I was immediately curious as to the Vision Pro could take advantage of my work when it was announced. I quickly learned that the underlying eye gaze coordinates are never provided raw to userland code (unless an indirect/direct gesture is underway). This is instead done declaratively either through the APIs mentioned in the parent RFC or by default with Apple’s provided UI elements like UIButton — but this made me curious as to what Apple was doing for websites/Safari.

Turns out Apple may have done the API design for us already! In the WWDC 2023 session titled “Meet Safari for spatial computing” around 4:58 they address this gap by saying that Safari will provide the hover highlight effect 1) on elements with roles that suggest they’re interactive or — most importantly — elements with the CSS style cursor: pointer.

This is the API I think will best provide interoperability between all of the platforms I can think of, and makes the introduction of a new API to the core more palatable as its no longer specific to Apple platforms. Using cursor: pointer will be a clean match with the web and there are cursor APIs on Android and (I’m guessing) Windows/MacOS.

The only rub with this API design is that Apple’s API doesn’t just have one UIHoverEffect that is used, it has multiples and you’ve addressed that in your RFC but mine currently doesn’t. I need to spend the time to do more verification on the Vision Pro’s simulator but I’d guess that Safari assigns the interactive elements on a web page the UIHoverAutomaticEffect and that’s what I believe that React Native should map cursor: pointer too. Here’s a quick video of a prototype I had made taking this approach.

As for the other, more specific effects I think those should be platform specific and we should use “prefixed” values like the web does (at least they used to). The core can consider the cursor StyleSheet property to be a string value and if there are specific cursor styles that don’t fit into the keywords enumerated in the web specification the platform’s underlying view should listen for their own, platform prefixed values (and other platforms can ignore/no-op on values they don’t handle). As an example: on Apple platforms the two explicit, platform specific hover styles are lift and highlight so I think that if we want to expose them as an value for cursor they should be exposed as -rn-apple-lift and -rn-apple-highlight.

To begin with though (and to keep the scope of this RFC small) I’d suggest to at least start with just the cursor: pointer use-case and we can address the platform-specific values later as that as a separate RFC would probably include discussions on auxiliary gaps such as defining multiple/fallback values or having functionality equivalent to @support queries.

Saadnajmi · 2024-01-16T19:21:32Z

@vincentriemer Thanks for the detailed response! I spoke with @okwasniewski offline and I think we both agree that pointer:cursor mapping to the automatic hover effect (or whatever Apple is doing for web) makes sense! A couple of extra thoughts:

React Native macOS (with help from Infinite Red and Meta's Messenger desktop team) previously implemented the cursor style prop, using as much of the W3C names as possible: Add cursor prop to View and Touchables microsoft/react-native-macos#760. If the implementation is fine, maybe the approach we could take is to upstream part of that PR and combine it with Oskars' visionOS/iOS implementation. A caveat is that macOS supports many more cursors than iOS/iPad do, and I have no idea what Android does. Therefore, I think leaving the cursor prop as a string makes sense, with the caveat that maybe only the value pointer is supported in React Native iOS, while other platforms (like macOS/Windows/Web) may support a whole lot more? I don't know how far along your branch / draft is, I'm curious what you think of that?
For macOS, cursor: pointer adds the pointing hand cursor that I see used on web, but not actually in native Appkit experiences much. iPad also sparingly uses hover/pointer effects, I don't think they are intended to be on every button. For that reason, I think it might make sense to leave the prop off by default on all platforms, rather than my original thinking of having Pressable set cursor: pointer by default.

okwasniewski · 2024-01-19T13:02:22Z

Thanks, @vincentriemer @Saadnajmi @NickGerleman @matthargett for great feedback on this RFC. I've updated the proposed solution as @vincentriemer described in his comment.

Saadnajmi · 2024-01-19T19:57:22Z

proposals/0000-hover-style-ipad-api.md

+Here we initialize a new `UIShape` which reassembles the current’s view layer `cornerRadius`. Next we are creating a `UIHoverEffect` instance based on the value passed by user.
+
+The current API implementation might need some polish to correctly retrieve corner radius.


This would require some refactoring, but it would be nice if we could support the full custom border via separate-corner-radiuses-per-corner that RN supports (see RCTBorderDrawing.h). This is easier to do on Fabric than on Paper I think, but should be doable with both.

Saadnajmi · 2024-01-19T19:58:25Z

proposals/0000-hover-style-ipad-api.md

+  - Only Android equivalent I found is: https://developer.android.com/reference/android/view/MotionEvent which doesn’t provide any built-in effect but allows to create custom ones.
+- There might be different plan for the Pointer Events API.


I'd be curious to connect a mouse to an Android phone or tablet and see if the pointer ever changes around the system UI. I know you get a mouse cursor, I can test later. Regardless, this RFC should be apple-only to start IMO

Android has a cursor that's more similar to web than Apple platforms via this method on View. I'd rather limit the RFC to apply to all platforms (at least theoretically) but limit the APIs surface area (like focusing on just pointer: cursor instead of all the different types).

vincentriemer · 2024-01-19T20:37:37Z

React Native macOS (with help from Infinite Red and Meta's Messenger desktop team) previously implemented the cursor style prop, using as much of the W3C names as possible: Add cursor prop to View and Touchables microsoft/react-native-macos#760. If the implementation is fine, maybe the approach we could take is to upstream part of that PR and combine it with Oskars' visionOS/iOS implementation. A caveat is that macOS supports many more cursors than iOS/iPad do, and I have no idea what Android does. Therefore, I think leaving the cursor prop as a string makes sense, with the caveat that maybe only the value pointer is supported in React Native iOS, while other platforms (like macOS/Windows/Web) may support a whole lot more? I don't know how far along your branch / draft is, I'm curious what you think of that?

I think to start with we should say that cursor: pointer and cursor: auto (the implicit value when cursor is unspecified) are the only values officially supported by the core but other platforms can support additional values they choose — though I'd add the caveat that unless the extra value that platform wants to support is listed in the W3C spec, they should prefix the value to avoid any naming/platform conflicts in the future.

For macOS, cursor: pointer adds the pointing hand cursor that I see used on web, but not actually in native Appkit experiences much. iPad also sparingly uses hover/pointer effects, I don't think they are intended to be on every button. For that reason, I think it might make sense to leave the prop off by default on all platforms, rather than my original thinking of having Pressable set cursor: pointer by default.

Agreed. I do think it's worth exploring putting cursor: pointer on Pressable by default but it's going to be potentially messy and ultimately out of scope of this RFC.

okwasniewski · 2024-02-02T12:48:34Z

@vincentriemer I agree that limiting this to cursor: "pointer" and cursor: "auto" and instead having this on all platforms is a better option. How can we move forward with this RFC? I could prepare a PR for iOS side of things. I could also try Android but I am not that familiar with it

Saadnajmi · 2024-02-18T09:30:53Z

Update: Ready for review!
I've started working on an implementation of this RFC in both React Native (for just iOS) and React Native macOS (for visionOS, since macOS already had a cursor prop). Still rough but wanted to have my working branches up as we're already getting asks for this prop internally.

Summary: Implement the cursor style prop for iOS (and consequently, visionOS), as described in this RFC: react-native-community/discussions-and-proposals#750 See related PR in React Native macOS, where we target macOS and visionOS (not running in iPad compatibility mode) with the same change: microsoft#2080 Docs update: facebook/react-native-website#4033 ## Changelog: [IOS] [ADDED] - Implement cursor style prop Pull Request resolved: #43078 Test Plan: See the added example page, running on iOS with the new architecture enabled. This also runs the same on the old architecture. https://github.com/facebook/react-native/assets/6722175/2af60a0c-1c1f-45c4-8d66-a20f6d5815df See the example page running on all three apple platforms. The JS is slightly different because: 1. The "macOS Cursors" example is not part of this PR but the one in React Native macOS. 2. This PR (and exapmple) has went though a bunch of iterations and It got hard taking videos of every change 😅 https://github.com/facebook/react-native/assets/6722175/7775ba7c-8624-4873-a735-7665b94b7233 ## Notes - React Native macOS added the cursor prop to View with microsoft#760 and Text with microsoft#1469 . Much of the implementation comes from there. - Due to an Apple bug, as of iOS 17.4 Beta 4, the shape of the iOS cursor hover effect doesn't render in the correct bounds (but it does on visionOS). I've worked around it with an ifdef. The result is that the hover effect will work on iOS and visionOS, but not iPad apps running in compatibility mode on visionOS. Reviewed By: NickGerleman Differential Revision: D54512945 Pulled By: vincentriemer fbshipit-source-id: 699e3a01a901f55a466a2c1a19f667aede5aab80

vincentriemer

lgtm

Summary: Implement the cursor style prop for iOS (and consequently, visionOS), as described in this RFC: react-native-community/discussions-and-proposals#750 See related PR in React Native macOS, where we target macOS and visionOS (not running in iPad compatibility mode) with the same change: microsoft#2080 Docs update: facebook/react-native-website#4033 ## Changelog: [IOS] [ADDED] - Implement cursor style prop Pull Request resolved: #43078 Test Plan: See the added example page, running on iOS with the new architecture enabled. This also runs the same on the old architecture. https://github.com/facebook/react-native/assets/6722175/2af60a0c-1c1f-45c4-8d66-a20f6d5815df See the example page running on all three apple platforms. The JS is slightly different because: 1. The "macOS Cursors" example is not part of this PR but the one in React Native macOS. 2. This PR (and exapmple) has went though a bunch of iterations and It got hard taking videos of every change 😅 https://github.com/facebook/react-native/assets/6722175/7775ba7c-8624-4873-a735-7665b94b7233 ## Notes - React Native macOS added the cursor prop to View with microsoft#760 and Text with microsoft#1469 . Much of the implementation comes from there. - Due to an Apple bug, as of iOS 17.4 Beta 4, the shape of the iOS cursor hover effect doesn't render in the correct bounds (but it does on visionOS). I've worked around it with an ifdef. The result is that the hover effect will work on iOS and visionOS, but not iPad apps running in compatibility mode on visionOS. Reviewed By: NickGerleman Differential Revision: D54512945 Pulled By: vincentriemer fbshipit-source-id: 699e3a01a901f55a466a2c1a19f667aede5aab80

okwasniewski · 2024-03-17T18:09:12Z

Hey! As this got implemented in facebook/react-native#43078 I think this PR can be merged (?)

Saadnajmi · 2024-03-18T06:17:38Z

While we're on the subject, are there opinions against extending the "pointer-events" prop (not the new w3c api but the old one that controlled touch events) to also apply to cursors? That particular would help with some edge cases related to nested cursors, but I also know it's an old prop and may not be one we want to reuse. I think can be outside the scope of this rfc since the initial implementation already merged.

dlindenkreuz · 2024-06-12T10:13:59Z

@Saadnajmi Great addition!

I haven't seen any discussion about UIPointerInteraction yet, which has a slightly different appearance, is mostly used in iOS system apps, supports actual cursor styles (e.g. crosshair, i-beam) and is supported since iOS 13.4.

See this WWDC 2020 video (timestamped link) for more details.

Does it make sense to add support for UIPointerInteraction along the same lines?

Saadnajmi · 2024-06-12T17:30:44Z

@dlindenkreuz I decided not to try UIPointerInteraction because:

The API was way more complex to implement
It felt like Apple was replacing it with UIHoverStyle?

I didn't realize it allowed the other cursor styles though! I'm not sure it makes sense to have both UIPointerStyle and UIHoverStyle in the same view? Nevermind that more stuff is moving to SwiftUI..

dlindenkreuz · 2024-06-13T08:13:11Z

The API was way more complex to implement

Agree.

There are four effect styles: UIPointerEffect docs

https://developer.apple.com/documentation/uikit/pointer_interactions/integrating_pointer_interactions_into_your_ipad_app

I don't think a CSS rule cursor: pointer can represent the complexity of this system.

It felt like Apple was replacing it with UIHoverStyle?

Looks like UIHoverStyle is a convenient shorthand to UIPointerInteraction – I wouldn't be surprised if it uses a subset of UIPointerInteraction under the hood. Would be great to get >= iOS 13 support as well, maybe by replicating the functionality with UIPointerInteraction? 😬

Initial version of the proposal

9b8b0fc

okwasniewski force-pushed the hover-style-api branch from 7736dbc to 9b8b0fc Compare January 8, 2024 10:45

NickGerleman reviewed Jan 8, 2024

View reviewed changes

feat: update hoverStyle proposal with reviewers suggestions

25c2ec4

Saadnajmi reviewed Jan 19, 2024

View reviewed changes

okwasniewski changed the title ~~Hover style API for iPad / Apple Vision Pro~~ Cursor pointer API Feb 2, 2024

Saadnajmi mentioned this pull request Feb 18, 2024

feat(iOS): Implement cursor style prop facebook/react-native#43078

Closed

vincentriemer approved these changes Mar 5, 2024

View reviewed changes

okwasniewski closed this Jul 31, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cursor pointer API #750

Cursor pointer API #750

okwasniewski commented Jan 8, 2024 •

edited

Loading

NickGerleman Jan 8, 2024

Saadnajmi Jan 8, 2024

NickGerleman Jan 11, 2024

NickGerleman Jan 8, 2024

Saadnajmi Jan 8, 2024 •

edited

Loading

matthargett Jan 14, 2024

Saadnajmi Jan 16, 2024

NickGerleman Jan 19, 2024

NickGerleman Jan 19, 2024

NickGerleman Jan 8, 2024

vincentriemer commented Jan 10, 2024

Saadnajmi commented Jan 16, 2024 •

edited

Loading

okwasniewski commented Jan 19, 2024

Saadnajmi Jan 19, 2024

Saadnajmi Jan 19, 2024

vincentriemer Jan 19, 2024 •

edited

Loading

vincentriemer commented Jan 19, 2024 •

edited

Loading

okwasniewski commented Feb 2, 2024

Saadnajmi commented Feb 18, 2024 •

edited

Loading

vincentriemer left a comment

okwasniewski commented Mar 17, 2024

Saadnajmi commented Mar 18, 2024

dlindenkreuz commented Jun 12, 2024

Saadnajmi commented Jun 12, 2024

dlindenkreuz commented Jun 13, 2024


		Additionally, We have lots of React Native experiences in brownfield contexts at Microsoft. The value add of React Native over something like a webview is that we can access the system APIs, and provide the best UX for the platform. This includes iPadOS / visionOS, where hover events are now a part of the UX language and built into UIKit controls. In order for React Native experiences to thrive living side by side with native experiences, we would live to have as close UX as possible.

		This API can work side by side with the Pointer Events API. Users can choose to either spin up their own custom interactions with the pointer events API, or use Apple OS default. Adding a hover effect shouldn’t affect pointer events being fired or not. https://reactnative.dev/blog/2022/12/13/pointer-events-in-react-native


		## Alternatives

		- This API might be also implemented using Pointer Events API by detecting hover and applying some custom styling to the view but this would be less performant than using native solution.

		Here we initialize a new `UIShape` which reassembles the current’s view layer `cornerRadius`. Next we are creating a `UIHoverEffect` instance based on the value passed by user.

		The current API implementation might need some polish to correctly retrieve corner radius.

		- Only Android equivalent I found is: https://developer.android.com/reference/android/view/MotionEvent which doesn’t provide any built-in effect but allows to create custom ones.
		- There might be different plan for the Pointer Events API.

Cursor pointer API #750

Cursor pointer API #750

Conversation

okwasniewski commented Jan 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Saadnajmi Jan 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vincentriemer commented Jan 10, 2024

Saadnajmi commented Jan 16, 2024 • edited Loading

okwasniewski commented Jan 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vincentriemer Jan 19, 2024 • edited Loading

Choose a reason for hiding this comment

vincentriemer commented Jan 19, 2024 • edited Loading

okwasniewski commented Feb 2, 2024

Saadnajmi commented Feb 18, 2024 • edited Loading

vincentriemer left a comment

Choose a reason for hiding this comment

okwasniewski commented Mar 17, 2024

Saadnajmi commented Mar 18, 2024

dlindenkreuz commented Jun 12, 2024

Saadnajmi commented Jun 12, 2024

dlindenkreuz commented Jun 13, 2024

okwasniewski commented Jan 8, 2024 •

edited

Loading

Saadnajmi Jan 8, 2024 •

edited

Loading

Saadnajmi commented Jan 16, 2024 •

edited

Loading

vincentriemer Jan 19, 2024 •

edited

Loading

vincentriemer commented Jan 19, 2024 •

edited

Loading

Saadnajmi commented Feb 18, 2024 •

edited

Loading