Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WebXR Mesh Detection #5791

Merged
merged 9 commits into from
Dec 4, 2023
Merged

Conversation

Maksims
Copy link
Collaborator

@Maksims Maksims commented Nov 5, 2023

Integration of WebXR Mesh Detection API.

Tested on Quest 3. This data on Quest 3 comes from pre-scanned information and added furniture. You can trigger a request for manual room capture, so users can provide refined information.
Includes an example of building geometry and labeling them.
This API design based on planeDetection API.

Test link: https://examples.playcanvas.meta.space/#/xr/ar-mesh-detection

This API can enable many cool AR applications:

  1. Efficient depth based occlusion of virtual geometry.
  2. Physics interactions of virtual objects with mesh that approximates real-world.
  3. Using semantic labels it is possible to identify where screens are, table surfaces, and more.
  4. And much more.

New APIs:

// pc.XrManager
xr.meshDetection // XrMeshDetection interface that provides an access to XrMeshe's
xr.start(camera, pc.XRTYPE_AR, pc.XRSPACE_LOCALFLOOR, {
    meshDetection: true // new option to request mesh detection features
});

// pc.XrMeshDetection
// this class provides an access to meshes.
// XrMesh instances can be added/removed mid-session by underlying system, due to real-time scanning, tracking state changes, manual scanning, etc.
meshDetection.supported // true if mesh detection is supported
meshDetection.available // true if mesh detection is available. This information is only available when session has started.
meshDetection.meshes = [ ]; // an array of XrMesh instances
meshDetection.on('available', () => { }); // fired when mesh detection becomes available
meshDetection.on('unavailable', () => { }); // fired when mesh detection becomes unavailable
meshDetection.on('add', (xrMesh) => { }); // fired when a new XrMesh have been added
meshDetection.on('remove', (xrMesh) => { }); // fired when XrMesh have been removed

// pc.XrMesh
// this class represents a specific XrMesh, its vertices data and attributes
xrMesh.label // string - a semantic label of this mesh
// can be any of: https://github.com/immersive-web/semantic-labels/blob/master/labels.json
// but not limited to this list. e.g. Quest 3 also uses "global mesh".
xrMesh.vertices // Float32Array - local positions of vertex data. This data can change during lifetime of XrMesh, when "change" event fired.
xrMesh.indices // Uint32Array - indices for vertices to construct triangles. This data can change during lifetime of XrMesh, when "change" event fired.
xrMesh.getPosition() // global position of XrMesh (Vec3), this data can change without firing "change" event.
xrMesh.getRotation() // global rotation of XrMesh (Quat), this data can change without firing "change" event.
xrMesh.on('change', () => { }); // fired when any of its attributes: vertices, indices, label have been changed.
xrMesh.on('remove', () => { }); // fired when XrMesh have been removed

Fixes #5699

a95e570cc5a5ba934b551e679016a956.mp4

I confirm I have read the contributing guidelines and signed the Contributor License Agreement.

@Maksims Maksims marked this pull request as ready for review November 5, 2023 17:27
@mvaligursky
Copy link
Contributor

Great PR and the example! Thanks so much @Maksims .
I left a few small comments, but this is pretty much ready to be merged.

@Maksims
Copy link
Collaborator Author

Maksims commented Dec 4, 2023

Made a quick update based on comments, and removed ID as it is nowadays redundant, as Map is widely used for indexes.

@mvaligursky mvaligursky added feature area: xr XR related issue labels Dec 4, 2023
@mvaligursky mvaligursky merged commit 706d7bc into playcanvas:main Dec 4, 2023
7 checks passed
@Maksims Maksims deleted the webxr-mesh-detection branch December 4, 2023 10:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: xr XR related issue feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

WebXR Mesh Detection
2 participants