Not able to highlight floor of a 3D building on click of 3D floor label(three js sprite object).

[gireeshbhogireddy]

I was trying to highlight floor of a 3D building on click of floor label.
As mapbox gl does not have possibility for 3D label, I have added label with the help of three js. Able to add floor labels but not able to raise onclick event for three js sprite label.
I tried raycaster method to find the clicked label from this reference (https://threejs.org/docs/#api/en/core/Raycaster).
But it is not working properly, it is working at particular tilt of camera and also not working for every label.
How to raise click event for labels properly in every angle? . Kindly help me.

Thanks,
Gireesh

[rreusser]

Hi, @gireeshbhogireddy. I'd love for this to work more easily, but at the moment, I think the answer is that the THREE.js raycaster doesn't work with the combined projection*view matrix Mapbox provides. Since Mapbox deals with tiles and scaling in a very particular way, it doesn't exactly use a single, cleanly separated projection and view matrix as is common in computer graphics.

As far as THREE.js goes, this line points out that the raycaster doesn't work with a generic Camera instance:

https://github.com/mrdoob/three.js/blob/e48fc94dfeaecfcbfa977ba67549e6108b370cbf/src/core/Raycaster.js#L89

I've created a codepen which I think would function if a PerspectiveCamera with its projectionMatrix set to the matrix Mapbox provides and its matrixWorldInverse set to the identity were valid input to the raycaster. Rendering works, but it seems that the raycaster does not. I don't know whether that means the camera needs to be specified by its location, direction, fov, aspect ratio etc, or whether setting matrixWorldInverse and projectionMatrix separately would be adequate.

https://codepen.io/rsreusser/pen/XWpxWyv?editors=0010

At the moment, I think the only answer I can offer is that Mapbox's custom layer matrix output is not fully compatible with three.js's PerspectiveCamera.

[astojilj]

As mentioned, the issue is related to Raycaster initialization and different coordinate spaces used.

I've forked @rreusser example to https://codepen.io/astojilj/pen/jOyQNLW?editors=0010 and added one more cube for raycast verification.

Untitled.mov

I'll use raycast code to explain coordinate spaces and transformations:

    var m = new THREE.Matrix4().fromArray(matrix);
    var l = new THREE.Matrix4()
      .makeTranslation(
        modelTransform.translateX,
        modelTransform.translateY,
        modelTransform.translateZ
      )
      .scale(
        new THREE.Vector3(
          modelTransform.scale,
          -modelTransform.scale,
          modelTransform.scale
        )
      );
    
    this.camera.projectionMatrix = m.clone().multiply(l);
    this.camera.matrixWorldInverse = new THREE.Matrix4();
    this.renderer.resetState();
    const freeCamera = this.map.getFreeCameraOptions();
    let cameraPosition = new THREE.Vector4(freeCamera.position.x, freeCamera.position.y, freeCamera.position.z, 1);
    cameraPosition.applyMatrix4(l.invert());
    let direction = mouse.clone().applyMatrix4(this.camera.projectionMatrix.clone().invert());
    direction.divideScalar(direction.w);
    this.raycaster.set(cameraPosition, direction.sub(cameraPosition).normalize());

    const intersects = this.raycaster.intersectObjects( this.scene.children );
    console.log('Intersection count:', intersects.length)

Cubes are in scene coordinate system, 300 meters wide, 1000 meters apart.
this.map.getFreeCameraOptions() returns camera position (including elevation) in normalized Web Mercator coordinates.
The matrix l in example converts from Three.js scene coordinate system to map coordinate system.
The example bakes map projection matrix m (projects from map to GL space) with scene to map l to camera.projectionMatrix used to render.
In order to do raycast in Three.js scene coordinate system, ray needs to be defined in scene coordinate system. Ray origins at camera. Camera position in scene coordinate system is calculated by applying map to scene (inverted scene to map l)
transform to camera map coordinates (Web Mercator): cameraPosition.applyMatrix4(l.invert());.

The direction of the ray is defined by camera position as origin and point on far plane, converted vec4(mouse.x, mouse.y, 1, 1) to scene coordinates.
Mouse.x and mouse.y are in GL coordinates. To convert them to scene coordinates, we need to use inverted scene to GL transformation (that is projection matrix): let direction = mouse.clone().applyMatrix4(this.camera.projectionMatrix.clone().invert());