Need to display Gaussian Splats without viewer.start() while using custom renderer and OrbitControls

[ryouhei0622]

Thanks for your contribution!

I want to add Three.js objects to a GS viewer scene. I'm using Viewer instead of DropInViewer because I need to handle onClick events on the scene. I also want to control the camera angle, so I created a separate renderer to modify OrbitControls settings.
In this case, I cannot use viewer.start(), so I tried adding viewer.splatMesh to the Three.js scene to display the GS file. However, while Three.js objects are displayed correctly, the Gaussian Splats are not visible.
Question: Is it possible to display both Three.js objects and Gaussian Splats simultaneously when using custom Viewer, renderer, and OrbitControls without calling viewer.start()?

Expected Behavior: Both Three.js objects and Gaussian Splats should be visible in the scene.
Actual Behavior: Only Three.js objects are visible; Gaussian Splats are not rendering.

[ryouhei0622]

Especially I want to modify the OrbitControls' rotation behavior by setting its target property to specific coordinate points within the Gaussian Splats.

function setupRenderer() {
  const renderWidth = window.innerWidth;
  const renderHeight = window.innerHeight;

  const rootElement = document.getElementById('renderer');
  if (!rootElement) {
    throw new Error('Root element not found');
  }
  rootElement.style.width = renderWidth + 'px';
  rootElement.style.height = renderHeight + 'px';
  rootElement.style.position = 'relative';

  const renderer = new THREE.WebGLRenderer({
    antialias: false
  });
  renderer.setSize(renderWidth, renderHeight);
  rootElement.appendChild(renderer.domElement);

  return renderer
}

function setupCamera(renderWidth: number, renderHeight: number) {
  const camera = new THREE.PerspectiveCamera(65, renderWidth / renderHeight, 0.1, 500);
  camera.position.copy(new THREE.Vector3().fromArray([-1, -4, 6]));
  camera.lookAt(new THREE.Vector3().fromArray([0, 4, -0]));
  camera.up = new THREE.Vector3().fromArray([0, -1, -0.6]).normalize();
  return camera;
}

 renderer.current = setupRenderer();
controlRef.current = setupControls(cameraRef.current, renderer.current);
controlRef.current.target.x = foo
controlRef.current.target.y = foo
controlRef.current.target.z = foo

[seppestaes]

"modify the OrbitControls' rotation behavior by setting its target property to specific coordinate points within the Gaussian Splats."

Can you give more details?

I'd

• use the DropInViewer (pass scene, rootElement, renderer e.a. as arguments)
• implement own mouse/touch listeners (cf setupEventHandlers in Viewer)
• use the Raycaster (as provided)
• (global coordinates) of raycast hits
• a lib like camera-controls (yomotsu) to animate the camera
• manage the update

[ryouhei0622]

I apologize for the lack of explanation.
What I really want to do is to create my own rendering process using the GaussianSplats3D.Viewer method. I believe this will allow me to control the camera using OrbitControls and effectively manage the rendering order.
I have seen an example of creating a custom renderer in dropinviewer.html, but I could not find an example specifically using a custom renderer with GaussianSplats3D.Viewer. Could you provide any examples or guidance on this?

[seppestaes]

If I understand your intention, you want to have control over the camera animation based on interactions with the splats.
Splat raytracing is part of the Viewer (useBuiltInControls flag), but disabled in DropInViewer.

You tweak the camera animation of the Viewer (i avoid making changes to this repo), or implement mouse/touch listeners, use the DropInViewer and the RayCaster.js (setFromCameraAndScreenPosition, intersectSplatMesh).

It is possible to pass your renderer in the Viewer.

What do you mean by managing the rendering order?

[seppestaes]

Hi @ryouhei0622 , I'll try to find some time for an example.

[ryouhei0622]

Hi. @seppestaes
Thank you for your response.

Currently, I would like to achieve the following:

1. Use the raycaster functionality of GaussianSplats3D.Viewer to click on Gaussian point clouds and add Three.js objects.
2. Specify the rotation center of the camera (the target of OrbitControls).
3. Render Three.js objects in front of the Gaussian point clouds .

To satisfy points 2 and 3, I believe it is necessary to implement custom controls and a custom renderer.

Currently, I am initializing the viewer as shown below and using viewer.start(). However, with this implementation, I cannot pass custom controls or a custom renderer. I would greatly appreciate it if you could provide an example for this scenario.

const viewer = new GaussianSplats3D.Viewer({
        rootElement: containerRef.current,
        threeScene: threeScene,
        cameraUp,
        initialCameraPosition,
        initialCameraLookAt,
        webWorkerConfig: {
          crossOriginIsolated: self.crossOriginIsolated,
          useSharedMemory: self.crossOriginIsolated,
        },
        sharedMemoryForWorkers: self.crossOriginIsolated,
        gpuAcceleratedSort: true,
      });

[ryouhei0622]

What do you mean by managing the rendering order?
This is related to point 3, where I want to render Three.js objects in front of the Gaussian point clouds.

In the following definition:
threeScene → splatMesh,
it seems that Three.js objects are not rendered in front of the Gaussian point clouds. Therefore, I would like to change the rendering order to splatMesh → threeMesh so that Three.js objects are rendered in front.

 render = function() {

        return function() {
            if (!this.initialized || !this.splatRenderReady || this.isDisposingOrDisposed()) return;

            const hasRenderables = (threeScene) => {
                for (let child of threeScene.children) {
                    if (child.visible) return true;
                }
                return false;
            };

            const savedAuoClear = this.renderer.autoClear;
            if (hasRenderables(this.threeScene)) {
                this.renderer.render(this.threeScene, this.camera);
                this.renderer.autoClear = false;
            }
            this.renderer.render(this.splatMesh, this.camera);
            this.renderer.autoClear = false;
            if (this.sceneHelper.getFocusMarkerOpacity() > 0.0) this.renderer.render(this.sceneHelper.focusMarker, this.camera);
            if (this.showControlPlane) this.renderer.render(this.sceneHelper.controlPlane, this.camera);
            this.renderer.autoClear = savedAuoClear;
        };

    }();