How to assign custom shader to Gsplat Entity

[KEMSHlM]

How to assign custom shader to Gsplat Entity using @playcanvas/react

Issue Description

Vertex shader compilation fails when implementing Gaussian Splatting with PlayCanvas custom shaders using raw-loader for GLSL files.

Note: This issue spans both @playcanvas/react and @playcanvas/engine. As I'm primarily working with @playcanvas/react and have limited experience with @playcanvas/engine internals, the shader implementation details may need review from those familiar with the engine's shader system.

Steps to Reproduce

1. Configure raw-loader for GLSL files
2. Implement GsplatComponent with provided code
3. Load vertex and fragment shaders using raw-loader
4. Render the component

Error Details

Main vertex shader errors:

• Undefined functions: calcSplatUV, getCenter, getCovariance, calcV1V2
• Undefined variables: matrix_view, matrix_model, matrix_projection
• Undefined variables: vertex_position, viewport
• Type mismatch in vector/matrix operations

Failed to compile vertex shader:

ERROR: 0:80: 'calcSplatUV' : no matching overloaded function found
ERROR: 0:80: '!' : wrong operand type - no operation '!' exists that takes an operand of type const mediump float (or there is no acceptable conversion)
ERROR: 0:80: '' : boolean expression expected
ERROR: 0:85: 'getCenter' : no matching overloaded function found
ERROR: 0:85: '=' : dimension mismatch
ERROR: 0:85: '=' : cannot convert from 'const mediump float' to 'highp 3-component vector of float'
ERROR: 0:87: 'getCovariance' : no matching overloaded function found
ERROR: 0:91: 'matrix_view' : undeclared identifier
ERROR: 0:91: 'matrix_model' : undeclared identifier
ERROR: 0:91: '=' : dimension mismatch
ERROR: 0:91: '=' : cannot convert from 'const highp float' to 'highp 4X4 matrix of float'
ERROR: 0:93: 'matrix_projection' : undeclared identifier
ERROR: 0:100: 'calcV1V2' : no matching overloaded function found
ERROR: 0:100: '=' : dimension mismatch
ERROR: 0:100: '=' : cannot convert from 'const mediump float' to 'highp 4-component vector of float'
ERROR: 0:101: 'splatUV' : undeclared identifier
ERROR: 0:101: 'texelFetch' : no matching overloaded function found
ERROR: 0:101: '=' : dimension mismatch
ERROR: 0:101: 'assign' : cannot convert from 'const mediump float' to 'out mediump 4-component vector of float'
ERROR: 0:110: 'vertex_position' : undeclared identifier
ERROR: 0:110: 'x' :  field selection requires structure, vector, or interface block on left hand side
ERROR: 0:110: 'vertex_position' : undeclared identifier
ERROR: 0:110: 'y' :  field selection requires structure, vector, or interface block on left hand side
ERROR: 0:110: 'viewport' : undeclared identifier
ERROR: 0:111: 'vertex_position' : undeclared identifier
ERROR: 0:111: 'xy' :  field selection requires structure, vector, or interface block on left hand side
ERROR: 0:111: '=' : dimension mismatch
ERROR: 0:111: 'assign' : cannot convert from 'highp float' to 'out mediump 2-component vector of float'
�
75:	varying mediump vec4 color;
76:	
77:	mediump vec4 discardVec = vec4(0.0, 0.0, 2.0, 1.0);
78:	
79:	void main(void) {
80:	    if (!calcSplatUV()) {
81:	        gl_Position = discardVec;
82:	        return;
83:	    }
84:	
85:	    vec3 center = getCenter();
 while rendering undefined

Expected Behavior

1. GLSL files should be properly loaded using raw-loader
2. Vertex shader should compile successfully and render Gaussian Splatting properly

Current Behavior

Shader compilation fails with multiple undefined reference errors and type mismatches.

Additional Context

1. Error suggests missing shader uniform definitions and helper functions required for GSplat implementation
2. Issue may require guidance from @playcanvas/engine team regarding proper shader implementation
3. Implementation is based on React wrapper (@playcanvas/react) without deep understanding of engine internals
4. Using raw-loader to import GLSL files as strings

Environment

• @playcanvas/react: 0.2.0
• @playcanvas: 2.3.3

This is code ...

"use client";

import { useLayoutEffect, useRef, type FC } from "react";
import type { Asset, Entity as PcEntity } from "playcanvas";
import { Entity } from "@playcanvas/react";
import { useParent } from "@playcanvas/react/hooks";
import vertexShader from "../../playcanvas/shaders/gsplat.vert";
import fragmentShader from "../../playcanvas/shaders/gsplat.frag";

interface GsplatProps {
  asset: Asset;
}

const GSplat: FC<GsplatProps> = ({ asset }) => {
  const parent: PcEntity = useParent();
  const assetRef = useRef<PcEntity | null>(null);

  useLayoutEffect(() => {
    if (asset) {
      assetRef.current = asset.resource.instantiate({
        vertex: vertexShader,
        fragment: fragmentShader,
      });
      if (assetRef.current) parent.addChild(assetRef.current);
    }

    return () => {
      if (!assetRef.current) return;
      parent.removeChild(assetRef.current);
    };
  }, [asset, parent]);

  return null;
};

[marklundin]

What does your vertex shader look like? You may want to check the latest shader in the engine and base yours off this.

[KEMSHlM]

@marklundin Thank you for the assistance.

I simplified the problem by adapting and applying the Shader script from this sample:

Same error..

Failed to compile vertex shader:

ERROR: 0:318: 'SplatState' : undeclared identifier
ERROR: 0:318: 'state' : syntax error
�
313:	    return clr;
314:	}
315:	
316:	void main(void) {
317:	
318:	    SplatState state;
319:	    if (!readCenter(state)) {
320:	        gl_Position = discardVec;
321:	        return;
322:	    }
323:	
 while rendering undefined

I also modified the Shader application method as follows:

import { Mat4, Script as PcScript, Entity as PcEntity } from "playcanvas";
import type { SplatMaterialOptions } from "playcanvas/build/playcanvas/src/scene/gsplat/gsplat-material.js";
import { Script } from "@playcanvas/react/components";
import vertexShader from "../../playcanvas/shaders/gsplat.vert";

class GsplatShaderScript extends PcScript {
  timer = 0;
  materialOptions: SplatMaterialOptions = {
    vertex: vertexShader,
  };

  private getGsplatComponents() {
    return (this.entity.parent as PcEntity)
      .findComponents("gsplat")
      .map((component) => component.entity.gsplat)
      .filter((gsplat): gsplat is NonNullable<typeof gsplat> => gsplat !== null);
  }

  initialize() {
    for (const gsplat of this.getGsplatComponents()) {
      gsplat.materialOptions = this.materialOptions;
    }
  }

  update(dt: number) {
    for (const gsplat of this.getGsplatComponents()) {
      const material = gsplat.instance?.material;
      if (material) {
        material.setParameter("uTime", this.timer);
      }
    }
    this.timer += dt;
  }
}

export const GsplatShader = () => <Script script={GsplatShaderScript as PcScript} />;

Are there any specific configuration requirements needed to apply this Shader?

[marklundin]

Hey @KEMSHlM, I've created a simple transition example with a custom shader that you can see here. I'll probably update the splat component directly to accept custom vertex shaders etc too, so this should simplify things.

[KEMSHlM]

@marklundin It works perfectly!! Thank you for your thorough support. Also, I've been receiving so many benefits thanks to this repository (playcanvas/react). Thank you very much!

[marklundin]

Brilliant, glad it's helpful! Pls hit the repo star button @KEMSHlM

Closing this

[marklundin]

FYI @KEMSHlM, I've added the ability to set shaders on the Splat component in this PR #33. It will be available in the next release

[KEMSHlM]

@marklundin Nice! I'm impressed by how quickly the updates are being made.
By the way, when referencing variables outside of a Shader, do we still need to create them individually as shown in the code you provided last time? For example, when I want to reference something like time.

#33

[marklundin]

Yep you'll need to expose uniforms that you want to control via JS, and set them via the material as per the demo. Let me know if that works for you