Sprite Material support

[gigablox]

Imagine a RPG style game where each player has a nameplate. For example ( https://i.imgur.com/KxgTpuu.jpg ).

Working with MeshBasicMaterial or MeshPhongMaterial we would need to use an onBeforeRender handler that adjusts the whole group's world matrix based on camera frustum on each frame as you've suggested here.

A Sprite Material has these optimizations baked in to handle this exact scenario. It feels like following your suggestion will lead to rolling our own Sprite Material, which is likely going to have much worse performance and usability than Three's.

It will also really complicate solutions. Imagine all the other use cases in the scene that need a Sprite's behavior but belong to different object groups with different object hierarchy's. There would never be a one size fits all in the same way a Three's Sprite behaves.

Would really appreciate support for Sprite Material as it would both improve performance of our implementation along with developer usability of your library.

[lojjic]

I'm not deeply familiar with Three's SpriteMaterial. Can that be applied to any Mesh? If so then I don't see why it wouldn't also work for Text.

At first glance, though, it appears pretty specific to Sprite objects, making assumptions based on that, so I doubt it's useful for this scenario. It sounds like what you really want is a material that performs the same "orient toward camera" transform in its shader. Such a material could work on any mesh, including Text.

That's something that could be generically useful. I may take a swing at it at some point but no guarantees. There may be something already existing out there though.

[gigablox]

Hey there thanks for your reply - actually I think SpriteMaterial can only be applied to Sprite. If we could construct Three's Sprite using your Text material that would be the end goal here. It sounds like what you really want is a material that performs the same "orient toward camera" transform in its shader. - I believe this is exactly what Sprite + SpriteMaterial achieve.

https://threejs.org/docs/#api/en/objects/Sprite
https://threejs.org/docs/#api/en/materials/SpriteMaterial
https://threejs.org/examples/?q=sprite#webgl_sprites

[lojjic]

Right, Sprite/SpriteMaterial is one specific way of getting that facing-camera behavior. (You'll often see this referred to as "billboard".) But it also comes with other assumptions which don't really apply for Text or other Meshes. I still think what you really want is a vertex shader that mimics that same behavior but is done in a Mesh-oriented material.

I was able to make a quick proof-of-concept shader that does that. Here's the relevant code from the vertex shader:

  vec4 mvPosition = modelViewMatrix * vec4( 0.0, 0.0, 0.0, 1.0 );
  mvPosition.xyz += position;
  gl_Position = projectionMatrix * mvPosition;

Basically it uses the modelViewMatrix to find the object's origin, and then does a direct addition of the vertex position ignoring the matrix's rotation component. It should probably also maintain the matrix's scale components like SpriteMaterial does.

With that code in a ShaderMaterial, I was able to assign it to Text instances and it worked as expected. I may try to bundle that up as a reusable material for any mesh (nothing Text-specific here), but like I said before no guarantees. ;)

[lojjic]

Added maintaining the scale:

vec4 mvPosition = modelViewMatrix * vec4( 0.0, 0.0, 0.0, 1.0 );
vec3 scale = vec3( length(modelViewMatrix[0].xyz), length(modelViewMatrix[1].xyz), length(modelViewMatrix[2].xyz) );
mvPosition.xyz += position * scale;
gl_Position = projectionMatrix * mvPosition;

[lojjic]

Here's an example in action, using createDerivedMaterial from troika-three-utils. You should be able to use that directly.
https://codesandbox.io/s/createbillboardmaterial-xl6mt?file=/src/createBillboardMaterial.js

I do think I'll formalize this, with some fixes to normals so lighting works as expected. It's a very flexible approach -- any material can be used as the base, and it can be applied to any mesh.

Keep in mind though that it does have drawbacks, like not being raycastable.

[gigablox]

Ah hah! Yes this is excellent!

I really enjoy using your implementation, there are not many text libraries with SDF that work well or are as flexible. Adding support for this use case is really helpful. Thanks so much for your time in putting together an example and look forward to seeing this in your library in the future.

[KamyarTaher]

Hello, this is really great.
Was there a direct implementation of this in the library?
If not, is the codeSandBox example still up to date? I tried to update the packages but it seems to break the code.

[lojjic]

I've updated the CodeSandbox example to the latest libraries.