Implement Projectors to support textured lighting

[usefulthink]

Adds a new light-type ProjectorLight, a light-source with a textured rectangular beam with configurable aspect and rotation.

Example: https://cdn.rawgit.com/usefulthink/three.js/projector-lights-preview/examples/webgl_lights_projectorlight.html

Usage:

var projector = new THREE.ProjectorLight(0xffffff, 1);
projector.fov = 20; // vertical FOV in degrees
projector.aspect = 16 / 9; // texture aspect-ratio
projector.map = new THREE.TextureLoader().load(textureUrl);

scene.add(projector);

Note: this PR is not yet fully completed and there are some minor parts I am not really happy with. I am posting it mostly to get some early feedback on general interest from other developers and feedback on implementation details.

[takahirox]

Looks interesting...

[RemusMar]

I am posting it mostly to get some early feedback on general interest from other developers and feedback on implementation details.

An altered SpotLight is always useful.
👍

[WestLangley]

This is great!

Bat-signal implemented with a grayscale RGB texture.

One suggestion would be to attenuate the intensity with the alpha channel as well.

vec4 texture = texture2D( projectorTexture, projectorUv );

directLight.color = projectorLight.color;
directLight.color *= texture.rgb;
directLight.color *= texture.a; // attenuate

I also think support for texture offset/repeat/center/rotation would be a worthwhile addition.

[tentone]

@usefulthink Thanks a lot! Nice!

I would aim into merging it as it is, and improve it on new pull requests!

[usefulthink]

@WestLangley thinking out loud here, but shouldn't it be possible with relatively few changes to support both perspective and orthographic modes (i.e. ProjectiveSpotLight and ProjectiveDirectionalLight) with the same basic light-type or am I missing something here?

Good point with the attenuation from alpha-channel, added that.
I'm not yet familiar with how the texture-parameters (offset/repeat/center/rotation) are handled, will dig into that and see if it can be added.

[looeee]

This is a great addition.

Would it be better to add a .projectiveMap parameter to existing SpotLight and DirectionalLight, rather than creating a new light type though?

We could add a coneShape (and aspect ) settings to SpotLight too, to switch between square or circular cones.

[takahirox]

I prefer adding projective map property rather than adding a new light type if it works fine.

[WestLangley]

I would keep these new lights as separate classes and separate from each other at first, and merge the feature set into existing lights later only if it makes sense.

[looeee]

@WestLangley that sounds like a way to grow THREE.Legacy.js even faster. Is there a reason why it doesn't make sense now?

We should agree on a final API before merging new features, rather than adding something that we intend to change later.

[Mugen87]

I tend to vote for @WestLangley's suggestion.

[usefulthink]

also added support for uvTransform now so all common texture-features should now be supported.

@looeee @takahirox I first tried implementing it as a parameter for the spotlight, but it turns out that it effectively results in mostly seperate code-paths and lots of complications within the shaders and in WebGLLights.js. Those could be combined to some degree (if we see a spotlight as a special case of a projective light), but that would slightly slow down the codepath for regular spotlights.

[takahirox]

"if it works fine" I mentioned meant "I prefer map property approach if we can easily do", so that's ok if it's difficult. (I haven't read through the implementation yet).

BTW, curious to know if similar project-light type exists in other engines, for example unity.

[Mugen87]

BTW, curious to know if similar project-light type exists in other engines, for example unity.

As far as i know, Unity has no separate light types for this. One concept they use is Cookies.

See https://www.youtube.com/watch?v=WERan6gjEn4

It's also possible to create similar effects with Projector, although this entity is focused on material/shadow projection.

[usefulthink]

I searched through some of the docs:

• unreal does this using "light-functions"
• in cryengine it's a property of the light-entities (although limited to 512x512 quadratic textures)
• unity supports "cookies" as textures on all types of lights.

So yes, all of them seem to implement it as part of the regular lights, and I think we could probably get there at some point.

[takahirox]

Thanks

I think we could probably get there at some point.

👍

[Mugen87]

If we would enhance the basic light types, i would vote for the Cookies approach of Unity. This would only work grayscale textures but you could achieve the same effect like in @WestLangley's demo (with a SpotLight and a respective map).

[looeee]

@Mugen87 is that for performance reasons? Seems a shame to limit this to B&W if we can do full color and videos.

[Mugen87]

From a conceptual point of view, it might be a good idea to treat such maps as pure shaped masks, like mentioned in the Unity docs.

For shadow and material/texture projections (which can also be used to fake light effects) Unity uses a more general Projector class. I've worked a little bit with Unity in the past and i liked this separation. So it's just a personal preference 😊 .

More information about Projectors: https://www.youtube.com/watch?v=44Nad3QwuoA&

[looeee]

I was thinking of it more in terms of modelling of a physical process - that is, a light being partially or fully occluded while shining past / through objects in front of it. In the case of a real world projector, the occluding object is a semi-transparent LCD screen. Other examples are stained glass or a narrow window.

Since unity cookies allow grayscale, they are not actually just cutouts and are modelling a limited version of this physical process.

On the other hand, I presume that the reason cookies and projectors are separate in Unity is that cookies just multiply the light's intensity by the gray level of the cookie image, which is probably a cheap operation.
Projectors have to adjust the color and the intensity which may be more costly, so it would make more sense to keep them separate if this is the case.

[RemusMar]

Would it be better to add a .projectiveMap parameter to existing SpotLight

This would be the best approach in my opinion.

[WestLangley]

I like the Projector nomenclature as a class name.

It could support both perspective an orthographic frustums, and a color-map, cookie, or video.

It would serve as an additional light source and cast shadows.

A shortcoming may be the specular highlights generated by the projector. They may not be correct. RectAreaLight has proper specular highlights.

As I said, I would keep it as a separate class and leave existing lights alone until this is sorted out.

[Mugen87]

Just a little addition on Cookies. They are actually alpha maps but Unity offers an option to convert the luminosity expressed in a grayscale texture to alpha. If we ever add Cookies to three.js we should do the same 😁 .

[WestLangley]

Unity offers an option to convert the luminosity expressed in a grayscale texture to alpha.

@Mugen87 That is not required. The code I posted above is sufficient to handle all cases.

All we need is a Projector.map property which can be an RGB or RBGA texture, or a video texture.

[usefulthink]

Good points here, I did go with @WestLangley and removed the Light suffix from the Projector, I like that a lot more.

Regarding the specular highlights, I don't think I can solve that with my current knowledge of how the BSDFs work. I'm updating the IncidentLight with the color seen at the fragment, and to me the highlights look quite plausible, what am I missing?

[WestLangley]

We could also add add a RectLight?

I'm not exactly sure what you are asking, but if it pertains to adding a map property to RectAreaLight, the answer is basically no: the light would not 'project' the map.

Real projectors use lenses to emit nearly-parallel beams of light. Our spotlights and directional lights are modeled as such emitters.

RectAreaLight models something akin to a computer monitor. Light is emitted in every hemispherical direction, and consequently, it does not project images well.

[looeee]

What about adding SpotRectLight?

Or adding a parameter to SpotLight to control the frustum shape?

[RemusMar]

SpotLight has a conical frustum which will expand up to 180 degrees, but a rectangular shadow frustum.

As I said before,this "projector" is an altered SpotLight and in my opinion the best approach would be to add a few more parameters for this light.

• one for frustum (conical or rectangular)
• one for projected map (none or texture)
  etc

[haroldiedema]

RectAreaLight models something akin to a computer monitor. Light is emitted in every hemispherical direction, and consequently, it does not project images well.

I have to disagree here. It completely depends on what kind of effect you're trying to achieve of course, but imagine having a "roster" in front of your lights, it would project the texture precisely as you'd expect without the need of using "real" shadows.

I vote for adding the extra properties to the existing lights, as @RemusMar said.

[WestLangley]

@haroldiedema An image added to a RectAreaLight will reflect specularly in other objects, but the light will not project the image any more than a computer monitor projects its image.

Sorry, I do not understand what you are trying to say. What do you mean by "roster", and what does it have to do with shadows?

[haroldiedema]

@haroldiedema An image added to a RectAreaLight will reflect specularly in other objects, but the light will not project the image any more than a computer monitor projects its image.

Oh my apologies, then I misinterpret the implementation of RectAreaLight..

Sorry, I do not understand what you are trying to say. What do you mean by "roster", and what does it have to do with shadows?

I mean that with this, you can effectively fake shadows.
Like this image

[WestLangley]

I mean that with this, you can effectively fake shadows.

Right. And that is a compelling reason for this PR. Projector can do that. RectAreaLight cannot -- even if texture support were added.

[mrdoob]

We could also add add a RectLight?

I'm not exactly sure what you are asking, but if it pertains to adding a map property to RectAreaLight, the answer is basically no: the light would not 'project' the map.

No no, I'm proposing creating a new type of light (RectLight) which would basically be a SpotLight but without the circle code.

I think we're all suggesting the same.

[RemusMar]

I'm proposing creating a new type of light (RectLight) which would basically be a SpotLight but without the circle code.

Why Rect if this is a Spot?
The frustum can be: triangular, rectangular, ..., hexagonal, ..., circular (cone).

[mrdoob]

spot
noun
1. a small round or roundish mark

🤔

[usefulthink]

@mrdoob I would like to focus the name more on the texturing than on the beam-shape, you can achieve any shape you want with it... What do think about ProjectorLight, TexturedLight or something like that?

Also, would you prefer a fully integrated approach (i.e. additional properties on SpotLight) or this "standalone" solution?

[RemusMar]

spot
noun

1. a small round or roundish mark

SpotLight has a conical frustum which will expand up to 180 degrees, but a rectangular shadow frustum.

Or maybe a "roundish" shadow frustum for a better SPOT.
ha ha ha

[mrdoob]

If we follow 3D Studio implementation, it would be a matter of adding shape (Circle, Rectangle) and aspect to SpotLight.

[usefulthink]

So in that case, I would propose implementing it like this (hopefully I can tackle that over the weekend):

• spotLight.beamShape (constants Circle or Rectangle): controls the shape of the beam

• spotLight.aspect (number): width/height ratio of the beam for rectangular beams (spotLight.angle is interpreted as vertical FOV if aspect is specified)

• spotLight.map (Texture): projected texture. Should work with both beam-shapes

  • rectangular: texture completely fills the beam-region
  • round: texture is clipped by the beam

• handle rotation around the look-axis (see Implement Projectors to support textured lighting #13057 (comment)). I'd implement the target/rotation refactoring mentioned in that comment (for all light-types: make target-property optional, if no target is specified, the default light.rotation would be used instead).

I dind't think too much about other light-types yet (and will probably not implement them right away). PointLight should be pretty simple: we would just use the map-property (in this case a CubeTexture), all others don't really make sense. Texture-lookups can probably reuse the logic from shadowmap-lookups.

[WestLangley]

spotLight.beamShape, spotLight.aspect

I would highly recommend a simpler solution if you want to add a map property to Spotlight: set the rectangular frustum to be just large enough to enclose the existing conical frustum. That means it will be square, and defined by Spotlight.angle. That way it will adapt to the angle.

You only need to add one property, map, to SpotLight.

Also, it is very important that the shadow frustum be separate from this frustum so we can still support tight, quality shadows.

for all light-types: make target-property optional,

If you want to propose that, please, please do it in a separate PR.

handle rotation around the look-axis

This seams to be an extra feature you are pushing for. I would omit it. Just add the map. Texture maps can be rotated now, which achieves the same effect.

PointLight should be pretty simple

I'm not sure about PointLight, but support for DirectionalLight.map is definitely a good idea, and should have priority. Which frustum to use in the case of directional light is to-be-determined.

[RemusMar]

set the rectangular frustum to be just large enough to enclose the existing conical frustum.
That means it will be square, and defined by Spotlight.angle

Square is also a better option for textures.
👍

[usefulthink]

@WestLangley I think we are thinking about different usage-scenarios.

What would you think about adding a map-property to the SpotLight (working in a simplified way as you described) and keeping this class (renamed to whatever we decide on) for the full-fledged projector use-case?

That means it will be square, and defined by Spotlight.angle. That way it will adapt to the angle.
You only need to add one property, map, to SpotLight.

A square frustum with the normal round beam would be the default, but I wouldn't want to stop there as it doesn't really cover what I am about to do: I want to simulate actual projectors, so I need for instance a beam with 16:9 aspect-ratio using a texture from a 16:9 video-source. I don't see how that could be achieved with a square-beam without having to resample the video to a size that fits inside the spotlights beam.

Also note that supporting the aspect/shape cases won't necessarily need much extra-work in the shaders and no extra uniforms (well, at least if it's implemented the way I did it, there may be more efficient ways).

Also, it is very important that the shadow frustum be separate from this frustum so we can still support tight, quality shadows.

I will consider that, but I would still try to provide a default that covers the whole light-volume of the light, ideally nothing more.

If you want to propose that, please, please do it in a separate PR.

Will do.

This seams to be an extra feature you are pushing for.
I would omit it. Just add the map. Texture maps can be rotated now, which achieves the same effect.

😄 it is, mostly because I think it wouldn't be complete if we left a useful degree-of-freedom inaccessible to the user. Not sure about that "same effect" – especially the shadow-camera will not know that the texture was rotated. What I don't like about the texture-rotation is that it doesn't work well with a shadow-camera that is synchronized with the light-volume (which should be the default behaviour).

[WestLangley]

I want to simulate actual projectors, so I need for instance a beam with 16:9 aspect-ratio

A new light projector class is fine with me (leaving the existing lights as-is), but the others in this thread prefer you just add a map property to Spotlight and DirectionalLight. That means keeping it simple. It is also the best way to get your PR merged.

In real life, studio lights have "barn doors", as I mentioned. You can model those with "cookies", so the conical frustum is masked, and renders as a rectangular one.

Your other option is to continue to argue for the separate ProjectorLight.

[usefulthink]

That means keeping it simple. It is also the best way to get your PR merged.

Jip. Will give that a try then. I'd still hope to keep the full projector-implementation around :)

The only question I have regarding the spotlight.map-version is how to handle the texture-uniforms properly. If I declare them as

uniform sampler2D spotLightTextures[NUM_SPOT_LIGHTS];

but only assign a texture to one of the indices, would that waste any gpu-resources for the lights without textures?

[mbredif]

#14621 has a proposal to add the map property to spotlights.
It also refrains from using a texture slot if the map property is undefined. (addressing the concern #13057 (comment)).

@mrdoob , @WestLangley, which alternative would you prefer getting merged ? as a new Light type (this PR) or as a SpotLight with a map property (#14621, to be reviewed) ?

[WestLangley]

@mbredif Based on the discussion in this thread, adding .map to SpotLight appears to be the preference.

[sndrgb]

Any news about this?

[Mugen87]

@usefulthink Thanks for your great work here! But according to #13057 (comment), closing this in favor of #14621.

[usefulthink]

@Mugen87 Sure thing! Apologies for not finding the time and energy to move this ahead. Awesome that someone else did push it through!