Should lightMap/aoMap occlude lights?

[mrdoob]

I was messing with this example the other day, and I noticed that the shadow gets affected by the specular.

At the moment, the example is using this texture as color map. I tried creating a plane in editor and applying the texture both as lightMap and aoMap and the specular goes through it.

This other example has the same issue.

I'm confused about the difference between lightMap and aoMap, but I bet one of them should be able to occlude specular lights 😇

Related: #6573

/cc @WestLangley

[Nefsen402]

Specular reflection is a method in most shaders that employ the Phong shader model to make it seem like the light is bouncing off the material directly into the camera to make certain parts brighter instead of just diffuse shading. Shadows is created by something blocking the light from ever reaching anything behind the object in question. If the shadow is supposed to block light, then why is it not blocking specular lighting? This defies the very reason to implement shadow maps in the first place. In my opinion this possible bug looks terrible.

I looked at the shader model in Three.js and shaders are created specifically for each material and their sub-properties. I always thought that this specuar/shadow ordeal is just a bug in how these shader blocks are concatenated like they are put in the wrong order.

Change it? Yes. This would apply to the Ambient Occlusion Map and the regular texture of the object. Blocked light should simply be treated like no light. Just my two cents.

[WestLangley]

I'm confused about the difference between lightMap and aoMap, but I bet one of them should be able to occlude specular lights

A lightMap does not occlude anything. In three.js, a lightMap is a source of ambient light that reflects diffusely. Ambient light is also known as indirect light. It is additive to total incoming light, just as AmbientLight is.

Currently, there are three sources of ambient (or indirect) diffuse light in three.js: AmbientLight, HemiSphereLight, and LightMap. Someday, we will add image-based lights and/or light probes to that list; they both have a diffuse lighting component.

An aoMap is an ambient occlusion map, and like its name says, it occludes ambient light -- that is, indirect light. That is all it occludes.

So the aoMap occludes the three sources mentioned above. It does not occlude direct light sources at all. Direct light sources include light from DirectionalLight, SpotLight, and PointLight.

---

Aside: Indirect light can reflect diffusely or specularly. I have only mentioned indirect diffuse light. The other is indirect specular light. three.js does not have any models currently for indirect specular light. envMap, which is a form of an image-based light that reflects specularly, should be modeled as indirect specular. We will get there eventually. When we do, we will decide if specular light reflected from an indirect light source should be occluded by the aoMap, too -- or by how much.

[WestLangley]

If the shadow is supposed to block light, then why is it not blocking specular lighting?

The shadow mapping code in three.js is a hack. It only darkens the reflected light. The hotspot is so bright, that even when darkened, it still has a value greater than 1, so it appears white.

The shadow mapping methodology in three.js will hopefully be changed to properly block incoming light sources. There have been multiple discussions on this board about that topic.

This would apply to the Ambient Occlusion Map

No. See my post above.

[Nefsen402]

@WestLangley I misunderstood how the ambient occlusion map. I know what ambient occlusion is as I read countless articles and implemented such a thing when playing around in OpenGL in c++ using ray marching (kind of). I just have never herd of ambient maps but I guess such a technique is needed because WebGL is so limiting.

If the shadow mapping is a 'hack' that simply darkens the region in shadow why not go the whole nine yards and darken it to 0 and use a regular ambient light or the lightMap to lighten it back up a little bit? I did this when creating shaders that process shadows and it looks fine.

[WestLangley]

If the shadow mapping is a 'hack' that simply darkens the region in shadow why not go the whole nine yards and darken it to 0 and use a regular ambient light or the lightMap to lighten it back up a little bit?

Is that not what the library does now if shadowDarkess = 0?

[Nefsen402]

Is that not what the library does now if shadowDarkess = 0?

I'm just speaking out of ignorance of the library now: why would such a feature to control shadow darkness be implemented then if it causes the issues that @mrdoob proposed but can be done differently while not having such issues? This would just bloat the shader and javascript needing to manage an extra uniform or however you send lighting data to the shader.

[mrdoob]

I think talking about shadowmaps is not relevant here.

I'm arguing that, either lightMap or aoMap should be able to darken specular too. At the moment neither affect specular and the result is unrealistic.

Technically this is simple, just multiply the specular with the pixel value of the greyscale map.

So... seems like, either we are mistaken on how lightMap and aoMap are supposed to behave or we're missing a type of map that occludes both diffuse and specular.

[mrdoob]

I mean, is there a way we can produce the desired effect right now?

[mrdoob]

Hmm... Wait I guess the solution is to use the same texture as map and specularMap?

D'oh...

[WestLangley]

I'm arguing that, either lightMap or aoMap should be able to darken specular too.

lightMap used to be able to darken reflected light in earlier revisions of three.js. The behavior of lightMap was changed to be only additive. Consequently, it does not darken anything.

aoMap only darkens. And it only darkens light from ambient (indirect) light sources -- as it should.

In particular, aoMap currently only darkens diffusely-reflected light from ambient sources. It remains to determine if it will darken specularly-reflected light from ambient sources, too. It will never darken specular reflections from direct sources, such as lights.

[mrdoob]

Having said that... I think something is still wrong...

aoMap:

totalAmbientLight *= ( texture2D( aoMap, vUv2 ).r - 1.0 ) * aoMapIntensity + 1.0;

I think this one is fine. However...

lightMap:

totalAmbientLight += texture2D( lightMap, vUv2 ).xyz * lightMapIntensity;

As far as I understand (http://wiki.polycount.com/wiki/Light_map), lightMap is supposed to multiply diffuse instead of add to ambient?

I think this would be more useful?

diffuseColor.rgb *= texture2D( lightMap, vUv2 ).xyz * lightMapIntensity;

That way, aoMap multiplies Ambient, and lightMap multiplies Diffuse (and probably Specular).

[mrdoob]

Hmm... I'm experimenting and it's difficult.

If anything, if lightMap is going ton contribute to light, we may want to contribute to totalDiffuseLight instead, so shadows affect it?

totalDiffuseLight += texture2D( lightMap, vUv2 ).xyz * lightMapIntensity;

[WestLangley]

As far as I understand (http://wiki.polycount.com/wiki/Light_map), lightMap is supposed to multiply diffuse instead of add to ambient?

See the detailed discussion and history in #6263

I happen to agree with @bhouston on this one as far as the decision to treat light maps as a source of light. To treat them as multiplicative, as we currently do shadows, is an option, but it will not be consistent with our decision to move to more PBR-based approaches.

[WestLangley]

If anything, if lightMap is going to contribute to light, we may want to contribute to totalDiffuseLight instead, so shadows affect it?

This is a reasonable debate to have. Some uses may use light maps to bake direct light-sources, so in that sense they can represent both indirect and direct light.

Whatever we do, users who think of light maps as multiplicative will have a problem with our approach.

/ping @bhouston

[WestLangley]

@bhouston explains our current implementation of light maps well:

Light maps are not baked shadow maps, rather they represent by definition incoming light to a surface...
Lighting for the most part has an "associative property", where each light is independent of the others. Light maps are just precomputed lights and they should exhibit the same "associative property." And you apply them such that you can have a real-time light or a baked light and they generally are the same thing (you only loose directionality effects with baked lights.)

[bhouston]

Hi guys!

Lightmaps represent the full additive contribution of a baked light. If you have other lights in the scene, they may be able to light an area that is in shadow of a baked light -- that is okay, because it is a different light, only the baked light is for sure occluded in that spot, not the dynamic light (it may or may not be occluded at that spot, it is based on the dynamics light position.) Dynamic lights need to use shadow maps in order to case shadows. The main thing in the first example is that the dynamic light should be using its own shadow map to have it occluded by the object.

I think one of the confusing issues is that our shadow maps often are not actually occluding lights completely, they just slightly darken them (or at least they did last time I checked), which is really not that affective at all on a specular highlight.

[mrdoob]

That makes sense. So, then it would be logical that lightMap would be additive to totalDiffuseLight instead of totalAmbientLight. No?

[bhouston]

You are correct, our lightMap only contains the diffuseLight component, not the specularLight (which is directional) or ambientLight. There are more advanced lightMap implementations that contain directional components, but they are not well supported by baking tools (I do not think any generic baking tools support them.)

[WestLangley]

No. lighMap is additive to total ambient diffuse light (a.k.a indirect diffuse light). For us, that would be to totalAmbientLight.

[WestLangley]

The components of light we model are indirect diffuse, indirect specular, direct diffuse, and direct specular.

I will modify the code to use those terms so we do not have this confusion.

[bhouston]

Can we use the term direct and indirect rather than direct and ambient? I got confused because I thought that ambient has multiple meanings.

[WestLangley]

Agreed.

The components of light we model are indirect diffuse, indirect specular, direct diffuse, and direct specular.

I know this is confusing. I will fix the code to use consistent terms.

[bhouston]

@WestLangley BTW if you wanted to contribute to this PR it basically works and uses this terminology: #7324 I was thinking that maybe we should add these structs anyhow, even if they are incompatible with @unconed's shader graph stuff.

[mrdoob]

Say that we want bake the illumination in a scene (so we have nice shadows) and then we put dynamic lights in the same position so we can produce specular lights.

Would one need to do a custom shader for this?

[bhouston]

I've seen an option on lights to "affect diffuse" or "affect specular" -- thus it is a light option.

[bhouston]

This light option can be put on all lights that offer specular.

[mrdoob]

I was thinking that maybe we should add these structs anyhow

Yeah, I was studying structs now. Was aiming to clean/implement this week.

[bhouston]

I can spend a bit of time tomorrow to clean it up a bit. It does work in its current state, just some of the examples are broken.

[mrdoob]

I've seen an option on lights to "affect diffuse" or "affect specular" -- thus it is a light option.
This light option can be put on all lights that offer specular.

But that's the thing, neither lightMap nor aoMap occlude specular.

Using specularMap solved the first example. But the second example has a specularMap for the bricks already...

[mrdoob]

I can spend a bit of time tomorrow to clean it up a bit. It does work in its current state, just some of the examples are broken.

That'd be great! 😊

[bhouston]

But that's the thing, neither lightMap nor aoMap occlude specular.

They do not by design though. I'd argue that the use of the specularMap in the first example to occlude the specular only appears to work but it really blocks too much -- it will screw up specular from lights that are not behind the car, which shouldn't happen. The specularMap basically says that the surface is very rough and doesn't not reflect specular light, no matter where it comes from.

My understanding is that dynamics lights really should use their shadow maps to reduce their contributions. And you could have them be only specular if you need them to be so that you can combine then with baked shadows (but that will only really work if the lights still have shadow maps to avoid specular shining through objects.)

What we should do is have all shadow maps calculated first and their indices stored with the light structs that are passed to the renderer. Then as one is calculating each light contribution you check its shadow map to see if it is occluded, and that determines if you accumulate the lighting result or not. This is significantly more accurate and realistic than the darkening post-accumulation effect we currently do.

[mrdoob]

What we should do is have all shadow maps calculated first and their indices stored with the light structs that are passed to the renderer. Then as one is calculating each light contribution you check its shadow map to see if it is occluded, and that determines if you accumulate the lighting result or not. This is significantly more accurate and realistic than the darkening post-accumulation effect we currently do.

Sounds good! I'll experiment a bit.