Restricting ambient occlusion to modulate only indirect lighting

[bhouston]

In the FrostBite 4 engine, they make a distinction between different types of lighting and they only modulate what they call "indirect diffuse" lighting by their ambient occlusion maps. See pages 74/75 of this PDF:

http://www.frostbite.com/wp-content/uploads/2014/11/course_notes_moving_frostbite_to_pbr.pdf

I am a little fuzzy as to what exactly is indirect diffuse lighting as per the FB4 definition. Probably the following is considered to be indirect diffuse:

• lightMap
• ambientLight
• hemisphereLight (should be diffuse only, no specular)

But are the following considered to be "indirect diffuse" for the purposes of ambient occlusion modulation?

• diffuse from spotLight
• diffuse from directionalLight
• diffuse from pointLight
• diffuse from areaLight
• diffuse from lightProbe (diffuse as in the blurred component?) I do not know even how to separate that from the specular component.

[WestLangley]

This is my understanding. It is not based on the particular reference you sited:

Light types

• direct light is light from direct sources -- spotLight, directionalLight, pointLight, areaLight
• indirect light is light from indirect sources -- ambientLight, hemiLight (maybe... if not, it is direct light)

Terminology

• directDiffuse refers to light from direct sources that is reflected diffusely (modulated by the diffuseMap).
• indirectDiffuse refers to light from indirect sources that is reflected diffusely.
• directSpecular refers to light from direct sources that is reflected specularly (modulated by the specularMap).
• indirectSpecular refers to light from indirect sources that is reflected specularly.

Our direct light sources contribute to directDiffuse and directSpecular.

Our indirect light sources contribute to indirectDiffuse and (in the future) indirectSpecular. We need indirectSpecular to make metals look correct, IMO. (How that is done is a separate issue.)

Ambient occlusion modulates only ambient diffuse light -- what is referred to above as indirectDiffuse.

Light probes -- not sure... we can defer that to much later, IMO.

It remains to discuss map-based light sources:

• lightmaps
• environment maps
• image-based-lights
• matCaps (a.k.a. lit-spheres)

[bhouston]

I think these are easy:

It remains to discuss map-based light sources:

• lightmaps
• environment maps
• image-based-lights
• matCaps (a.k.a. lit-spheres)

• Non-directional light maps -- yes.
• Directional light maps (which we do not support yet) - no.
• Environment maps - no, because they are directional and specular.
• Image-based lights - generally they are directional and specular. (BTW IBL are generally just the evolution of environment maps and usually in HDR rather than LDR. Also light probes are the generalization of IBL to have multiple IBL sources.)

Except for:

• matCaps - very strange things, but luckily we do not support them yet so I do not have to figure out what it means physically.

[bhouston]

Here is the definition of a matCap - it apparently has its full lighting (both diffuse and specular) baked into it so there isn't much one can do to manipulate it properly in real-time:

http://cgcookie.com/unity/2013/02/18/introduction-to-matcaps-in-unity/

[WestLangley]

Reposting from a previous comment:

Here is how I would implement the aoMap calculation:

totalAmbientLight = ambientLightColor * ( 1.0 + aoScale * ( aoMap - 1.0 ) );

outgoingLight += diffuseColor.rgb * ( totalDiffuseLight + totalAmbientLight ) + totalSpecularLight + emissive;

[pailhead]

AO can also be dubbed a "dirt map" and used slightly differently.

We don't have true indirect lighting in realtime just yet, it's too expensive. You might want to check out light propagation volumes, or the spherical harmonics paper, this is about as close as you get to indirect lighting in a game engine. Another algorithm to calculate the first bounce of light is called reflective shadow maps.

So what we do is, we fake all this stuff. Blinn-phong specular reflection is a complete approximation but it works great. Ambient 'lighting' used to be just slapping a constant color to everything that was not lit. It of course, looks very bad. You can add some variety by using a cubemap for this, or spherical harmonics. You can simulate things like a bright sky and a dark floor, or perhaps reflection (indirect lighting) coming from the green and red walls in a cornell box. This will already give a bit of a kick but it's not really enough. The next phenomenon we can approximate is AO - how much of the ambience is occluded at any given point.

A raytracer would take the normal at that point, the hemisphere around it (past 90 degrees no lighting affects the surface, the same with N dot L) and send a bunch of samples around that hemisphere, and check for stuff. In this case, it would check if it hits another surface or not - hitting the sky). Normalize this and you get the AO at that point.

In realtime we approximate this in screen space, by treating what we rendered as a heightfield is my understanding. Instead of raytacing we do ray marching - we shoot rays from every pixel and walk along them looking at the depth buffer at every step. A lot of assumptions have to be made.

Anyway, once we're done with all this, we can say that this result - AO - is how much we see of that previous cubemap, or just straight up constant color. In case of the cornell box, this cubemap would be a low resolution, blurry (convolved - look up modifiedCubeMapGen ) capture of the surrounding box. A crease in the model would know that it needs to get less lighting that bounced off a wall.

I'm not sure how this applies to lightmaps. Lightmaps already store this information, but you do probably want the dynamic objects to cast some shadow on them, and AO would simulate this effect.

The way I used AO was to render to texture, and then set that texture available in every shader. That way i could tweak where it happens and what it affects, as opposed to using it as a post effect and multiplying everything.

Perhaps just exposing some flags like AOAffectDirect, AOAffectAmbient, AOAffectSpecular (reflection) and include all in the shader?

[bhouston]

@pailhead Thanks for the detailed explanation.

BTW we know how to do blurry cubemaps in real-time using a WebGL Extension: #5847 (comment) Also we have Cornell boxes with lightmaps: https://clara.io/view/0d967025-ccbc-402c-a554-c0a4c4d491fc/webgl

[crobi]

Anyway, once we're done with all this, we can say that this result - AO - is how much we see of that previous cubemap [...]

This probably doesn't fit in here, but one can also extend AO from a scalar to include directional information. This directional occlusion would then not just tell you how much you see of the environment map, but rather which parts of the map you see.

I'm not sure this information can be stored in a texture, but it can be calculated efficiently in screen space: "Approximating Dynamic Global Illumination in Image Space", also known as SSDO. This is used by some game engines (cryengine for example).

[WestLangley]

@bhouston I believe this issue was fixed in #6369.

[bhouston]

sounds good! The only issue is that some people may use the AO map as a dirty map.