Fix custom fog, and make FOG built-in readable

[briansemrau]

Fixes shader custom fog (#41415), which was likely broken since #45023.

Additionally, the fragment shader built-in FOG allows for modification of standard fog, rather than just overwriting it. This allows for simple tweaks, such as:

FOG *= vec4(0.5, 0.5, 1, 1); // Modulate fog color

Linear fog could be implemented like so (taken from #54248):

float fog_linear = clamp((length(VERTEX) - fog_linear_start) / (fog_linear_end - fog_linear_start), 0, 1);
FOG.a = max(FOG.a, fog_linear);

One difference to note from the original custom fog PR is that custom fog now mixes before volumetric fog rather than after.

Will need updating in the docs (see: godotengine/godot-docs#4301)

[clayjohn]

Why calculate volumetric fog after? Wouldn't it make sense to do fixed fog and volumetric fog before custom fog?

[briansemrau]

Why calculate volumetric fog after? Wouldn't it make sense to do fixed fog and volumetric fog before custom fog?

I thought this would be better because I consider volumetric fog to be an additional effect beyond standard fog, and that this custom fog applies to standard fog only. For the linear fog example, this ordering is important.

However, I can see why the opposite might be preferred for other uses.

Perhaps this could be made configurable by adding a render_mode option such as fog_premix_volumetric.

[briansemrau]

@clayjohn Would you prefer I calculate volumetric fog alongside regular fog before custom fog, restoring original behavior? Or do you think adding a render mode option is a good idea?

[clayjohn]

@briansemrau That is a difficult question. To me it made sense to do them together, but for you it was more intuitive to have FOG only override fixed fog.

This highlights the issue when devs add features without proposals. I clearly don't have a good sense of what users need and didn't realize that users may have different expectations for this feature.

You or I should write a proposal and try to get some feedback from users.

[clayjohn]

I was thinking about this again. I wonder if the ideal approach is just to expose the fog_process and volumetric_fog_process to users and then leave the current fog alone. I.e. if users use FOG they will be responsible for integrating the other types of fog.

[RPicster]

I was thinking about this again. I wonder if the ideal approach is just to expose the fog_process and volumetric_fog_process to users and then leave the current fog alone. I.e. if users use FOG they will be responsible for integrating the other types of fog.

I fear that will make things a bit more complicated than most use cases would demand it.
So far my experiences were almost always fragment related, just having FOG exposed and useable would help me much more.

Another reason is that volumetric and normal fog are blended in magical ways no one understands, so having to reliably "reconstruct" the real fog color that is applied from two different functions... wouldn't really help me personally a lot 😵

Why not both? :D