Add blend_premul_alpha to SpatialMaterial and shaders

[jitspoe]

Describe the project you are working on

2.5D platformer with lasers.

Describe the problem or limitation you are having in your project

There's a blend_premul_alpha, but it does not work in 3D shaders. I'd like to be able to have things like lasers that are mostly additive blended, but still have them able to show up on light backgrounds, so I want SOME of the alpha to darken things, but not a straight alpha blend.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

blend_premul_alpha allows the alpha darkening and color lightening to be controlled independently, so the background can be partially darkened, but not so much that it looks like a standard blend.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

This is with blend_mix:

Note how it just looks unnatural and not like light.

This is with blend_add:

Note how the laser gets lost in front of bright areas, like the train light. This is especially problematic with lasers in front of bright sky. Also, the color saturation gets lost and the lasers don't looks as red.

This is with blend_premul_alpha, using tweaked alpha values:

Notice how the intensity of the saturation still pops, but it doesn't look as unnatural as a pure mix blend. Parts can be made to draw on top of white as well, for better visibility

If this enhancement will not be used often, can it be worked around with a few lines of script?

Workarounds are costly, as they require multiple passes.

Is there a reason why this should be core and not an add-on in the asset library?

This is a core shader feature that can't be done with addons.

I've already created a pull request for this here: godotengine/godot#36747

[mortarroad]

How does this work in combination with lighting?
For example, one application for this would be having smoke + fire in one pass (e.g. in a particle system).
The fire itself should be emissive, but the smoke should be lit by light sources.

Currently your blending formula is essentially like this (correct me if I'm wrong):
OUTPUT = ALBEDO + OLD.rgb * (1 - ALPHA)

(where OUTPUT is the final value in the framebuffer and OLD is the previous value in the framebuffer)

what I propose is essentially the following:
OUTPUT = [EMISSION + ALBEDO * LIGHTING * ALPHA] + OLD.rgb * (1 - ALPHA)
(the part in brackets can be computed in the shader)

This blending mode could be called "premul_emission" or similar (because only the EMISSION is premultiplied).
Your example with the lasers could be done exactly the same, but instead writing to EMISSION instead of ALBEDO. But additionally, we would have support for lighting

[jitspoe]

Unfortunately, the output value is not directly programmable in OpenGL, so you're limited to things like glBlendFuncSeparate(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA, GL_ONE, GL_ONE_MINUS_SRC_ALPHA); (for normal mix), and I added glBlendFuncSeparate(GL_ONE, GL_ONE_MINUS_SRC_ALPHA, GL_ONE, GL_ONE_MINUS_SRC_ALPHA); for the premul alpha.

I think you could do what you're suggesting using the premul alpha mix mode, though, since you can basically do <whatever> + OLD.rgb * (1 - ALPHA). Just need to specify that <whatever> in the shader.

[mortarroad]

Exactly, it should be computed in the shader. But that should happen invisibly to the user.
At least to me, the way I suggest is more intuitive.

[jitspoe]

Perhaps that behavior could be some sort of flag for the material? I'm not sure that renaming the blend mode to be different from what's going on under the hood and what is established terminology is the best approach.

What you describe does feel like it should be the default behavior, though. How are things like additive blend mode handled with lit surfaces?

[mortarroad]

How are things like additive blend mode handled with lit surfaces?

Like any other additive surface. The lit color is computed, multiplied with the ALPHA and added on top.

[sserafimescu]

This feature would be great. The most common use case of using a transparent background viewport in 3D is to render a transparent image that will later get composited with the main scene. But with blend_mix, the color output gets multiplied by alpha while being rendered to the viewport, and then multiplied again when the viewport texture is rendered to the screen, resulting in incorrect (darker) colors in semi-transparent areas. The workaround is to divide colors by alpha in the shader, which is ugly, imprecise and adds unnecessary shader instructions. The second most common use case is storing non-color data in RGBA format, such as a density-velocity field resulting from a fluid simulation written in a shader; in this case, the fact that the renderer multiplies RGB by A is horrifying.

[OhiraKyou]

Lighting

First, I'd like to address concerns about lighting. And, to do that, I need to describe what lighting is and how it relates to fragment shaders. This is all based on my understanding of lighting calculations, but I would consider Godot's implementation broken if it didn't work this way.

To simulate light on a surface, you simply multiply the surface color by the light color for every light, and add the results together. Finally, you add emission (light not affected by—and, therefore, not multiplied with—the surface color). In a raw CG program, this is what you would provide as your shader's output color.

However, engines like Godot often provide surface shader languages that break this single output down into multiple intermediate outputs, so that the engine can handle the meta stuff (like looping through each light affecting a surface). But, ultimately, the final output of the shader is still just a color. And, that color is what is blended as rendered pixels are layered onto the screen.

In other words, the final color includes the combined influence of surface color (ALBEDO), light, and emission. You could do light calculations yourself and apply them to ALBEDO in an "unshaded" (or "unlit") shader. That's what I do, for control. In Unity and Blender, that would be done through emission instead, because that's the value to which lighting is not applied and that represents light from the object itself. But, in Godot, you use ALBEDO for that.

In any case, if you are using an engine's lighting features, it makes sense to use the separate, intermediate shader outputs provided by the engine (e.g., ALBEDO and EMISSION) instead. Ultimately, however, these are just abstractions. The result used for blending is just a color that is the result of calculations the engine did for you based on those intermediate outputs.

Because lighting is multiplicative with the surface color, and the order of operands doesn't matter for multiplication, a premultiplied ALBEDO output should automatically effect all surface lighting and exclude emission. And, if you want emission multiplied by alpha too, you can just multiply EMISSION by alpha too. So, there should be no need for special treatment of lit shaders; you just multiply alpha with ALBEDO and/or EMISSION in your fragment shaders as you please.

@mortarroad (original comment)

what I propose is essentially the following: OUTPUT = [EMISSION + ALBEDO * LIGHTING * ALPHA] + OLD.rgb * (1 - ALPHA) (the part in brackets can be computed in the shader)

This blending mode could be called "premul_emission" or similar (because only the EMISSION is premultiplied).

The emission is the only thing not premultiplied in both your example and my description of a typical lighting model.

Applying alpha

The expected workflow for appying alpha premultiplication is as follows:

• If you want just your surface color and lighting premultiplied, multiply your ALBEDO output by alpha in your fragment shader.
• If you want just your emission premultiplied, multiply EMISSION by alpha in your fragment shader.
• If you want both premultiplied, multiply both ALBEDO and EMISSION by alpha in your fragment shader.

There is no need to complicate this with superfluous blend modes specific to intermediate shader outputs; just manipulate the data in your shader as you please.

Standard material

Like any shader, the standard material shader could include properties to control what its alpha affects, assuming it is modified to support premultiplied alpha blending. But, that's a UI issue rather than a blend mode issue. You could just add a checkbox for each thing you might want to multiply with alpha. Then, you could do the following:

ALBEDO = mix(ALBEDO, ALBEDO * ALPHA, albedo_alpha_multiplication);
EMISSION = mix(EMISSION, EMISSION * ALPHA, emission_alpha_multiplication);

With those names, which do not imply a binary state (i.e., not "x_enabled"), you could even use 0-1 float sliders instead of bool checkboxes, to apply only partial premultiplication.

[OhiraKyou]

Premultiplied alpha blending

Because premultiplied alpha blending is regularly misunderstood by engine developers and users alike, an example of its usage and a description of its intended effects is in order, for readers visiting from anywhere and anytime.

On the engine side, in particular, Unity had recurring incidents of the blend mode implementation being unusable due to automatic operations being applied to shader output. See this Unity forum thread for details.

Description

The blend operation for premultiplied alpha is additive. So, a pixel's color is added to the existing background pixel. And, their contributions to the final pixel are defined by two multipliers—the blend factors—which are defined as One OneMinusSrcAlpha. These represent the following:

• A multiplier that our shader's output RGB will be multiplied by before blending.
• A multiplier that the existing background RGB will be multiplied by before blending.

Source (our layer)

The first One means that our shader's output RGB will always be added to the background as-is, because it will be multiplied by "one" (rather than something else) before the additive blending. Therefore, it is both our responsibility and our privilege to perform any multiplication or other operations we want to our RGB value before committing it to the shader's output. We control precisely how much of our RGB hits the canvas.

Traditionally, for premultiplied blending, this is done by simply multiplying the source RGB with the source alpha value. And, for shaders with multiple intermediate outputs (e.g., ALBEDO and/or EMISSION), we can specifically multiply any of those by alpha to affect the final color. And, we could also involve a separate mask texture or calculation, which could, then, be used to scale the output RGB or its intermediate data as desired. We can mask or otherwise modify any part of our data in any way we want before it is added to the background.

Destination (the background)

The next part of the blend operation—OneMinusSrcAlpha—says that the existing background will be multiplied by one minus whatever alpha value our shader outputs. In other words, alpha output becomes background obscurance. And, through our shader's alpha value we can control how much of the existing background pixel makes its way into the final blended pixel.

If we output an alpha (background obscurance) of 1.0, the background will be completely obscured or nullified, resulting in a blank (black) canvas for our shader's RGB output. And, if we output an alpha of 0.0, the background will remain fully unobscured and be added to our shader's output RGB. We may output the same alpha value we multiplied the RGB by, which is common. But, because the shader's output alpha simply controls background obscurance in this blend mode, we can do much more interesting things with it.

Background obscurance remapping

A significant benefit of premultiplied alpha blending is that a single shader can be used for additive blending, alpha blending, and anything in between. And, this is done by by manipulating the alpha output. The following steps remap alpha to background obscurance:

1. Multiply source RGB with source alpha (premultiplication).
2. Set ALBEDO to the premultiplied RGB.
3. Remap the [0, 1] alpha value to a custom range (mix(min_obscurance, max_obscurance, alpha)
4. Set ALPHA to the remapped alpha-to-obscurance value.

With that done, and a min_obscurance value set to 0.0, you can toggle between additive and alpha blending by changing the max_obscurance value between 0.0 (additive blending) and 1.0 (alpha blending). Or, you can select something in between the two.

Example shader

Of course, the following example shader is untested in-engine. And, it's very basic, with no additional source or destination masking or processing steps.

shader_type spatial;
render_mode unshaded, blend_premul_alpha;

uniform vec4 base_color: source_color = vec4(1.0, 1.0, 1.0, 1.0);

// Alpha to obscurance remap start value, associated with an alpha of 0.0
uniform float transparent_background_obscurance: hint_range(0.0, 1.0) = 0.0;

// Alpha to obscurance remap end value, associated with an alpha of 1.0
// When transparent obscurance is 0.0 and opaque obscurance is 1.0, this shader acts like blend_alpha
// When transparent obscurance is 0.0 and opaque obscurance is 0.0, this shader acts like blend_add
uniform float opaque_background_obscurance: hint_range(0.0, 1.0) = 1.0;


void fragment() {
	vec3 base_rgb = base_color.rgb;
	
	// Vertical alpha gradient, for testing
	float base_alpha = UV.y * base_color.a;
	
	// This is the premultiplication part
	vec3 premultiplied_rgb = base_rgb * base_alpha;
	
	// And, this is where the power of premultiplied alpha comes in.
	// In this blend mode, alpha output controls background obscurance.
	// So, we can choose how and when obscurance occurs as we please.
	// You could, for example, sample a separate obscurance texture.
	// In this example, we simply remap alpha to desired obscurance.
	float background_obscurance = mix(
		transparent_background_obscurance,
		opaque_background_obscurance,
		alpha
	);
	
	// Commit unshaded output
	ALBEDO = premultiplied_rgb;
	ALPHA = background_obscurance;
}

Expectations

All that being said, what follows are my expectations from a premultiplied blending mode implementation.

I expect to manually multiply my shader's output values by alpha, because it is my responsibility as a shader author to select which intermediate values (e.g., ALBEDO and/or EMISSION) I wish to affect by simply affecting them myself. Multiplication is such a simple operation that any abstraction layered on top would be excessively convoluted.

I expect to manually set the ALPHA output of my shaders, because I want full control over if and how alpha is mapped to background obscurance.

I expect the ALPHA output of my shaders to directly control background obscurance as a result of the One OneMinusSrcAlpha blend factors.

[jitspoe]

Regarding the lighting, it would be nice if this could work by having alpha apply to the albedo and emission effectively always be additive, however, the lighting does more than just multiply the light value with the albedo value. There are things like specular, (reduced) roughness, and Fresnel effects that add light to even a solid black albedo. This results in artifacts of additive light showing up on the alpha'd out areas. Perhaps this could be worked around by multiplying the alpha after the lighting pass, but I'm not sure if that would result in any other artifacts.

[clayjohn]

To simulate light on a surface, you simply multiply the surface color by the light color for every light, and add the results together. Finally, you add emission (light not affected by—and, therefore, not multiplied with—the surface color). In a raw CG program, this is what you would provide as your shader's output color.

This is only correct when talking about diffuse light. Its not true for specular lighting which uses a more complicated formula than light_color * albedo. Also the GL Compatibility backend treats ALBEDO as an sRGB color which is converted to linear before calculating lighting. Premultiplying before the sRGB -> linear conversion will result in very wrong results. The premultiplication needs to happen right before writing the final color (GL Compatibility will be a problem for the current draft PRs as well). So unfortunately, things aren't so simple for shaded materials,

I still found your comments very helpful, especially the link to the unity forum post.

I note especially Ben Golus' comment in the linked forum post:

The only reason to include the option for premultiplication as a blend mode is if you have content that is already setup to be used with a premultiplied blend mode. If the shader is forcibly multiplying the color by the alpha, then that negates the usefulness of having the blend mode to begin with. By multiplying the color by the alpha it means Alpha Blend and Premultiplied Alpha produce identical results, which they should not.

What makes the current PR so useful is that users can control the amount of alpha blending. So it would be nice to capture that ability in shaded materials as well.

[OhiraKyou]

This is only correct when talking about diffuse light. Its not true for specular lighting which uses a more complicated formula than light_color * albedo.

I treat dielectric reflections, like Fresnel and specular (any light that does not interact with the surface color) as emission, because it's just more additive light. Although, now that Fresnel was mentioned, I do remember a RIM output in the docs. So, I see that Godot actually treats them as separate values. But, it would be easier to manage if all additive light were considered the same and all multiplicative light considered the same and things like RIM_TINT just split the input between the two.

Edit: I should note that I'm only referring here to how the standard material shader calculates and modifies additive light. Obviously, you can do whatever you want in a custom shader, including outputting rim and specular effects as part of the emission output.

Also the GL Compatibility backend treats ALBEDO as an sRGB color which is converted to linear before calculating lighting. Premultiplying before the sRGB -> linear conversion will result in very wrong results.

I'd really like to see that changed, if possible. Although, I just use unlit shaders anyway.

[clayjohn]

I'd really like to see that changed, if possible. Although, I just use unlit shaders anyway.

It is that way because older phones have really bad support for mixing sRGB and non-sRGB framebuffers/textures. Which means all inputs/outputs need to be in sRGB space. We can't really change that design decision without compromising the entire reason we have a Compatibility backend (i.e. to provide good support for older devices).

For the purposes of this proposal, it's best to treat that design as fixed and discuss ways to achieve the desired effect without throwing out compatibility with older devices

[OhiraKyou]

Fair enough on the color spaces (was just a side note).

Also, I should clarify the difference with the Ben Golus quote and that old Unity situation, because someone might think the quote means you shouldn't multiply a color by alpha in your own shader (that isn't the case).

The issue with Unity was that it was treating One OneMinusSrcAlpha as SrcAlpha OneMinusSrcAlpha due to the automatic multiplication of the source color by alpha. In other words, it was modifying both the source and destination by the same value (alpha), resulting in the two being coupled.

The important distinction in my previous example shader is that the source and destination are not both multiplied by alpha. Rather, the source is multiplied by alpha, and the destination is multiplied by a custom value that is alpha remapped to background obscurance via uniform sliders (which could be replaced with any other arbitrary, configurable mask). So, unlike alpha blending, you have decoupled control over the source and destination multiplication, allowing the toggle between additive and alpha blending with an opaque_background_obscurance slider (or, again, any other mask).

In other words, the value of premultiplied alpha is that the shader's alpha output isn't actually alpha; it's just background obscurance, and you can derive that from anything you'd like.

Example

This additional example shader isn't particularly useful. But, it specifically illustrates the point on decoupling.

shader_type spatial;
render_mode unshaded, blend_premul_alpha;

uniform float albedo_alpha: hint_range(0.0, 1.0) = 1.0;
uniform float background_obscurance: hint_range(0.0, 1.0) = 1.0;


void fragment() {
	// You can apply alpha to your color...
	vec3 base_rgb = vec3(1.0);
	ALBEDO = base_rgb * albedo_alpha;
	
	// ...and apply an entirely different "alpha" as background obscurance.
	ALPHA = background_obscurance;
}

[clayjohn]

@OhiraKyou @jitspoe We discussed this proposal in our weekly rendering meeting this week as I wanted a few more eyes on it and we came up with another solution that may be acceptable. We can add a built in called PREMUL_ALPHA to the spatial material fragment shader. PREMUL_ALPHA will be multiplied by the final color at the end of the fragment shader.

OhiraKyou's last example could then look like:

shader_type spatial;
render_mode unshaded, blend_premul_alpha;

uniform float albedo_alpha: hint_range(0.0, 1.0) = 1.0;
uniform float background_obscurance: hint_range(0.0, 1.0) = 1.0;

void fragment() {
	// You can apply alpha to your color...
	vec3 base_rgb = vec3(1.0);
	ALBEDO = base_rgb;
	PREMUL_ALPHA = albedo_alpha;

	// ...and apply an entirely different "alpha" as background obscurance.
	ALPHA = background_obscurance;
}

For unshaded cases this would result in the exact same output as the existing PR. For shaded cases this would allow specifying the premultiplication amount globally (would apply equally to emission + light + ambient). Importantly, this would work with the Compatibility renderer as well.

I don't love adding a new built in, but I think it might be unavoidable in this case

The only thing it misses is the workflow described by Mortarroad where you mix smoke and fire in the same shader (for example). But I think that Mortarroad's suggestion probably deserves a separate blend mode anyway if there is demand for it as it is not clearly premultiplied alpha anyway.

[OhiraKyou]

The new built-in could also be given a more general-purpose and descriptive name, like OUTPUT_RGB_MULTIPLIER (not OUTPUT_COLOR_MULTIPLIER, seeing as it doesn't multiply alpha), and simply applied anytime it's assigned (or default it to 1.0) rather than just in the premultiplied alpha blend mode. It could be useful any time the source blend factor is One, really, including additive blending.

[jitspoe]

I feel like if we're going to add another built-in, we might as well make it allow for the emission feature, as I picture that being a key use of this blend mode (fire + smoke). Effectively, we'd have 2 alphas. Not sure what the best names of them are so I'll just say:

Alpha 1 - controls the complete transparency/fade (so things like particles with premul alpha could fade out). Multiplies emission, albedo, and alpha. If set to 0, nothing is visible.
Alpha 2 - Controls the alpha channel for darkening the background. Albedo and emission are added on top of the result. Lighting is also multiplied by this, so areas that have an alpha2 of 0 will not get any lighting added.

Not sure if that'd be 100% correct looking, or which one should be the default "ALPHA" (the alpha alpha?), but it could allow for some nice FX.

[OhiraKyou]

Alpha 2 - Controls the alpha channel for darkening the background. Albedo and emission are added on top of the result. Lighting is also multiplied by this, so areas that have an alpha2 of 0 will not get any lighting added.

In your example, "Alpha 2" should not influence lighting at all, because it should only be responsible for background obscurance ("darkening the background"). This is the existing ALPHA shader output, whose name is already appropriate due to its role in the blending factor OneMinusSrcAlpha (the part that obscures the background by multiplying it against 1-ALPHA).

The proposed additional multiplier represents a modifier against the remaining output channels (RGB). The reason I suggest OUTPUT_RGB_MULTIPLIER for this is because it multiplies the RGB output used in the blending operation, which is a result of ALBEDO, lighting, EMISSION, and any other intermediate shader outputs that contribute to the shader's RGB.

[jitspoe]

If you don't scale the lighting, it will end up added to black areas, like this:

Unshaded premul alpha on the left, lit premul alpha in the middle, and lit mix mode on the right.

I think the expected behavior would be the one on the right + emission used on the flame?

[OhiraKyou]

@jitspoe

In your examples, you said the following:

Alpha 1 - controls the complete transparency/fade

It's not controlling the complete transparency/fade if it's not doing so for lighting.

Alpha 2 - Controls the alpha channel for darkening the background

This is the responsibility of the existing ALPHA output.

That being said, if you mean to have multiple multipliers to allow emission to be affected separately, than it would make sense to have one multiplier for everything other than emission and one for emission itself.

• SHADED_RGB_OUTPUT_MULTIPLIER - multiplies/masks everything other than emission.
• EMISSION_OUTPUT_MULTIPLIER - multiplies/masks just emission.
• ALPHA - controls background obscurance, as usual.

The names may give off some major code smell, despite their accuracy, because "why wouldn't I just multiply emission myself?" But, from what I gather from @clayjohn's comments, the emission would be converted from sRGB to linear in the compatibility renderer before its multiplier is automatically applied.

Also, see my comment about the importance of decoupling any and all RGB output multipliers from the background obscurance (ALPHA), which should only ever have that one job.

[QbieShay]

Just note @OhiraKyou that we would like to avoid adding a lot of built-in, especially for advanced use cases. Adding built-in complicates documentation and onboarding time to shader and it is a priority for us to keep Godot as simple as possible. It would be easier to reach consensus if we can find a solution that expands minimally the built-in (one is ideal) and addresses your concerns as well.

[OhiraKyou]

If you want to multiply lit RGB and emission separately, than you need separate multipliers for each. That's pretty much all there is to it. ALPHA should only ever control background obscurance as a natural consequence of the blend operation. It should never have additional responsibility tacked on.

[QbieShay]

Alright! So in order to move this forward, what's the next steps?

For me premul alpha is something I'd only use with unshaded so I'm curious as of why people want to use emission, if it's necessary

[OhiraKyou]

Steps

So in order to move this forward, what's the next steps?

First, I would just merge the blend mode feature immediately so that it can be used with unlit shaders. Lit shader support can be added in an additional pull request. Premultiplied alpha blending is a basic shader feature that shouldn't be held back by lighting implementation details.

Usage

For me premul alpha is something I'd only use with unshaded so I'm curious as of why people want to use emission, if it's necessary

The example brought up a few times here is fire and smoke in a single pass. The fire is emissive and unshaded. The smoke is shaded. Another example was a candle and its flame. I'm assuming these effects would use separate textures for the flame elements. This would result in separate alpha channels and, therefore, separate multipliers for the lit RGB output and emissive RGB output. And, the output ALPHA may be adjusted in the shader to change how additive the flame is.

The lit RGB multiplier has to be stored as a built-in so that it can be automatically applied to all light effects (including specular and Fresnel) and to ensure that this takes place after the compatibility renderer's delayed sRGB to linear ALBEDO conversion. However, I'm not sure about EMISSION. Assuming it's just standalone additive light, perhaps users could just multiply their emission outputs themselves in-shader rather than assigning a second built-in. In that case, there are two options:

1. An overall RGB output multiplier controls the overall effect fade, including emission. Emission can still be faded by multiplying it manually, but lit RGB cannot be faded separately. In other words, you can't fade the smoke without also fading the fire. But, you can fade the fire without fading the smoke.
2. A lit RGB output multiplier fades everything except emission, and users are expected to multiply their emission output themselves. In other words, the fire wouldn't fade automatically with the smoke; you would have to modify your EMISSION output to do that. But, you can fade either layer independently.

Bear in mind that I haven't looked into the engine's source. So, I can't confirm the viability of a theoretical implementation.

As an additional usage example, I should note that I simply don't use additive or alpha (mix) blending when premultiplied alpha is available, because the former two become redundant; premultiplied alpha can do both and anything in between. So, I would, typically, use a premultiplied alpha shader to fade out otherwise opaque, lit, and possibly emissive objects. That would include fading out defeated enemies, fragments of exploded objects, and objects blocking line of sight from the camera to the player.

[jitspoe]

I think there are 3 things that are of concern here:

1. Simplicity.
2. Intuitiveness.
3. Consistency

For simplicity, we want to avoid adding lots of built-ins that bloat the documentation and make it harder to figure out how to use the straightforward stuff.

For intuitiveness, we kind of want things to "just work" the way people would expect if they've used premul alpha before.

And finally, consistency: Currently, you can use ALPHA to fade out multiple different materials with different blend modes. Blend, Add, and Subtract will all disappear with an alpha of 0 (Though testing this I notice there is an inconsistency with multiply). Also, it should probably behave consistently with how premul alpha works in 2D.

So here are the gotchas:
If ALPHA is intuitive in that it acts like normal premul alpha and ONLY darkens the background to mask, and everything else is added, we can't use ALPHA to fade sprites out like we would with the other blend modes.

The other gotcha is that, since ALPHA is already multiplied with the emission and some other values, it means we can't use ALPHA to mask out the background and EMISSION to add on top (where there is no mask). Another intuitive break. We could disable the alpha multiply of other things for just this blend mode, but then it's a mark against consistency.

We could add a new built-in, say ALPHA_PM, which would be the output alpha that's used for the mask. This is less intuitive, but that means the existing ALPHA could behave like it currently does and be multiplied with the EMISSION, ALBEDO, and ALPHA_PM to basically be a full fade control for consistency. This would be a breaking change if we merge the current implementation and add it later after people used ALPHA with premul alpha.

The simplest solution would be to not add a new built-in and special case the way ALPHA behaves in this mode. Don't multiply anything by alpha (emission, etc) EXCEPT we add a new multiply with alpha for the lighting so "shiny" light doesn't show up in the areas that are black on the texture and have 0 alpha. That covers simplicity + intuitiveness, I think, but breaks consistency. (Also, I'm not sure how involved it is to special case the shader to only multiply emission and things based on the blend mode, so the implementation itself may not be simple).

Hopefully that all made sense. Not super awake right now. 😅

[OhiraKyou]

Replying to the previous comment by @jitspoe

For simplicity, we want to avoid adding lots of built-ins that bloat the documentation and make it harder to figure out how to use the straightforward stuff.

In the pursuit of simplicity, complexity is often traded for convolution.

The simplest solution is to let the blending mode work the way it's intended and assume that users know why it produces the results that it does.

For intuitiveness, we kind of want things to "just work" the way people would expect if they've used premul alpha before.

Yes. Users who know how premultiplied alpha works shouldn't have to look up how and why Godot's implementation is different.

Currently, you can use ALPHA to fade out multiple different materials with different blend modes.

The whole point of premultiplied alpha blending is that alpha's role of fading is replaced with background obscurance. By merging the role of fading the foreground color into RGB, you gain the ability to control the background and foreground opacity independently.

When the point is that it works differently, trying to make it work the same doesn't make much sense.

Blend, Add, and Subtract will all disappear with an alpha of 0 (Though testing this I notice there is an inconsistency with multiply).

What you observed with the multiply blend mode is another natural consequence of a blend mode working as intended. The factors for multiplicative blending are GL_DST_COLOR, GL_ZERO. The source color is multiplied by the destination color. The result is added to zero multiplied by the destination color (so just zero). Because the alpha channel is not used for the blending operation, it has no visible effects, regardless of the alpha channel's multiplication result.

The alpha channel is only relevant when using a transparent canvas background, and Godot just uses the destination alpha otherwise. See these multiplicative blending source lines for implementation details. Also see this Unity forum reply for further engine-agnostic explanation.

As for additive blending, these additive blending source lines suggest that GL_SRC_ALPHA, GL_ONE may be used (assuming PASS_MODE_COLOR_TRANSPARENT simply means it's rendering transparent geometry). So, for additive blending as well, visible affects of alpha are a consequence of the blend factors.

Shaders must be authored with their blend mode in mind and implement their own alpha response when one is desired and not part of the blending operation. For multiplicative shaders, I usually lerp (mix) from white (no change to the background) to the intended output, using the alpha as the lookup. Pseudocode: rgb = mix(white, rgb, alpha)

As another example, One One is also additive blending, but without alpha. So, to apply alpha, you'd have to multiply your source RGB and alpha manually in-shader. Pseudocode: rgb *= alpha

Similarly, in premultiplied alpha, the alpha channel does not affect the source (foreground) color. So, we have to do that ourselves in-shader, if and how we desire. But, alpha does automatically affect the background color, and that decoupling is its magic trick.

We could add a new built-in, say ALPHA_PM, which would be the output alpha that's used for the mask. This is less intuitive, but that means the existing ALPHA could behave like it currently does and be multiplied with the EMISSION, ALBEDO, and ALPHA_PM to basically be a full fade control for consistency.

Again, the point of premultiplied alpha blending is that alpha no longer represents opacity. If I do anything that changes alpha, I expect it to manifest as a change to background obscurance.

Don't multiply anything by alpha (emission, etc) EXCEPT we add a new multiply with alpha for the lighting so "shiny" light doesn't show up in the areas that are black on the texture and have 0 alpha.

This would cause a lighting fade along an authored background obscurance (ALPHA) gradient. For example, a background obscurance (ALPHA) of 0.5 would show full emission, but half lighting. So, even if you wanted a full fade opacity slider, this wouldn't be it.

Ultimately, when using premultiplied alpha blending, shaders, materials, scripts, and users must all be aware of the shift in alpha's role (from opacity to background obscurance) and the implications that has on the resulting workflow. If you want a mask or full opacity slider, you add it to your shader.

[QbieShay]

What we can do is to add one built-in, like clay suggested some time ago called PREMUL_ALPHA and at this point we can alter how the light is calculated for shaded materials when premul alpha is used. What I'd do moving forward is

• add premul alpha built in
• support it only for unshaded materials (for now)
• in a follow up PR and discussion, add support for shaded materials

So at this point we can have alpha and premultiplied alpha to work with, which I gather should be enough?

So the question is, how to use alpha and premul alpha built-in to make sure they both work in an expected way. Would alpha as opacity and premul alpha as background obscurance work?

While I don't understand 100% the actual way premul alpha works, I known it's rather important for VFX and I'd like to unblock this conversation and possibly have the feature, at least for unshaded, merged by 4.3, if we can

[OhiraKyou]

The existing PR already adds functional premultiplied alpha blending for unlit shaders as-is. This is why I suggested simply merging it and dealing with lighting in an additional PR.

The purpose of the built-in (and the hold-up on merging the PR) was to enable premultiplication of lighting, including specular and Fresnel effects (and emission, separately). As far as I know, there's only one benefit it could have on unlit shaders. According to this comment by @clayjohn, "the GL Compatibility backend treats ALBEDO as an sRGB color which is converted to linear before calculating lighting". Because applying the built-in multiplier would be delayed until after this conversion, if nothing else, it would ensure consistent math across renderers. So, there's that, for whatever it's worth.

Would alpha as opacity and premul alpha as background obscurance work?

ALPHA should be background obscurance, as specified in the blend factors GL_ONE, GL_ONE_MINUS_SRC_ALPHA. Coming to Godot with prior experience writing premultiplied shaders, I expect anything that affects alpha to affect background obscurance, because that's the whole point of premultiplied alpha blending.

[QbieShay]

Hey,

Sorry for the long time before an answer here. I have previously not taken enough time to fully understand how premul alpha works, so apologies for that, and thank you @OhiraKyou for your patience in explaining it to me.

I think the best way forward now will be to implement like clay suggested. I am skeptic of using premul alpha for shaded materials, but I am curious to see what people will do with it.

Since 4.2 released today, we'll take some time to rest and recollect, then we'll get back to work. I expect to write a technical proposal for premul alpha that will supersede this issue (just for clarity so that people don't need to skim through the comments to find the approach we want to go with).

Thanks y'all 🙌

[QbieShay]

Hey 👋

Before approaching a design document, i tried to dig in the code a bit to see if there's any unknowns that should be considered. In order to help this effort, could you all test your projects against godotengine/godot#85609 (once it's done building)

In particular, im looking for feedback for the lit case. Note that it's a rather pedestrian implementation and works only for forward+.

I don't know if it covers properly all the usecases outlined above, so i need y'all to make sure it does ^^

@OhiraKyou @jitspoe I am following this formula : result = source.RGB + (dest.RGB * (1 - source.A)) from https://microsoft.github.io/Win2D/WinUI3/html/PremultipliedAlpha.htm#:~:text=To%20convert%20a%20straight%20alpha,G%2C%20and%20B%20by%20A.

It seems that the formula in jittspoe's PR doesn't allow to control obscurance separately, but the formula I'm using should.

Let me know if it works! I have tested locally with unshaded materials.
Right now this is shader-only, it's not used by spatial materials. I think maybe for spatial material obscurance can be controlled together with alpha?

[OhiraKyou]

@QbieShay

I am following this formula : result = source.RGB + (dest.RGB * (1 - source.A)) from https://microsoft.github.io/Win2D/WinUI3/html/PremultipliedAlpha.htm#:~:text=To%20convert%20a%20straight%20alpha,G%2C%20and%20B%20by%20A.

It seems that the formula in jittspoe's PR doesn't allow to control obscurance separately, but the formula I'm using should.

You are both using the same formula.

The part that allows controlling obscurance separately is the blend factors (multipliers), which only include alpha as part of the second (background, or "destination") blend factor and not the first (foreground, or "source") blend factor. These blend factors are One OneMinusSrcAlpha, which both @jitspoe's PR (here) and yours (here) use for RGB. Because alpha is a value exposed to shaders, users have control over background obscurance through alpha. And, because alpha is only part of the background blend factor, it is decoupled from the source opacity—which users can set in any way that they want in their shaders by modifying the output RGB.

For canvases with opaque backgrounds (i.e., rendering to the screen), the RGB values are the imporant part, because the alpha channel is, typically, full of junk and ignored. However, for canvases with transparent backgrounds, your alpha component blend factors (here) are strange. Because blend factors are multipliers, you're instructing the source alpha to be multiplied by itself (squared) and then added to the unchanged destination alpha (which is multiplied by one). One OneMinusSrcAlpha (as in @jitspoe's PR here) seems more practical.

Also, your PR seems to be missing the additions to drivers/gles3/rasterizer_scene_gles3.cpp in @jitspoe's PR at these lines. I'm not familiar with the inner workings of the engine, but I'm assuming that's relevant to the compatibility renderer.

@jitspoe's PR also has a line that claims to force transparency when using the premultiplied alpha blend mode here. And, it adds the necessary blend_premul_alpha render_mode to material.cpp and to spatial shaders through shader_types.cpp (in addition to updating associated property information), allowing it to actually be used by shaders. It also updates the documentation.

So, @jitspoe's PR seems significantly more correct and complete. I believe it should be merged by itself, to immediately enable using the blend mode in custom spatial shaders. Any additional built-ins can, then, be added as part of a separate PR. Using the blend mode raw simply requires setting RGB as desired and using alpha to control obscurance. The multiplier built-in addresses timing (after linear conversion), which is an issue separate from the blend mode (render_mode) implementation itself.

[QbieShay]

Yes my PR is a work in progress for now ^^ that's why it's incomplete. Sorry for not mentioning it explicitly.
I need it as an intermediate step to make sure that the approach taken addresses correctly the concerns for lit objects.

Thank you for reviewing the formula. I'll adapt it and push an update later! Same for correctly assigning the object to the alpha queue.

Jitspoe's PR will not be merged, this is what we discussed in the rendering meeting: the render mode doesn't cut it for lit objects. For this reason, we need a different solution. ( But regardless many thanks for opening it. We didn't know either that we needed a different solution and having all this discussion has saved everyone a lot of future pain)

It is also not a good idea to merge both: offering two overlapping solutions will make the usage of premultiplied alpha even more confusing (and harder to maintain) down the line.

I understand that you wish for premultiplied alpha to be included asap, I share this wish, but merging a limited solution and expanding it later is an approach that we cannot take in this case because the two implementations are different both behind the scenes, and in their user facing API.

The best way to move this process forward is to test the PR I sent and verify it doesn't have shortcomings with lit objects, so it can be expanded to all rendering back and completed. I need testing only for lit objects, the rest is incomplete (missing backends, alpha queue)

Lastly, because this is text communication I want to be clear, I appreciate a lot everyone's involvement in this feature and I can't wait to have it ^^ I think all my VFX will look much getter once I start including it!

[OhiraKyou]

Posted my additional thoughts about the second PR's details on the PR itself, to keep the context close.

Moved the contents of my original comment to here, and restructured my PR comment to be more PR-specific and actionable.

[AThousandShips]

I would suggest making that comment here instead as it's more meta and not directly related to that PR as it talks of another PR being merged instead, which is more related to the proposal than that specific implementation

Discussing what should be done in general should be done here (i.e. what exactly to do, which solution to use etc.) whereas what to do specifically for the code in a PR should be done there 🙂

[OhiraKyou]

Detecting the use of a PREMULT_ALPHA built-in to enable a specific blend mode is inconsistent with other blend modes and doesn't always make sense.

Consider the following common use case: a user is intending to use a texture whose RGB is already premultiplied. This would require the user to assign a dummy PREMULT_ALPHA value of 1 to enable the blend mode without redundant multiplication instead of simply enabling render_mode blend_premul_alpha like any other blend mode.

I suggest the following:

1. Merge @jitspoe's existing blend mode PR to allow the blend mode to be used on its own.
2. Add a general-purpose DELAYED_RGB_MULTIPLIER vec3 built-in as a separate feature, instead of a blend mode specific PREMULT_ALPHA. This multiplier would be particularly useful any time the source blend factor is one, including both premultiplied alpha and additive blending. But, it should be available to any blend mode and enabled on use.

The "DELAYED" part of the general purpose built-in's name is important, as it communicates that it's the timing of the multiplier's application (after the sRGB to linear conversion) that justifies its usage. And, the vec3 type, instead of float, enables each of the RGB channels to be modified independently.

Basically, using the premultiplied blend mode shouldn't require using the delayed RGB multiplier, and using the delayed RGB multiplier shouldn't require using the premultiplied blend mode. These two features can be—and, I believe, should be—separate, for both consistency and versatility.

[QbieShay]

Like I said previously: this approach was discussed in a rendering team meeting and the outcome is that premultiplied_alpha will not be a render mode.

Feel free to join the team meeting https://chat.godotengine.org/channel/rendering and discuss this, I have no strong feelings either way, so I'll shelve this work for now.

[QbieShay]

Regardless of whether the render mode is implemented or not, if we implement the delayed multiplier like you suggested, it still needs to be tested with lit objects. Testing of the PR with lit object is still relevant, and welcome.

[QbieShay]

Hey good news! I was wrong :D I misunderstood the resolution of the rendering meeting. we'll have the blend mode and the built-in. I'll rebase my work on top of jitspoe's so we can have premul alpha soon ^^

[akien-mga]

Implemented by godotengine/godot#85609.

[jitspoe]

Migrating discussions from godotengine/godot#85609 to here so everything is in a more consistent location.

Based on the latest changes, we now have the option to use premultiplied alpha, but I can't seem to get the desired results using the new PREMUL_ALPHA_FACTOR.

Without it, I get this:

If I try to set it to the alpha value, it properly masks out the lighting on the black areas, but also masks out the additive flame:

Effectively making it the same as a normal blend mix or alpha blend mode (shown on the right).

Here's the test project if anybody wants to tinker with it:
test_premul_alpha_2.zip

I think the expected behavior would be to not require the PREMUL_ALPHA_FACTOR, and instead, when we're using the premul blend mode, scale the lighting and fog by the ALPHA, but not the entire color, so the flame would be added in and not impacted by lighting and fog.

[clayjohn]

@jitspoe you don't have to write anything to PREMUL_ALPHA_FACTOR. It just allows you to post-multiply a pixel by a given value. This gives you control over whether you want that pixel to behave like an additive or a mix blend mode.

Indeed, just multiplying every pixel by alpha will result in normal mix blending while writing nothing will result in additive blending. That's how premultiplied alpha works.

You have to mask the area you want manually as the engine can't guess for you

[jitspoe]

The area is already masked with the alpha channel. I think the expected behavior would be that anything with an alpha of 0 would not receive lighting and should be treated like additive blending when it comes to fog (which I assume fades toward black, instead of the fog color, but I haven't actually tested this).

So in short, some pseudo code...

if (blend_mode == mix) {
  color = lighting_calculations(color);
  color = mix(color, fog_color, fog_amount);
} else if (blend_mode == premul_alpha) {
  color = mix(color, lighting_calculations(color), alpha);
  color = mix(color, mix(0, fog_color, alpha), fog_amount);
}

[clayjohn]

@jitspoe I think this is a different issue entirely. I am completely against having a blend mode silently change the behaviour of fog in a way that is hidden from users. I think you are running into the more general problem of fog with additive materials (See godotengine/godot#56374 where we ultimately added the fog_disabled mode).

[jitspoe]

Ah, I would argue that additive blending should also have fog use black instead of the fog color by default. I think things should "just work" in an intuitive manner by default and if you want additive stuff to glow more with fog, that should be a flag or something manual, because that's not usually the intended behavior (can't actually think of a legit use case for that). Usually people want things to fade out with the fog.

Edit: Also, I don't think there's currently a way to get the desired behavior even via manually doing things. If I have a fog where I want to fade something with premul alpha out, how do I even do that? Not to mention the issue with lighting. The way the shaders are structured, you're at the mercy of whatever Godot does outside of the vertex(), fragment(), and light() functions, with no way to alter the final color before the final mix, right?

[clayjohn]

Ah, I would argue that additive blending should also have fog use black instead of the fog color by default. I think things should "just work" in an intuitive manner by default and if you want additive stuff to glow more with fog, that should be a flag or something manual, because that's not usually the intended behavior (can't actually think of a legit use case for that). Usually people want things to fade out with the fog.

You can easily disable fog or set FOG = vec4(0.0); if you want. These sort of workflow issues are pretty tricky. As I agree, it would be convenient just to automatically disable fog when using additive blending. I can't think of an obvious use-case for having fog enabled when using additive blending. That being said. Making a hidden change like that increases complexity in two negative ways:

1. In the engine: as it creates a new, non obvious codepath to maintain
2. For the user: Since this would unexpectedly change something, we need to provide an optional toggle for users to undo the change which increases the number of random settings that users have to look at. Keeping the number of exposes settings down is an important design goal to keep Godot feeling sleek and easy to use. This is a death by a thousand cuts type of thing

Edit: Also, I don't think there's currently a way to get the desired behavior even via manually doing things. If I have a fog where I want to fade something with premul alpha out, how do I even do that?

In that case you have to write the fog color yourself with FOG right now.

Not to mention the issue with lighting. The way the shaders are structured, you're at the mercy of whatever Godot does outside of the vertex(), fragment(), and light() functions, with no way to alter the final color before the final mix, right?

You can use FOG or PREMUL_ALPHA_FACTOR to alter the final pixel color. Or you can use an unshaded shader and write the final color directly.

[jitspoe]

It's not a matter of just disabling fog, though. Fog reduces the visibility of things in the distance, so if you disable it, you're going to have glowing flames standing out by themselves in areas with dense fog where other things aren't visible. You just have to change the color of the fog to black for additive blending (and 0-alpha premult alpha blending) so things fade properly. Perhaps I should whip up an example scene to illustrate the issue and create a proposal for this.

[Calinou]

It's not a matter of just disabling fog, though. Fog reduces the visibility of things in the distance, so if you disable it, you're going to have glowing flames standing out by themselves in areas with dense fog where other things aren't visible. You just have to change the color of the fog to black for additive blending (and 0-alpha premult alpha blending) so things fade properly. Perhaps I should whip up an example scene to illustrate the issue and create a proposal for this.

I made a testing project: test_material_fog.zip

Here's how it looks in master:

Forward+	Mobile	Compatibility

Using FOG = vec4(0.0); in fragment() doesn't fix the issue though:

Forward+	Mobile	Compatibility

I've also tried FOG.a = 0.0; and FOG.rgb = vec3(0.0); to no avail.

I've added the following to material.cpp at the beginning of the code that writes fragment():

	if (blend_mode != BLEND_MODE_MIX) {
		code += R"(
	// Disable writing to fog for material using non-Mix blend mode.
	// This prevents the material's silhouhette from being visible in the fog.
	FOG.rgb = vec3(0.0);
)";
	}