Fix viewport with transparent bg changed to solid black if enable fxaa or debanding.

[Kinwailo]

If enable fxaa or debanding in viewport with transparent bg, the transparent bg will change to solid black color. It is because the shader always output alpha as 1.0.

Bugsquad edit: This closes #45208.

[Calinou]

Good catch 🙂

Does this fully address #45208, or are there still parts of the tonemapping shader that don't play well with Transparent Bg?

[Kinwailo]

Good catch slightly_smiling_face

Does this fully address #45208, or are there still parts of the tonemapping shader that don't play well with Transparent Bg?

Post processing is bypass if transparent is enabled in driver code, and the shader of glow / dof produce "black edge". I have changed the code and shader, should fix #45208.

[Calinou]

I tested this PR rebased against the latest 3.x branch as of writing. It indeed fixes the issue in the original MRP 🙂

Before

After (with this PR)

When Transparent Bg is disabled, the appearance is identical:

Before

After (with this PR)

[bfloch]

I was able to verify this with a project that was suffering from the issue in 3.4.3. Works great, no complaints.

[bfloch]

EDIT: The issue and the transparency not working was since my root node that hosted the viewport was 3D not 2D. So I don't think this is a real blocker. Original message below.

A negative side-effect I just noticed: In transparent viewports that are setup via a ViewportTexture in a TextureRect and not ViewportController the (supposedly transparent) background looks overly bright.
Somehow I noticed that transparency does not seem to work in either 3.4.3 or this PR this way, but that may be a separate issue or something wrong on my end.

The screenshot shows the PR at the top:

[Calinou]

@Kinwailo Looks good to me, please squash the commits together so we can merge this. See PR workflow for instructions 🙂

[clayjohn]

This looks pretty good and is very promising.

The only issue I noticed while testing is that FXAA doesn't weight the transparent samples by their alpha, so you end up with black edges when using FXAA.

This shouldn't be too hard to fix, it may just be a matter of premultiplying the sampled colour by its transparency so that transparent pixels don't contribute their color.

FXAA on

FXAA off

[Kinwailo]

fixed fxaa with alpha.

[Kinwailo]

fixed a mistake.

[clayjohn]

Just the one change needed. Other than that, this is looking good to go!

[Kinwailo]

Just the one change needed. Other than that, this is looking good to go!

should change the shader for gles3 too?

[clayjohn]

Just the one change needed. Other than that, this is looking good to go!

should change the shader for gles3 too?

Yes please

[Kinwailo]

Just the one change needed. Other than that, this is looking good to go!

should change the shader for gles3 too?

Yes please

changed

[clayjohn]

Looks great! Thank you!

[Calinou]

Thanks! Congratulations for your first merged pull request 🎉