Use DwmEnableBlurBehindWindow for Windows Per Pixel Transparency

[Technohacker]

The current Windows code (from #14622) uses WS_EX_LAYERED which affects the non-client area (preventing bordered transparent windows) and is CPU intensive as the image is copied from the GPU to main memory for the alpha premultiply

This PR changes the method used to DwmEnableBlurBehindWindow which uses DWM from Windows to make the window background transparent, allowing the glClearColor background alpha to work without any need for the CPU to apply the alpha manually

This PR also removes the force enabled borderless window mode from set_window_per_pixel_transparency_enabled hence allowing bordered transparent windows across Windows, macOS and *nix

This change requires more testing to assess any performance impacts (such as #35628, as this change could remove the CPU bottleneck). The code was tested on Linux (transparency with and without borderless) and partially on Windows (transparency with borders)

[bruvzg]

Nice, this works great, I have never realized DwmEnableBlurBehindWindow is working this way.

The only minor issue, setting blur rect to (0, 0, 1, 1) is causing single detached pixel visible in the top left corner of the borderless window, but setting it to (0, 0, -1, -1) works perfectly.

[bruvzg]

This change requires more testing to assess any performance impacts (such as #35628, as this change could remove the CPU bottleneck). The code was tested on Linux (transparency with and without borderless) and partially on Windows (transparency with borders)

It should fix most of the performance penalty indeed.

[bruvzg]

This PR also removes the force enabled borderless window mode from set_window_per_pixel_transparency_enabled hence allowing bordered transparent windows across Windows, macOS and *nix

These restrictions were added to mimic WS_EX_LAYERED behavior and removing it is fine, both macOS and Linux should work with native borders.

[Technohacker]

I assume this PR wouldn't require more changes so I'll make it ready for review :)

[akien-mga]

I like PRs with this kind of ratio :)

Normally PRs should be made against the master branch, and later cherry-picked to stable branches like 3.2, but in this case I believe Per Pixel Transparency might be broken in the master branch after the Vulkan and DisplayServer refactor, so I guess that's good to merge against 3.2 as is.

@bruvzg @Technohacker Could you evaluate what parts of this PR are relevant on master nevertheless and make a PR there? Even if the feature doesn't work, we can still make the matching code changes for later when the feature gets reimplemented (provided that the old code is still there).

[akien-mga]

Thanks!

[bruvzg]

Could you evaluate what parts of this PR are relevant on master

Last time I tried to set Metal layer transparency on macOS it was not working at all, and I have not investigated what's going on Linux at all. Most of the removed code it this PR was already removed during DisplayServer/OS split (macOS still have forced borderless on setting transparency flag).

diff --git a/platform/osx/display_server_osx.mm b/platform/osx/display_server_osx.mm
index 71e4584dac..d2c9654622 100644
--- a/platform/osx/display_server_osx.mm
+++ b/platform/osx/display_server_osx.mm
@@ -2644,8 +2642,6 @@ void DisplayServerOSX::window_set_flag(WindowFlags p_flag, bool p_enabled, Windo
 			if (p_enabled) {
 				[wd.window_object setStyleMask:NSWindowStyleMaskBorderless];
 			} else {
-				if (wd.layered_window)
-					_set_window_per_pixel_transparency_enabled(false, p_window);
 				[wd.window_object setStyleMask:NSWindowStyleMaskTitled | NSWindowStyleMaskClosable | NSWindowStyleMaskMiniaturizable | (wd.resize_disabled ? 0 : NSWindowStyleMaskResizable)];
 				// Force update of the window styles
 				NSRect frameRect = [wd.window_object frame];
@@ -2665,11 +2661,6 @@ void DisplayServerOSX::window_set_flag(WindowFlags p_flag, bool p_enabled, Windo
 		} break;
 		case WINDOW_FLAG_TRANSPARENT: {
 			wd.layered_window = p_enabled;
-			if (p_enabled) {
-				[wd.window_object setStyleMask:NSWindowStyleMaskBorderless]; // force borderless
-			} else if (!wd.borderless) {
-				[wd.window_object setStyleMask:NSWindowStyleMaskTitled | NSWindowStyleMaskClosable | NSWindowStyleMaskMiniaturizable | (wd.resize_disabled ? 0 : NSWindowStyleMaskResizable)];
-			}
 			_set_window_per_pixel_transparency_enabled(p_enabled, p_window);
 
 		} break;

Something like this will give transparency on Windows, but colors are off, probably due wrong alpha pre-multiplication:

diff --git a/platform/windows/display_server_windows.cpp b/platform/windows/display_server_windows.cpp
index e794efb4fb..cc8eecb8a6 100644
--- a/platform/windows/display_server_windows.cpp
+++ b/platform/windows/display_server_windows.cpp
@@ -35,6 +35,7 @@
 #include "scene/resources/texture.h"
 
 #include <avrt.h>
+#include <dwmapi.h>
 
 #ifdef DEBUG_ENABLED
 static String format_error_message(DWORD id) {
@@ -1050,8 +1051,22 @@ void DisplayServerWindows::window_set_flag(WindowFlags p_flag, bool p_enabled, W
 			_update_window_style(p_window);
 		} break;
 		case WINDOW_FLAG_TRANSPARENT: {
-
-			// FIXME: Implement.
+			wd.layered_window = p_enabled;
+			if (p_enabled) {
+				DWM_BLURBEHIND bb = { 0 };
+				HRGN hRgn = CreateRectRgn(0, 0, -1, -1);
+				bb.dwFlags = DWM_BB_ENABLE | DWM_BB_BLURREGION;
+				bb.hRgnBlur = hRgn;
+				bb.fEnable = TRUE;
+				DwmEnableBlurBehindWindow(wd.hWnd, &bb);
+			} else {
+				DWM_BLURBEHIND bb = { 0 };
+				HRGN hRgn = CreateRectRgn(0, 0, -1, -1);
+				bb.dwFlags = DWM_BB_ENABLE | DWM_BB_BLURREGION;
+				bb.hRgnBlur = hRgn;
+				bb.fEnable = FALSE;
+				DwmEnableBlurBehindWindow(wd.hWnd, &bb);
+			}
 		} break;
 		case WINDOW_FLAG_NO_FOCUS: {
 
@@ -1084,7 +1099,7 @@ bool DisplayServerWindows::window_get_flag(WindowFlags p_flag, WindowID p_window
 		} break;
 		case WINDOW_FLAG_TRANSPARENT: {
 
-			// FIXME: Implement.
+			return wd.layered_window;
 		} break;
 		case WINDOW_FLAG_NO_FOCUS: {
 
@@ -2542,29 +2557,7 @@ LRESULT DisplayServerWindows::WndProc(HWND hWnd, UINT uMsg, WPARAM wParam, LPARA
 				windows[window_id].maximized = false;
 				windows[window_id].minimized = false;
 			}
-#if 0
-			if (is_layered_allowed() && layered_window) {
-				DeleteObject(hBitmap);
-
-				RECT r;
-				GetWindowRect(hWnd, &r);
-				dib_size = Size2i(r.right - r.left, r.bottom - r.top);
-
-				BITMAPINFO bmi;
-				ZeroMemory(&bmi, sizeof(BITMAPINFO));
-				bmi.bmiHeader.biSize = sizeof(BITMAPINFOHEADER);
-				bmi.bmiHeader.biWidth = dib_size.x;
-				bmi.bmiHeader.biHeight = dib_size.y;
-				bmi.bmiHeader.biPlanes = 1;
-				bmi.bmiHeader.biBitCount = 32;
-				bmi.bmiHeader.biCompression = BI_RGB;
-				bmi.bmiHeader.biSizeImage = dib_size.x * dib_size.y * 4;
-				hBitmap = CreateDIBSection(hDC_dib, &bmi, DIB_RGB_COLORS, (void **)&dib_data, nullptr, 0x0);
-				SelectObject(hDC_dib, hBitmap);
-
-				ZeroMemory(dib_data, dib_size.x * dib_size.y * 4);
-			}
-#endif
+
 			//return 0;								// Jump Back
 		} break;

[Technohacker]

@bruvzg Sorry for the ping but, could I know the current state of per pixel transparency in current master? I'd love to work on getting it restored for the 4.0 release :)

[bruvzg]

could I know the current state of per pixel transparency in current master? I'd love to work on getting it restored for the 4.0 release :)

It's not implemented on any platform. I have no idea how to implement it with Vulkan, or if it's possible at all (probably it will require composite alpha support on VkSurface (https://khronos.org/registry/vulkan/specs/1.2-extensions/man/html/VkCompositeAlphaFlagBitsKHR.html), and I do not think it is supported by any Vulkan implementation).

[Technohacker]

Ah bummer :/
I'm a Vulkan noob, but is there no support for framebuffers with alpha channels? EDIT: My bad, just noticed that VkSurface is what's displayed