How to handle window and surface with different dimensions?

[Lahvuun]

Version/Branch of Dear ImGui:

Version: 313676d
Branch: master

Back-end/Renderer/Compiler/OS

Back-ends: imgui_impl_win32.cpp + imgui_impl_dx9.cpp
Compiler: i686-w64-mingw32-g++
Operating System: Windows and Linux (Wine)

My Issue/Question:

I'm using Dear ImGui to make a UI with which to control discovered leftover/debug functionality in a proprietary video game that I have no source code for. It's compiled to a DLL, which the game then loads. From DllMain I hook the game's EndScene, which is then used to render everything.

It's working well, but the game can dynamically change the dimensions of the surface it's rendering to (the window resolution never changes). If the graphics quality are set to highest, the game renders everything after the intro at 2x width and height of the resolution selected in the settings. Since this depends on the graphics quality, my guess is that this is done for anti-aliasing. Unfortunately it also screws with the rendering and input of the UI. I attached a video showing what the problem looks like.

I found a few issues discussing similar problems:

• Rendering multiple times per frame, at different resolutions. #566
• Rendering ImGui content to surface 2x window dimensions #3811
• Inputs gets messed up with bigger supported window resolutions (1920x1080, 2560x1440, ...) #5750

There I didn't see a definitive answer for how to handle such situations. PathogenDavid suggested writing a custom backend, but I'm wondering if there is a simpler way.

Thank you in advance!

Screenshots/Video

2x.mp4

[GamingMinds-DanielC]

Easiest way would be to copy ImGui_ImplDX9_SetupRenderState() and adjust it as needed. You need to detect the scaling, probably by retrieving the render target and checking its resolution. Then adjust the viewport accordingly, the projection matrix should be fine since it stretches over the viewport. You can use ImDrawList::AddCallback() at the beginning of your frame to patch in your custom setup.

If scissor rects are a problem as well (they most likely are), you can iterate through the draw data before rendering (take a look at ImGui_ImplDX9_RenderDrawData() for reference) and scale the values in those rectangles.

Or you can indeed write your custom backend. It's not as hard as it may sound. In this case it would be a copy of the DX9 backend with adjustments in the mentioned functions.

[PathogenDavid]

Seconding Daniel. My wording is maybe a bit off in that old comment, I think what I actually meant was customizing an existing backends rather than writing one from scratch. (In your case it also appears that your inputs are fine it's just the visuals that are too small, which is much simpler to tweak.)

[Lahvuun]

Seconding Daniel. My wording is maybe a bit off in that old comment, I think what I actually meant was customizing an existing backends rather than writing one from scratch. (In your case it also appears that your inputs are fine it's just the visuals that are too small, which is much simpler to tweak.)

I thought I'd post here once I figured everything out, but since you mentioned inputs, they're not fine, actually. Dear ImGui behaves as if the window that I'm trying to interact with was still at its original position, even though it's now displayed at a different one. Here's another video where I tried to show it off better:

input.mp4

I also showed the dimensions of the display size and render target this time. You can see the render target's changing to 4k right once the game goes into a loading screen. Here's how I queried it:

		IDirect3DSurface9 *render_target = nullptr;
		HRESULT result = render_manager->d3d_device->GetRenderTarget(0, &render_target);
		if (!result) {
			D3DSURFACE_DESC desc = {};
			result = render_target->GetDesc(&desc);
			if (!result) {
				ImGui::Text("Current io.DisplaySize.x is %f", io.DisplaySize.x);
				ImGui::Text("Current io.DisplaySize.y is %f", io.DisplaySize.y);
				ImGui::Text("Current width of render target 0 in D3DSURFACE_DESC is %d", desc.Width);
				ImGui::Text("Current height of render target 0 in D3DSURFACE_DESC is %d", desc.Height);
			}
		}

An idea I got is simply doubling the mouse cursor position:

		LPARAM newpos = lParam;
		if (message == WM_MOUSEMOVE) {
			newpos = ((GET_Y_LPARAM(lParam) << 16) * 2) | (GET_X_LPARAM(lParam) * 2);
		}
		ImGui_ImplWin32_WndProcHandler(hwnd, message, wParam, newpos);

Surprisingly, it works, but this could potentially cause some problems, like if Dear ImGui tries to set the cursor's position based on this value, for example if ImGuiConfigFlags_NavEnableSetMousePos is used. I was thinking that maybe I could divide all the window's positions and dimensions by 2 somehow before inputs are handled? Or maybe there already is a way to make the library do exactly that, that I haven't found?

---

In the meantime I managed to solve the issues with the scissor rects thanks to pointers from Daniel. Once I knew where to look, I even found out that there's specifically a function for this very case, ScaleClipRects. But I tried using it and the result was not what I expected, with window contents (but not the windows themselves) cutting off. As I was trying to get input to work, I found out that setting io.DisplaySize to ImVec2(3840, 2160) right after ImGui_ImplWin32_NewFrame solves the issues with the clipping entirely! So that's one less problem for now.

[Lahvuun]

OK, I figured it all out.

Essentially, what happens under the win32 backend when you have a surface and window with different dimensions is that the backend ignores that (to be fair, it's not like it can be aware of that without calling into dx9) and tells Dear ImGui to work in the context of the window. So, when Render is called, draw data is generated with, say, a vertex at (5, 5), and it is then faithfully placed at (5, 5) in the surface coordinates, which are not (5, 5), but (2.5, 2.5) in window coordinates in case the surface has 2x the window dimensions. This causes a discrepancy between where Dear ImGui thinks its widgets are located, and where they actually are visible in the window. That's why the inputs are off, that's why widgets are confined to the top-left quarter of the window, and that's also why they are clipped when approaching the edges of the quarter.

I see two possible solutions to this:

• Tell Dear ImGui something along the lines of "no, actually please work in terms of 3840x2160 rather than 1920x1080". My understanding is that this is accomplished by setting io.DisplaySize after ImGui_ImplWin32_NewFrame. Unfortunately, this introduces one problem: the mouse movement events are still transmitted in window coordinates, so they need to be adjusted when passing into the library's domain, or going outside of them. And while it is certainly possible to update the incoming values before calling into ImGui_ImplWin32_WndProcHandler, values that are leaving Dear ImGui as part of ImGui_ImplWin32_NewFrame cannot be changed without also changing the backend.
• Manipulate the draw data itself. Update the position of the vertices, set the viewport correctly, and update the clip rects. This can be done without changing the backend:

		ImDrawData *draw_data = ImGui::GetDrawData();
		if (draw_data) {
			for (auto &list : draw_data->CmdLists) {
				for (auto &vert : list->VtxBuffer) {
					vert.pos.x *= 2;
					vert.pos.y *= 2;
				}
			}
			draw_data->DisplaySize.x = 3840;
			draw_data->DisplaySize.y = 2160;
			draw_data->ScaleClipRects(ImVec2(2, 2));
		}
		ImGui_ImplDX9_RenderDrawData(draw_data);

And while this approach is likely worse in terms of performance, I think I prefer it anyway.

While searching through some more issues I found the following comment by ocornut #6714 (comment):

Various other reasons (easier/standardized DPI framebuffer scale support, remapping of absolute multi-viewports coordinates to be positive) are likely to lead us toward backend supporting some form of inputs coordinate transforms, so it's not impossible we would support this out of the box in the future.

Meaning the first approach could potentially become supported at some point. Exciting!

[GamingMinds-DanielC]

I thought I'd post here once I figured everything out, but since you mentioned inputs, they're not fine, actually. Dear ImGui behaves as if the window that I'm trying to interact with was still at its original position, even though it's now displayed at a different one. Here's another video where I tried to show it off better:

Inputs only don't look fine as long as the display is scaled incorrectly. Once you fix your render scaling, the inputs should match up again after modification. Inputs should behave as if the window was still at its original position, you just need to scale rendering so that it is displayed at its original position as well.

If you do it the other way around and adjust your inputs to your non-scaled rendering, they will match up as well. But you will have changing sizes all over and in high resolution rendering your ImGui windows and widgets will be very small and hard to read.

[Lahvuun]

Inputs only don't look fine as long as the display is scaled incorrectly. Once you fix your render scaling, the inputs should match up again after modification. Inputs should behave as if the window was still at its original position, you just need to scale rendering so that it is displayed at its original position as well.

I (eventually) arrived at the same conclusion. The only downside to upscaling the rendering is that it looks less crisp:

upscale.mp4

In this video the render target's dimensions change at around the 2 second mark, and you can notice how the UI quality degrades slightly. I think the problem could be dealt with by using io.DisplaySize.x = 3840; io.DisplaySize.y = 2160, but then you'll have other issues with

changing sizes all over and in high resolution rendering your ImGui windows and widgets will be very small and hard to read

Although I think the size and reading experience can be improved by setting the font scale like
io.Fonts->Fonts[0]->Scale = 2
But me and my potential users will likely be fine without this.

[GamingMinds-DanielC]

I (eventually) arrived at the same conclusion. The only downside to upscaling the rendering is that it looks less crisp:

You could do an extra step. Create your own render target texture in the original window resolution and render into that one (after a clear). Then you won't need to scale anything. But you would need to adjust your blending setup with separate blending for the alpha channel. Then you could render the resulting texture as a fullscreen quad with point sampling and a fitting blending setup, that should preserve the crispness.