Re-render only on user interaction

[Person-93]

(you may also go to Demo>About Window, and click "Config/Build Information" to obtain a bunch of detailed information that you can paste here)

Version/Branch of Dear ImGui:

Version: 1.66
Branch: master

Back-end/Renderer/Compiler/OS

Back-ends: opengl3
Compiler: gcc, mingw, clang
Operating System: linux/windows
My Issue/Question:
Is there a way to tell if the user has interacted with the window since the last rendering?

I want to skip rendering most of the window if the user hasn't interacted. (i.e. clicked, typed, moved mouse).

Also, this might not be directly related to this library, but is there a way to tell if the window is completely covered by other windows? In that case, I would skip rendering entirely.

[ocornut]

Is there a way to tell if the user has interacted with the window since the last rendering?
I want to skip rendering most of the window if the user hasn't interacted. (i.e. clicked, typed, moved mouse).

Effectively you want to check for "clicked,typed,moved mouse" and you have this information already because your back-end is passing it to dear imgui and you are looking into using a custom back-end if you want to go idle. You need to setup a counter, and reset it to 2 or 3 whenever an input came (so a single button click will "wake up" rendering for 2 or 3 frames).

Also see #71 and #1206.

Also, this might not be directly related to this library, but is there a way to tell if the window is completely covered by other windows? In that case, I would skip rendering entirely.

This is a different question worthy of a different topic.
It's not possible currently and it's not something I put much energy on because the default styles have transparent window background. You may also want to measure the cost that you are trying to avoid.

[ocornut]

Addendum: Your subject line says "re-render only on user interaction" but the point of the technique highlighted is in my first paragraph is mainly to avoid running the ImGui:: functions and touching your source data. The actual rendering cost is generally lesser than the building of it.

[djdeath]

For a Gtk+/OpenGL backend I have a redraw on user interaction heuristic :
Render 2 frames on input events 1 and on text entry focused, redraw at the blinking speed of the cursor 2
For interactive graphs, I added a schedule frame method (a bit like requestAnimationFrame in Javascript) 3

[jpcy]

You can skip rendering a frame if the contents of ImDrawData hasn't changed since the last rendered frame.

[ocornut]

Thank you @djdeath for those links. I would like to eventually make it a more obvious first-party citizen.
We should probably expose various counters and timers carrying specific semantic (e.g. cursor blink timer which the user may choose to use or ignore or adapt).

For the blinking cursor I would like to implement this idea where by storing a persisting reference to the vertices used by the cursor, we could have a fast pass patching the alpha of the 4 vertices. So some ImGui:: function could be independently handling cursor patching, and when the timer elapse we only do patching + request GPU refresh.

[tomasiser]

Just wanted to put a comment here as I had to deal with this problem last year.

Also, this might not be directly related to this library, but is there a way to tell if the window is completely covered by other windows? In that case, I would skip rendering entirely.

According to my previous research, this is incredibly difficult to do on Windows, or at least I did not find a 100% working solution*. If someone knows, please let us know! Now I have no idea about other platforms, but let me explain how I managed to at least partially solve this problem on Windows.

So first of all, in Windows API, there is an IsIconic function available (provided you know the HWND of your main window, which should be easy to get from most rendering backends). The tricky part of this function is that it's almost useless as it only returns true if the window is actually minimized using the minimize button in the top right corner. It does not even return true when you press Win+D. So this is not going to help us.

Then there is an IsWindowVisible, but don't get misled by its name. It has nothing to do with your window being covered by other windows. It only returns whether the show/hide flag of the window is set to visible. So again, nothing which can help us.

Then we have GetForegroundWindow, GetActiveWindow, and GetTopWindow. These are also useless because you probably want to render animated stuff even if your window is not on top of everything else, but still visible on the screen. So again, nothing which can help us.

Finally, I decided to implement a solution based on manually finding windows from coordinates. The idea is fairly "simple". You just need to choose a few coordinates on your window and ask Windows which HWND is visible at that coordinates. In my implementation, I decided to use 3 points: the top-left corner, the window center, and the bottom-right corner. If all these 3 points are obscured by another window, I conclude that I do not need to render anything as my window is probably not visible at all.

Note that this solution also works if the window is "behind" desktop (i.e., when you press Win+D).

Example code copy pasted from my implementation:

// do not forget this:
#include "windows.h"

bool is_window_covered(HWND wnd) {
    if(IsIconic(wnd)) {
        return true;  // early return, window is minimized (iconic)
    }

    RECT windowRect;
    if(GetWindowRect(wnd, &windowRect)) {
        // check if window is obscured by another window at 3 diagonal points (top left, center, bottom right):
        bool isObscuredAtDiagonal = true;
        POINT checkpoint;
        // check window top left:
        checkpoint.x = windowRect.left;
        checkpoint.y = windowRect.top;
        auto wndAtCheckpoint = WindowFromPoint(checkpoint);
        isObscuredAtDiagonal &= (wndAtCheckpoint != wnd);
        // check window center:
        checkpoint.x = windowRect.left + (windowRect.right - windowRect.left) / 2;
        checkpoint.y = windowRect.top + (windowRect.bottom - windowRect.top) / 2;
        wndAtCheckpoint = WindowFromPoint(checkpoint);
        isObscuredAtDiagonal &= (wndAtCheckpoint != wnd);
        // check window bottom right:
        checkpoint.x = windowRect.right - 1;
        checkpoint.y = windowRect.bottom - 1;
        wndAtCheckpoint = WindowFromPoint(checkpoint);
        isObscuredAtDiagonal &= (wndAtCheckpoint != wnd);
        if(isObscuredAtDiagonal) {
            return true;
        }
    }

    return false;
}

---

* Note that there is another solution explained at the bottom of this page. It is based on clipping. You may want to check it out. I have no idea if it works as I have not tested that one.

[malamanteau]

In my current prototype I hash the current drawlist, and compare this against the previous hash. If the hash is the same I don't redraw. I force a full redraw every second to make sure that it get redrawn in certain edge cases (waking up from stand-by etc). I keep a separate hash for every ImGuiViewport.
XXHash64 works really well!

I also enable and disable vsync based on input events to reduce latency.

I limit the framerate to four times the monitor refresh rate (when vsync is off), and with all these changes CPU usage is very low, and only spikes a bit during heavy interaction.

static void ImGui_ImplDX11_RenderWindow(ImGuiViewport* viewport, void*)
{
	uint64 lastHValue = 0;
	if (viewport->UserData)
	{
		lastHValue = ((XXHash64*)viewport->UserData)->hash();

		delete (XXHash64*)viewport->UserData;
		viewport->UserData = nullptr;
	}

	viewport->UserData = new XXHash64(1105121757519812621);
	XXHash64 * h = (XXHash64 *)viewport->UserData;

	ImGuiIO & io = ImGui::GetIO();

	const float width_points  = io.DisplaySize.x;
	const float height_points = io.DisplaySize.y;

	ImDrawData * draw_data = viewport->DrawData;

	const int  width_pixels = (int)(draw_data->DisplaySize.x * io.DisplayFramebufferScale.x);
	const int height_pixels = (int)(draw_data->DisplaySize.y * io.DisplayFramebufferScale.y);

	h->add(&width_pixels,               4_u64);
	h->add(&height_pixels,              4_u64);
	h->add(&(draw_data->CmdListsCount), 4_u64);

	for (int i = 0; i < draw_data->CmdListsCount; i++)
	{
		h->add(&(draw_data->CmdLists[i]->VtxBuffer[0]), draw_data->CmdLists[i]->VtxBuffer.size() * sizeof(ImDrawVert));
		h->add(&(draw_data->CmdLists[i]->IdxBuffer[0]), draw_data->CmdLists[i]->IdxBuffer.size() * sizeof(ImDrawIdx));
		h->add(&(draw_data->CmdLists[i]->CmdBuffer[0]), draw_data->CmdLists[i]->CmdBuffer.size() * sizeof(ImDrawCmd));
	}
	
	uint64   nuHValue = h->hash();

	if (lastHValue == nuHValue)
	{
		viewport->NeedSwap = false;
		return;
	}
	else 
		viewport->NeedSwap = true;
....

and also:

static void ImGui_ImplDX11_SwapBuffers(ImGuiViewport* viewport, void*)
{
	if (!viewport->NeedSwap)
		return;
....

I had to add two items to ImGuiViewport to keep track of these things:

	bool                NeedSwap = false;
	void*               UserData = nullptr; 

Here is my function for detecting user input (I use this to temporarily disable vsync, to make the app more responsive):

inline bool AutoShouldVSync()
{
	ImGuiIO & io = ImGui::GetIO();

	static bool mouseDown0Prev = io.MouseDown[0];
	static bool mouseDown1Prev = io.MouseDown[1];
	static bool mouseDown2Prev = io.MouseDown[2];
	static bool mouseDown3Prev = io.MouseDown[3];
	static bool mouseDown4Prev = io.MouseDown[4];
	static Math::Vector2 mousePosPrev = io.MousePos;

	bool mouseDownChanged =
		(io.MouseDown[0] != mouseDown0Prev) ||
		(io.MouseDown[1] != mouseDown1Prev) ||
		(io.MouseDown[2] != mouseDown2Prev) ||
		(io.MouseDown[3] != mouseDown3Prev) ||
		(io.MouseDown[4] != mouseDown4Prev);
	bool mouseMoved = io.MousePos != mousePosPrev;

	bool mouseIsDown =
		io.MouseDown[0] ||
		io.MouseDown[1] ||
		io.MouseDown[2] ||
		io.MouseDown[3] ||
		io.MouseDown[4];

	mousePosPrev   = io.MousePos;
	mouseDown0Prev = io.MouseDown[0];
	mouseDown1Prev = io.MouseDown[1];
	mouseDown2Prev = io.MouseDown[2];
	mouseDown3Prev = io.MouseDown[3];
	mouseDown4Prev = io.MouseDown[4];

	bool mouseWheelChanged = io.MouseWheel != 0.0f || io.MouseWheelH != 0.0f;

	return !(mouseIsDown || mouseDownChanged || mouseMoved || mouseWheelChanged);
}

@ocornut is this a functionality you might want to merge into the official project?

[Lectem]

I think this is a very interesting topic, especially when deploying imgui to the web !
Maybe a hash could be computed directly by imgui each time a new draw call is added, and then expose this hash ?

[ocornut]

I think this is a very interesting topic, especially when deploying imgui to the web !
Maybe a hash could be computed directly by imgui each time a new draw call is added, and then expose this hash ?

It would be too costly for imgui to do this by default, so best left to the user.
We could easily optimize the best case (e.g. first hash and compare ImDrawCmd data only), but we have to deal with the worst case which is that nothing changed and would need hashing of the whole vertex data to confirm it.

An alternative would be to allow variable-frequency refresh of individual windows (e.g. non-focused windows could be configured to refresh at lower frequency), and the ImDrawList can carry a timestamp.

Note that all of this is different from the original post which was to render on user interaction.

[TTimo]

@malamanteau would you consider maintaining a branch/PR of your changes as a reference for this?

[malamanteau]

@malamanteau would you consider maintaining a branch/PR of your changes as a reference for this?

https://github.com/malamanteau/imgui/tree/docking

[TTimo]

Thank you! Constant/high GPU usage is the top negative feedback I receive about my decision to use imgui for our cross platform UI - hopefully some progress can be made in that area.

[malamanteau]

Thank you! Constant/high GPU usage is the top negative feedback I receive about my decision to use imgui for our cross platform UI - hopefully some progress can be made in that area.

Awesome, yeah sorry it took me so long to see your request there. Note that the frame hashing is only there for DX** and SW rendering, though I suspect GL would be the same.

The time rounded to the second is now included in the hash, so an integrity redraw is forced every a second.

Also, the hash won't catch things like a texture changing in vram, if the textureid changes though that will be enough to change the hash

[bubnikv]

We at Prusa Research are developing an open source FDM 3D printing slicer PrusaSlicer.
http://github.com/prusa3d/PrusaSlicer

Our software is C++ only, using wxWidgets for windowing and ImGUI for dialogs, panels, notifications and tooltips in the 3D scene. ImGUI serves us well, thanks to @ocornut and all the contributors.

Hundred thousands of 3D printing enthusiasts are using our application around the world on Windows, OSX and Linux on hardware ranging from cheapest laptops to powerful desktops and CAD workstations. On some laptops the OpenGL frame rate may drop quite low, thus we really want to be conservative with the screen updates. Also customers will not be happy about draining their laptop batteris with a continuous high frame rendering loop, which may lead to throttling the CPU/GPU SoC due to overheating, hurting performance of our CPU intensive slicing algorithms.

We have succeeded to trigger an OpenGL screen refresh on demand. We have found out that at least for the bits and pieces of ImGUI we are using, it is sufficient to only perform one additional screen refresh after some ImGUI window is resized. As noted already, we only use the ImGUI windows and tooltips, we don't use menus. Maybe menus and other ImGUI components require more additional screen refreshes. We don't use a blinking cursor.

Now comes my question: We would like to limit the screen refresh triggered by ImGUI update only to a viewport rectangle that needs to be updated. Namely, if one types in an edit field in one particular ImGUI Window, only the edit field needs to be refreshed, in worst case the particular ImGUI window hosting the edit field needs to be refreshed on the screen. If we update a tooltip, only a bounding box around the old and new tooltip needs to be refreshed. Having an update rectangle provided by ImGUI would allow us to set up an OpenGL viewport, which could clip many of the ImGUI generated triangles by the vertex shader before they are pixelated and textured, meaning a lot of GPU clocks and CO2 emissions conserved. The clipping would not only help to limit the GPU resources needed to render the ImGUI generated triangles. As noted, many of our customers run PrusaSlicer on low end laptops. Our application potentially renders complex scenes and we may go to the lengths of only rendering those 3D objects, which bounding geometry intersects with an update rectangle in the screen space. Thus our application would be more responsive when manipulating ImGUI elements if we would not have to render 3D models not overlapping with them.

I understand the ImGUI has been designed with simplicity in mind. However with a large user base, reasonable amount of funding and low end computers in mind, we may find it worthwile to handle the screen updates with finer granularity manually or by building our components around ImGUI Windows. Our application is aware of the values it modifies to be presented to the user so it can act accordingly. It is however difficult for us to react to UI actions piped to ImGUI.

As a minimum implementation, we would need to track ImGUI window opening / closing / moving (that accounts for tooltips as well I suppose) and ImGUI window focus. The update rectangle calculated by ImGUI that we need to handle screen refreshes based on user UI input could consist of a sum of old/new positions of moved / opened / closed windows + the bounding box of a window with keyboard / mouse focus.

Is there a way to calculate the update rectangle somehow? Namely, there is no API for accessing window state (position, attributes, state of the focus) from outside the Begin() / End() block, which I suppose has to be called from the rendering loop. The only API functions that accept the window name are the SetWindowPos()/SetWindowSize()/SetWindowCollapsed()/SetWindowFocus() functions, but there are no accessors.

What would you recommend? We are happy to hack ImGUI for our purpose.

Thanks,
Vojtech

[watbulb]

I just wanted to mention, if you are using SDL to manage your application you could do something like so to wait for an event that doesn't require immediate interaction, since SDL is also aware of changes to IO devices, as well as window/frame focus. This way the render thread (if implemented) can be paused during times of no interaction that would otherwise produce changes in draw calls.

Consider the following example:

    static bool done = false;
    static bool movement = false;
    SDL_Event event;
    while (!done && SDL_WaitEvent(&event))
    {
        // Pass to ImGui
        movement = ImGui_ImplSDL2_ProcessEvent(&event);

        // Handle Event Type
        switch (event.type)
        {
            case SDL_QUIT:
                done = true;
                break;
            case SDL_WINDOWEVENT:
                if (event.window.windowID != SDL_GetWindowID(window))
                    break;
                if (event.window.event == SDL_WINDOWEVENT_CLOSE) {
                    done = true;
                    break;
                }
                if (event.window.event == SDL_WINDOWEVENT_RESIZED) {
                    // Release all outstanding references to the swap chain's buffers before resizing.
                    CleanupRenderTarget();
                    g_pSwapChain->ResizeBuffers(0, 0, 0, DXGI_FORMAT_UNKNOWN, 0);
                    CreateRenderTarget();
                }
                break;
        }  // switch

        ...

        ImGui_ImplDX11_NewFrame();
        ImGui_ImplSDL2_NewFrame();
        ImGui::NewFrame();
        
        ...
        
        g_pSwapChain->Present(movement, 0);

By coupling SDL_WaitEvent with the render main event loop, this will cause the loop to pause when no interaction is detected on SDL's end, as well as the call to ImGui_ImplSDL2_ProcessEvent. Ideally this should be implemented in a separate render thread. I should also mention SDL_WaitEvent uses an efficient notify/await scheme depending on the platform you are on, which should result in a non busy-wait behaviour (few cycles).

Another option is to continue to use SDL_PollEvent in the main loop, and ask imgui via ImGui_ImplSDL2_ProcessEvent if movement is present, if not, in a default switch-case label the loop can be continued to wait for another event that is not movement related:

while (!done) {
    SDL_Event event;
    while (SDL_PollEvent(&event)) {
        bool movement = ImGui_ImplSDL2_ProcessEvent(&event);
        // decide what to do with this event.
        switch(event.type) {
            ...
            default:
                if(!movement)
                    continue;
                else break;
         }
         
        ...

        ImGui_ImplDX11_NewFrame();
        ImGui_ImplSDL2_NewFrame();
        ImGui::NewFrame();
        
        ...
        
        g_pSwapChain->Present(movement, 0);

Perhaps this is not an ideal solution for everyone and I recognize that, but it worked well enough for my use-cases, so maybe it will help a fellow traveller.

<3 thanks for the awesome library @ocornut

[rokups]

with SDL_WaitEvent(&event) blinking input cursor or plots will not render unless you keep poking window.

[watbulb]

Well, considering the question is "Re-render only on user interaction" I don't see much of an issue with that. Like I said, put it in a different thread, then using SDL notify the render thread that an animation needs processing, spinning the loop up. Wasting a frame on a blinking cursor seems like a design choice and not a necessity.

[jgarvin]

@rokups SDL_WaitEventTimeout(&event, timeout_msec); is sufficient to allow things like blinking cursors and animations while still dramatically reducing CPU usage. You can also adjust the time out based on how long the previous iteration took to get the effect of a constant refresh rate.

[etkramer]

Also wanted to suggest the possibility of continuing to draw every frame, but only regenerating the geometry when needed. I believe this is the route Unity's newer UI system takes, as it has to generate some particularly complex shapes.