Segfault in ImGui_ImplVulkan_RenderDrawData with pcmd->UserCallback = 0xfffffffffffffffe

[azonenberg]

I'm getting intermittent crashes in the ngscopeclient filter graph editor (using my fork at https://github.com/ngscopeclient/imgui-node-editor although most of the changes are build/warning related and I haven't found any way they could be causing problems).

I'm attempting to develop a reproducible test case but have so far not been successful.

What I do know is that the crash is due to the sentinel value ImDrawCallback_ImCanvas being called - which as far as I know should not ever happen.

Thread 1 "ngscopeclient" received signal SIGSEGV, Segmentation fault.
--Type <RET> for more, q to quit, c to continue without paging--c
0xfffffffffffffffe in ?? ()
(gdb) bt
#0  0xfffffffffffffffe in  ()
#1  0x0000555555c4a548 in ImGui_ImplVulkan_RenderDrawData(ImDrawData*, VkCommandBuffer_T*, VkPipeline_T*) (draw_data=0x612000005c68, command_buffer=0x621000074550, pipeline=0x61c0000188d0)
    at /ceph/fast/home/azonenberg/code/scopehal-apps/src/imgui/backends/imgui_impl_vulkan.cpp:566
#2  0x0000555556b145c1 in VulkanWindow::Render() (this=0x620000009080) at /ceph/fast/home/azonenberg/code/scopehal-apps/src/ngscopeclient/VulkanWindow.cpp:519
#3  0x0000555556869443 in MainWindow::Render() (this=0x620000009080) at /ceph/fast/home/azonenberg/code/scopehal-apps/src/ngscopeclient/MainWindow.cpp:397
#4  0x0000555556bab310 in main(int, char**) (argc=2, argv=0x7fffffffdf78) at /ceph/fast/home/azonenberg/code/scopehal-apps/src/ngscopeclient/main.cpp:127
(gdb) frame 2
#2  0x0000555556b145c1 in VulkanWindow::Render (this=0x620000009080) at /ceph/fast/home/azonenberg/code/scopehal-apps/src/ngscopeclient/VulkanWindow.cpp:519
519                     ImGui_ImplVulkan_RenderDrawData(main_draw_data, *cmdBuf);
(gdb) print cmdBuf
$1 = (vk::raii::CommandBuffer &) @0x603000069a90: {static objectType = vk::ObjectType::eCommandBuffer, static debugReportObjectType = vk::DebugReportObjectTypeEXT::eCommandBuffer, m_device = {
    static objectType = vk::ObjectType::eDevice, static debugReportObjectType = vk::DebugReportObjectTypeEXT::eDevice, m_device = 0x62d000258450}, m_commandPool = {static objectType = vk::ObjectType::eCommandPool, 
    static debugReportObjectType = vk::DebugReportObjectTypeEXT::eCommandPool, m_commandPool = 0x616000036cd0}, m_commandBuffer = {static objectType = vk::ObjectType::eCommandBuffer, 
    static debugReportObjectType = vk::DebugReportObjectTypeEXT::eCommandBuffer, m_commandBuffer = 0x621000074550}, m_dispatcher = 0x6210000b9100}
(gdb) print main_draw_data
$2 = (ImDrawData *) 0x612000005c68
(gdb) frame 1
#1  0x0000555555c4a548 in ImGui_ImplVulkan_RenderDrawData (draw_data=0x612000005c68, command_buffer=0x621000074550, pipeline=0x61c0000188d0) at /ceph/fast/home/azonenberg/code/scopehal-apps/src/imgui/backends/imgui_impl_vulkan.cpp:566
566                         pcmd->UserCallback(cmd_list, pcmd);
(gdb) print pcmd
$3 = (const ImDrawCmd *) 0x61c0000108b8
(gdb) print cmd_list
$4 = (const ImDrawList *) 0x6190001c6460
(gdb) print pcmd
$5 = (const ImDrawCmd *) 0x61c0000108b8
(gdb) print pcmd->UserCallback
$6 = (ImDrawCallback) 0xfffffffffffffffe

[azonenberg]

My nodes are doing custom draw commands in them so perhaps that's related, the sentinel is not being found in the expected state?

(gdb) frame 1
p#1  0x0000555555c4a548 in ImGui_ImplVulkan_RenderDrawData (draw_data=0x612000005c68, command_buffer=0x621000074550, pipeline=0x61c0000188d0) at /ceph/fast/home/azonenberg/code/scopehal-apps/src/imgui/backends/imgui_impl_vulkan.cpp:566
566                         pcmd->UserCallback(cmd_list, pcmd);
(gdb) print cmd_i
$7 = 1
(gdb) print cmd_list->CmdBuffer.Size
$8 = 11

[azonenberg]

I now have a semi reproducible test case, at least if you're able to build and run ngscopeclient.

I added the following code to imgui_canvas.cpp after the check for the sentinel:

        //DEBUG: Search the *entire* draw list for the sentinel command
        for(int i=0; i<m_DrawList->CmdBuffer.size(); i++)
        {
            if(m_DrawList->CmdBuffer[i].UserCallback == ImDrawCallback_ImCanvas)
            {
                fprintf(stderr, "found and removed sentinel at offset %d, expected %d or %d\n",
                    i,
                    m_DrawListCommadBufferSize,
                    m_DrawListCommadBufferSize-1);

                m_DrawList->CmdBuffer.erase(m_DrawList->CmdBuffer.Data + i);
            }
        }

The fault occurs while mousing over the Divide_1 node in the graph editor, at which point I get

found and removed sentinel at offset 1, expected 3 or 2

For whatever reason, it appears critical that the right hand "data" output port is partially clipped by the edge of the canvas. If the entire port text is on or off screen, the fault does not occur.

[azonenberg]

In general, the fault is triggered by mousing over any filter node that has output ports which are partially clipped by the edge of the canvas. Instrument channels do not appear to trigger it regardless of clip position.

[thedmd]

@azonenberg

ImDrawCallback_ImCanvas is a sentinel value and should not be left in draw command after canvas is done. ImGui internal plumbing changed and that exposed some issues.

There were fixes around that issue some time ago. Can you check if master or develop does work for you?

[pthom]

Hello,

This is probably related to #282
Could you test if the PR #285 fixes the issue?