Debugging: Unable to halt execution using arm gdb

[weinand]

From @am1ko on April 26, 2017 19:21

• VSCode Version: 1.11.2 (1.11.2)
• OS Version: MacOS Sierra 10.12.3

Steps to Reproduce:

1. Setup launch.json to use arm-none-eabi-gdb for debugging a c program on an embedded target (see attached launch.txt)
2. Set one breakpoint to beginning of main
3. Start debug session
4. Breakpoint in main fires. Press "play" button to continue
5. Try to halt the target by pressing "pause"

The result is that nothing happens. Any attempt to use the debug console results is
"Unable to perform this action because the process is running."

The way to halt the execution when using command line arm-none-eabi-gdb is to press ctrl-c (SIGINT?). Could this option be added to vscode debugger config?

launch.txt

Copied from original issue: microsoft/vscode#25467

[weinand]

From @timurey on June 18, 2017 18:30

Have the same issue.

[pieandcakes]

@am1ko @timurey If you set "stopAtEntry" to true does it stop? Did you compile with the -g flag?

[am1ko]

I just tried "stopAtEntry" and it seems to have no effect whatsoever.
I am compiling with -g flag. Here are the compiler flags I use:

arm-none-eabi-gcc -x assembler-with-cpp -c -mcpu=cortex-m7 -mthumb -mfpu=fpv5-sp-d16 -mfloat-abi=hard -O0 -Wall -fdata-sections -ffunction-sections -g -gdwarf-2

For me, execution halts on a breakpoint at the beginning of main regardless of "stopAtEntry" value. I can add multiple breakpoints while the target is halted and they all work. It's just that I cannot halt with the pause button or add new breakpoints anymore while it's running. I would need a way to send ctrl+c to the arm-none-eabi-gdb process.

[pieandcakes]

@am1ko Can you enable "logging" : { "trace": true, "traceResponse": true, "engineLogging": true } and retry your scenario and send me the log?

I have seen issues in GDB where it won't allow us to set breakpoints while running in MI mode.

[am1ko]

launch copy.json.txt

[am1ko]

debug_log.txt

[am1ko]

@pieandcakes logs attached

I tried to halt and add breakpoints from vscode when recording the log but I see no log entries written at those times.

The breakpoint you see being hit is something that I trigger by controlling the ui of the program being debugged.

[timurey]

My gcc options: -mcpu=cortex-m3 -mthumb -fmessage-length=0 -ffunction-sections -fdata-sections -std=c11 -Og -g3 -flto
launch.json.txt
And I have absolutely same reaction on trying to halt or add breakpoint while programs is running.

But if I connect to openocd via telnet, I can halting program by halt command.

[pieandcakes]

@timurey @am1ko Can either of you try and run -exec -gdb-set target-async off in the Debug Console when you are stopped and see if the breakpoints are responsive? I'm hearing that some of versions of gdb are not handling this correctly.

[timurey]

Result:

-exec -gdb-set target-async off
1: (8106) <-1018-gdb-set target-async off
1: (8106) ->1018^error,msg="Cannot change this setting while the inferior is running."
1: (8106) ->(gdb)
1: (8106) 1018: elapsed time 0
result-class: error
msg: Cannot change this setting while the inferior is running.

in telnet console:

> set target-async off
off

But still can't halt running or add breakpoint.

[am1ko]

vscode version 1.15.1 (1.15.1)
arm-none-eabi-gcc version 6.3.1 20170620

After updating vscode and gcc I now see debug output when trying to break. Still does not work though

C pause: {"threadId":57005}
R: {"success":true,"message":null,"request_seq":23,"command":"pause","body":null,"running":false,"refs":null,"seq":0,"type":"response"}

[am1ko]

@pieandcakes
Tried your suggestion. Same result as @timurey

C evaluate: {"expression":"-exec -gdb-set target-async off","frameId":1000,"context":"repl"}
1: (34420) <-1092-gdb-set target-async off
1: (34421) ->1092^error,msg="Cannot change this setting while the inferior is running."
1: (34421) ->(gdb)
1: (34421) 1092: elapsed time 0
E output: {"category":"stdout","output":"result-class: error\r\nmsg: Cannot change this setting while the inferior is running.\r\n\n","data":null,"type":"output"}
result-class: error
msg: Cannot change this setting while the inferior is running.

[am1ko]

If I do
"setupCommands": [
{
"text": "set target-async off"
},

I can break the execution successfully form bash by sending SIGINT to the gdb process
kill -2 31276

vscode pause button still doesn't work.

[chrisfou]

Did you try using "hardware breakpoint" instead of "software breakpoint" ?
"HW breakpoint" the only solution i found to debug on a STM32F746 evaluation board . i suppose the "SW breakpoint" not enough when the code is executed from the flash memory.

[biboc]

@pieandcakes, any news on this?
I have the same issue, can't Pause when target is running

[pieandcakes]

@robotdad @paulmaybee Is this something you've seen with embedded arm devices and arm gdb with MIEngine?

[biboc]

In my case, I use arm device, arm-none-eabi-gdb
I'm able to:

• Start debugging and stop on main()
• Add breakpoints and stop on breakpoints
• Restart the target
• Stop the debug

I'm not able to:

• Pause the target while running

[pieandcakes]

@biboc Would you be able to try changing "MIMode": "gdb" to "MIMode": "lldb" in your launch.json and trying to pause? We use a different method of stopping the debugger execution there and it would be interesting if that method works

EDIT: actually this won't change the way it works, so ignore this.

[am1ko]

@pieandcakes is there any method that would send SIGINT to the arm-none-eabi-gdb process? I have tested that it works, just don't know how to trigger it from vscode.

"I can break the execution successfully form bash by sending SIGINT to the gdb process
kill -2 31276"

[pieandcakes]

@am1ko you can try using -exec <command> to send commands to gdb, but we don't have a way today to send commands to the terminal.

When looking at this it looks like -exec -interpreter-exec console "signal 2" should do it, although I don't know if gdb is responsive during this time.

[iitgrad]

I'm also encountering this issue. Has there been any resolution?

[pieandcakes]

@iitgrad Can you try my previous response and let me know if this works?

[btashton]

@pieandcakes can you clarify where/how I should be sending -exec <command> I can reproduce this and am happy to work on it.

[pieandcakes]

@btashton the -exec <command> option is issued in the debug console.

With the version that was release, 0.18.0, there should be an available workaround. The change was here, but it requires you to use a pipeTransport option such as ssh to communicate to the debugger. We will use the same pipeTransport application to attempt to send a signal to the debuggee.

[btashton]

Interesting. When I run via GDB I am able to halt the processor without an issues. But using the pause button in vscode I get this where the pause does nothing and I eventually hit another breakpoint.

1: (9343) <--exec-continue
1: (9343) ->^running
1: (9344) ->*running,thread-id="all"
1: (9344) ->(gdb)
1: (9344) ->(gdb)
C pause: {"threadId":1073430440}
C pause: {"threadId":1073430440}
1: (20453) ->@"Target halted. PRO_CPU: PC=0x400D206D (active)    APP_CPU: PC=0x400E10BE \n"```

[pieandcakes]

@btashton

When I run via GDB I am able to halt the processor without an issues

How are you halting the processor when you are using GDB?

[btashton]

I send a SIGINT via ctrl-c or kill -2 The output looks like this:

Program received signal SIGINT, Interrupt.
[Switching to Thread 1073436932]
0x400e10be in esp_vApplicationWaitiHook ()
    at /home/user1/esp-idf/components/esp32/freertos_hooks.c:66
66	    asm("waiti 0");

[pieandcakes]

@btashton right. That's what we do based on my comment. When you are using gdbserver though I don't think it goes through (but i'll have to check).

[BillPlunkett]

Got it. Unfortunately, that seems to have no effect.

[pieandcakes]

@BillPlunkett can you try and add

"setupCommands": [
{
"text": "set target-async off"
},

to your launch.json and then try the local kill again? @am1ko above says that worked for them.

[BillPlunkett]

hahaha, you're one step ahead of me, @pieandcakes. When I add the following:
"setupCommands": [
{
"text": "set target-async off"
},
I am able to pause by issuing kill -2 pid locally to gdb!

[pieandcakes]

@BillPlunkett ok good to know. I'll see if I can get a fix in for next release.

[BillPlunkett]

Awesome, thank you.

[igrr]

Running into the same issue as @btashton and @BillPlunkett. When I press pause button to interrupt the program, i see the following logged in the console:

kill: 42000: No such process

Furthermore, when i stop debugging (via the stop button), i see kill: 42000: No such process logged once again. After this, i run ps and see that gdb process is still running.

So it seems that the extension is sending SIGINT and SIGKILL to the wrong process somehow?

I can reproduce this at will and would love to help fixing this @pieandcakes, if you can give some pointers where to look.

[BillPlunkett]

I'm very willing to help anyway I can in fixing this, too. I am a big fan of VS Code, but currently having to use Eclipse as a GDB front end for remote debug due to this issue.

[pieandcakes]

@igrr @BillPlunkett the debug adapter code is here and the instructions on how to get it set up are here for this extension.

I think there are two parts to this.

• we need to offer an option to users to not set target-async to on (which is currently the default.
• we should be sending kill -2 to gdb which will then pause the debuggee. If what @igrr is saying about the wrong pid is true, I'm curious as to what that pid is. The line of code that you can set a breakpoint in is here

Thank you for offering to look into it.

[igrr]

Thanks for the pointers, @pieandcakes.
I am now able to debug MIEngine using VS on Windows, but the issue doesn't happen there (i was originally testing on osx). Will need more time to set up remote debugging.

However I did notice that in Debugger.cs, CmdBreakInternal, you get the PID of an inferior and pass it to _transport.Interrupt. In my case, GDB is running locally (so i guess I'm using LocalTransport) and the target is a remote one. Actually, the target doesn't have any processes, it is a bare metal target. In this case (gdb running locally, and is connected to remote gdb server) the interrupt should be sent to debugger process.

Looking at CmdBreakInternal, I am not sure whether the case of local gdb and remote gdb server is handled correctly. In all cases, PID is taken from debuggee PIDs, but it should be debugger PID. I.e. we should send the signal to GDB process, which will in turn interrupt the remote.

[igrr]

I have confirmed (by adding some logging, can't get remote debugging of MIDebug working yet) that IsLocalGDB() == true (as expected), 42000 is the debuggeePid value (not sure where it comes from, but it is always the same on every launch), useSignal == true, and CmdBreakUnix is sending interrupt to pid 42000 instead of sending it to transport.DebuggerPid().

Edit: 42000 is the magic value which GDB uses as the PID if the remote target does not support multiprocess extensions and sends only the thread id (https://github.com/bminor/binutils-gdb/blob/417e50dbcfd4b8dd699f48df5ac9b9a733fd80e2/gdb/remote.c#L14834-L14836)

[pieandcakes]

@igrr Thank you for investigating.
if you change Debugger.cs:715 to be the following:

                        debuggeePid = _transport.DebuggerPid; 

and try your scenario with the updated bits, can you confirm it works?

[igrr]

With this change it does work on Mac (haven't tested on Linux, assume it's also going to work). I assume that the proper change would be to introduce a new option, remote, and decide which PID to interrupt based on that. One more place where this option needs to be checked is when debuggee is started. For remote target, gdb can't use run, and continue should be used instead.

However going back to Windows, I can't get a breakpoint in CmdBreakInternal to get hit, when I press the 'pause' button in VS Code. (Breakpoints in various initialisation functions do get hit.)
What would be a good way to trace whether VS Code is actually sending anything to MIDebug when I press the 'pause' button?

[igrr]

It seems that this is fixed by microsoft/MIEngine@ecc1b0e.

[sean-mcmanus]

@igrr So it's fixed with 0.24.0? It has an updated MIEngine.

[dcrivelli]

This seems not yet working either with VS2017 (15.9.19) and VS2019 (16.4.3). Still have to pause through 'kill -2'.

[pieandcakes]

@dcrivelli this is for the extension for VS Code, not VS.

Please open a detailed issue for your scenario on https://github.com/microsoft/miengine with repro steps and i'll look into it.

[heartacker]

It seems that this is fixed by microsoft/MIEngine@ecc1b0e.

@igrr So it's fixed with 0.24.0? It has an updated MIEngine.

hi. any update? I have the same problem

[stansotn]

I am having the same problem with embedded ARM. More details in #807

[dachalco]

I'm using a different compiler but having the same issue. Remote target is halted but VSCode interface not synced with remote target.

[BillPlunkett]

I also had some difficulty getting 'pause' to work in a remote gdb/gdbserver session. For me, the key was to set MIDebuggerServerAddress to the ip address:port of the gdb server. Without that, everything works fine except pausing the running target.

[brianhindman]

VSCode Version: 1.56.2
OS Version: MacOS Big Sur 11.2.3
arm-none-eabi-gcc version: 6.3.1
arm-none-eabi-gdb version: 7.12.1

Steps to Reproduce (same as @am1ko):

Quoting am1ko:
"Setup launch.json to use arm-none-eabi-gdb for debugging a c program on an embedded target (see attached launch.txt)
Set one breakpoint to beginning of main
Start debug session
Breakpoint in main fires. Press "play" button to continue
Try to halt the target by pressing "pause"
The result is that nothing happens. "

One addition is that trying to stop the target after pressing "play" with no other breakpoints set does not work.

I can kill the gdb process with the pid, and sending a kill -2 works. See launch.json below.

launch-json.txt

[fbs2016]

I meet the same issue, see details #7719 Unable to set breakpoint or pause on a running application with extended-remote connection debug