When connected to lldb-server's gdbserver on Windows, symbols are not loaded unless program file path given to lldb is the same

[DavidSpickett]

Adapted from https://discourse.llvm.org/t/does-anyone-know-why-the-breakpoint-is-unresolved-in-gdb-remote-debug/86647, which I confirmed locally.

Start a gdbserver running the program:

C:\Users\tcwg>llvm-worker\lldb-aarch64-windows\build\bin\lldb-server g :1234 -- C:\test.exe
Launched 'C:\test.exe' as process 22856...
lldb-server-local_build
Connection established.
lldb-server exiting...

Note the file path here. At this path is the exe and the PDB file.

Then start lldb, but create a target using the same files at a different path:

C:\Users\tcwg\llvm-worker\lldb-aarch64-windows\build>.\bin\lldb
(lldb) target create C:\Users\tcwg\test.exe
Current executable set to 'C:\Users\tcwg\test.exe' (aarch64).
(lldb) gdb-remote 127.0.0.1:1234
Process 22856 stopped
* thread #1, name = 'test.exe', stop reason = signal SIGTRAP
    frame #0: 0x00007ff9f3aaa7f4
->  0x7ff9f3aaa7f4: b      0x7ff9f3aaa7f8
    0x7ff9f3aaa7f8: ldp    x29, x30, [sp], #0x20
    0x7ff9f3aaa7fc: autibsp
    0x7ff9f3aaa800: ret
(lldb) b main
Breakpoint 1: where = test.exe`main at test.c:1, address = 0x0000000140014248
(lldb) br list
Current breakpoints:
1: name = 'main', locations = 1
  1.1: where = test.exe`main at test.c:1, address = test.exe[0x0000000140014248], unresolved, hit count = 0

(lldb) c
Process 22856 resuming
Process 22856 exited with status = 0 (0x00000000)

LLDB cannot resolve the breakpoint on main.

Now Windows will show some breakpoints as unresolved if you haven't started the process yet, but this is not the case here.

If you do the same thing and use the same path for each, the breakpoint is resolved:

C:\Users\tcwg>llvm-worker\lldb-aarch64-windows\build\bin\lldb-server g :1234 -- C:\Users\tcwg\test.exe
Launched 'C:\Users\tcwg\test.exe' as process 22976...
lldb-server-local_build
Connection established.

C:\Users\tcwg\llvm-worker\lldb-aarch64-windows\build>.\bin\lldb
(lldb) target create C:\Users\tcwg\test.exe
Current executable set to 'C:\Users\tcwg\test.exe' (aarch64).
(lldb) gdb-remote 127.0.0.1:1234
Process 22976 stopped
* thread #1, name = 'test.exe', stop reason = signal SIGTRAP
    frame #0: 0x00007ff9f3aaa7f4
->  0x7ff9f3aaa7f4: b      0x7ff9f3aaa7f8
    0x7ff9f3aaa7f8: ldp    x29, x30, [sp], #0x20
    0x7ff9f3aaa7fc: autibsp
    0x7ff9f3aaa800: ret
(lldb) b main
Breakpoint 1: where = test.exe`main at test.c:1, address = 0x00007ff68b264248
(lldb) br list
Current breakpoints:
1: name = 'main', locations = 1, resolved = 1, hit count = 0
  1.1: where = test.exe`main at test.c:1, address = 0x00007ff68b264248, resolved, hit count = 0

(lldb) c
Process 22976 resuming
Process 22976 stopped
* thread #1, name = 'test.exe', stop reason = breakpoint 1.1
    frame #0: 0x00007ff68b264248 test.exe`main at test.c:1
-> 1    int main() { return 0; }

Note that I'm using localhost here but I don't think that makes much difference at least on Windows. If anything I'd expect it to work better being on localhost.

The same setup works on Linux with same or different paths. I think the reason is that the DYLD plugin on Linux manages to get the load offset of the program file and realise that they are the same file, even if they're different copies of it.

We can see that Windows still thinks the path given to lldb is loaded after connection:

(lldb) image list
[  0] BC387FE5-BF4F-441B-8C85-0A548B56B7F2-00000001 0x0000000140000000 C:\Users\tcwg\test.exe

Later you'll see that on Linux, this gets replaced with the filename dyld finds.

On Windows, it gets as far as asking for the load offset but lldb-server can only look that up by path. Here I have enabled gdb-remote packet and lldb dyld logs:

 DynamicLoaderDarwin::UseDYLDSPI: Use new DynamicLoader plugin
 DynamicLoaderDarwin::UseDYLDSPI: Use new DynamicLoader plugin
 DynamicLoaderWindowsDYLD::virtual lldb_private::DynamicLoaderWindowsDYLD::DidAttach()
 <  65> send packet: $qFileLoadAddress:433a2f55736572732f746377672f746573742e657865#7c
 <   7> read packet: $E45#ae

433a2f55736572732f746377672f746573742e657865 is C:/Users/tcwg/test.exe in ASCII. You can see it fails to find the load address.

https://lldb.llvm.org/resources/lldbgdbremote.html#qfileloadaddress-file-path

If I use the same paths:

 DynamicLoaderDarwin::UseDYLDSPI: Use new DynamicLoader plugin
 DynamicLoaderDarwin::UseDYLDSPI: Use new DynamicLoader plugin
 DynamicLoaderWindowsDYLD::virtual lldb_private::DynamicLoaderWindowsDYLD::DidAttach()
 <  43> send packet: $qFileLoadAddress:433a2f746573742e657865#8b
 <  20> read packet: $00007ff7c2540000#b8
 <  18> send packet: $qShlibInfoAddr#6a
 <   4> read packet: $#00
 <  13> send packet: $qSymbol::#5b
 <   4> read packet: $#00

433a2f746573742e657865 is C:/test.exe in ASCII.

So clearly it can work, which means that it's not something silly like the / instead of Windows backslash.

On the Linux side I gave lldb /tmp/test.o and lldb-server ~/test.o:

lldb             <  20> read packet: $0000000000000000#00
lldb             <  12> send packet: $qOffsets#4b
lldb             <   4> read packet: $#00
lldb             <  26> send packet: $qStructuredDataPlugins#02
lldb             <   4> read packet: $#00
lldb             DynamicLoaderDarwin::UseDYLDSPI: Use new DynamicLoader plugin
lldb             DynamicLoaderDarwin::UseDYLDSPI: Use new DynamicLoader plugin
lldb             DYLDRendezvous::UpdateExecutablePath exe module executable path set: '/tmp/test.o'
lldb             DynamicLoaderPOSIXDYLD::DidAttach() pid 2224803
lldb             <  29> send packet: $qXfer:auxv:read::0,131071#d7
lldb             < 341> read packet: $...0#76
lldb             DynamicLoaderPOSIXDYLD::DidAttach pid 2224803 reloaded auxv data
lldb             DynamicLoaderPOSIXDYLD::ResolveExecutableModule - got executable by pid 2224803: /home/d
avid.spickett/test.o
lldb             <  13> send packet: $qSymbol::#5b
lldb             <   4> read packet: $#00
lldb             DYLDRendezvous::UpdateExecutablePath exe module executable path set: '/home/david.spicke
tt/test.o'
lldb             DynamicLoaderPOSIXDYLD::DidAttach pid 2224803 executable '/home/david.spickett/test.o', 
load_offset 0xaaaaaaaaa000
lldb             DynamicLoaderPOSIXDYLD::DidAttach pid 2224803 added executable '/home/david.spickett/tes
t.o' to module load list
lldb             <  34> send packet: $qMemoryRegionInfo:fffff7ffb000#07
lldb             <  73> read packet: $start:fffff7ffb000;size:1000;permissions:rx;flags:;name:5b7664736f5
d;#47
lldb             < 104> send packet: $qSupported:xmlRegisters=i386,arm,mips,arc;multiprocess+;fork-events
+;vfork-events+;swbreak+;hwbreak+#cd
lldb             < 260> read packet: $PacketSize=131072;QStartNoAckMode+;qEcho+;native-signals+;QThreadSu
ffixSupported+;QListThreadsInStopReply+;qXfer:features:read+;QNonStop+;QPassSignals+;qXfer:auxv:read+;qXf
er:libraries-svr4:read+;qXfer:siginfo:read+;multiprocess+;fork-events+;vfork-events+#82
lldb             <   8> send packet: $x0,0#04
lldb             <   6> read packet: $OK#9a
lldb             <  22> send packet: $xfffff7ffb000,1000#58
lldb             <4114> read packet: $...#fd
lldb             DYLDRendezvous::Resolve address size: 8, padding 4
lldb             <  18> send packet: $qShlibInfoAddr#6a
lldb             <   4> read packet: $#00
lldb             <  39> send packet: $qXfer:libraries-svr4:read::0,131071#8c
lldb             <  54> read packet: $Eff;496e76616c696420725f64656275672061646472657373#5b
lldb             ResolveRendezvousAddress info_location = 0xffffffffffffffff
lldb             ResolveRendezvousAddress resolved via direct object file approach to 0xaaaaaaabae70
lldb             ResolveRendezvousAddress reading pointer (8 bytes) from 0xaaaaaaabae70
lldb             <  21> send packet: $xaaaaaaabae00,200#65
lldb             < 516> read packet: $...
lldb             ResolveRendezvousAddress FAILED - the rendezvous address contained at 0xaaaaaaabae70 ret
urned a null value
lldb             DYLDRendezvous::Resolve cursor = 0xffffffffffffffff
lldb             DynamicLoaderPOSIXDYLD::LoadAllCurrentModules unable to resolve POSIX DYLD rendezvous ad
dress
lldb             DynamicLoaderPOSIXDYLD::DidAttach told the target about the modules that loaded:
lldb             -- [module] /home/david.spickett/test.o (pid 2224803)
lldb             Rendezvous structure is not set up yet. Trying to locate rendezvous breakpoint in the in
terpreter by symbol name.
lldb             <  34> send packet: $qMemoryRegionInfo:fffff7fc2000#d4
lldb             < 158> read packet: $start:fffff7fc2000;size:2b000;permissions:rx;flags:;name:2f7573722f
6c69622f616172636836342d6c696e75782d676e752f6c642d6c696e75782d616172636836342e736f2e31;#98
lldb             <  21> send packet: $Z0,fffff7fc4c20,4#0d
lldb             <   6> read packet: $OK#9a
lldb             Successfully set rendezvous breakpoint at address 0xfffff7fc4c20 for pid 2224803

(lldb)

Now it doesn't ask for a load offset for a specific file, I think it's looking for the dyld structure to effectively have lldb-server tell lldb what is loaded.

Plus it's doing a PID lookup and finding the program that way, which means it's actually ignored the file I gave it and used the one lldb-server loaded:

(lldb) image list
[  0] 385F7D75-67D0-132C-D09E-FF0A95AFBDF4-EF428CB0 0x0000aaaaaaaaa000 /home/david.spickett/test.o 
[  1] 92523E33-43D6-9A71-50DE-A7FE322D3B11-E29C895E 0x0000fffff7ffb000 [vdso] (0x0000fffff7ffb000)
[  2] 46771E9A-54FF-F825-3560-60BC1C8B4A44-C1108BF8 0x0000fffff7fc2000 /usr/lib/aarch64-linux-gnu/ld-linu
x-aarch64.so.1 
      /usr/lib/debug/.build-id/46/771e9a54fff825356060bc1c8b4a44c1108bf8.debug

This happens because despite using gdb-remote we like to assume localhost is special, which is fine in this case but not always.

So I'm not sure which of those tactics Windows could in fact do.

Perhaps we could do some UUID/Build ID check on the file that we're asked for the load address of, but we wouldn't be able to do that if this was truly a remote connection. We'd need a qFileLoadAddress:<some_unique_id> packet for that.

[llvmbot]

@llvm/issue-subscribers-lldb

Author: David Spickett (DavidSpickett)

Adapted from https://discourse.llvm.org/t/does-anyone-know-why-the-breakpoint-is-unresolved-in-gdb-remote-debug/86647, which I confirmed locally.

Start a gdbserver running the program:

C:\Users\tcwg>llvm-worker\lldb-aarch64-windows\build\bin\lldb-server g :1234 -- C:\test.exe
Launched 'C:\test.exe' as process 22856...
lldb-server-local_build
Connection established.
lldb-server exiting...

Note the file path here. At this path is the exe and the PDB file.

Then start lldb, but create a target using the same file at a different path:

C:\Users\tcwg\llvm-worker\lldb-aarch64-windows\build>.\bin\lldb
(lldb) target create C:\Users\tcwg\test.exe
Current executable set to 'C:\Users\tcwg\test.exe' (aarch64).
(lldb) gdb-remote 127.0.0.1:1234
Process 22856 stopped
* thread #<!-- -->1, name = 'test.exe', stop reason = signal SIGTRAP
    frame #<!-- -->0: 0x00007ff9f3aaa7f4
-&gt;  0x7ff9f3aaa7f4: b      0x7ff9f3aaa7f8
    0x7ff9f3aaa7f8: ldp    x29, x30, [sp], #<!-- -->0x20
    0x7ff9f3aaa7fc: autibsp
    0x7ff9f3aaa800: ret
(lldb) b main
Breakpoint 1: where = test.exe`main at test.c:1, address = 0x0000000140014248
(lldb) br list
Current breakpoints:
1: name = 'main', locations = 1
  1.1: where = test.exe`main at test.c:1, address = test.exe[0x0000000140014248], unresolved, hit count = 0

(lldb) c
Process 22856 resuming
Process 22856 exited with status = 0 (0x00000000)

LLDB cannot resolve the breakpoint on main.

Now Windows will show some breakpoints as unresolved if you haven't started the process yet, but this is not the case here.

If you do the same thing and use the same path for each, the breakpoint is resolved:

C:\Users\tcwg>llvm-worker\lldb-aarch64-windows\build\bin\lldb-server g :1234 -- C:\Users\tcwg\test.exe
Launched 'C:\Users\tcwg\test.exe' as process 22976...
lldb-server-local_build
Connection established.

C:\Users\tcwg\llvm-worker\lldb-aarch64-windows\build>.\bin\lldb
(lldb) target create C:\Users\tcwg\test.exe
Current executable set to 'C:\Users\tcwg\test.exe' (aarch64).
(lldb) gdb-remote 127.0.0.1:1234
Process 22976 stopped
* thread #<!-- -->1, name = 'test.exe', stop reason = signal SIGTRAP
    frame #<!-- -->0: 0x00007ff9f3aaa7f4
-&gt;  0x7ff9f3aaa7f4: b      0x7ff9f3aaa7f8
    0x7ff9f3aaa7f8: ldp    x29, x30, [sp], #<!-- -->0x20
    0x7ff9f3aaa7fc: autibsp
    0x7ff9f3aaa800: ret
(lldb) b main
Breakpoint 1: where = test.exe`main at test.c:1, address = 0x00007ff68b264248
(lldb) br list
Current breakpoints:
1: name = 'main', locations = 1, resolved = 1, hit count = 0
  1.1: where = test.exe`main at test.c:1, address = 0x00007ff68b264248, resolved, hit count = 0

(lldb) c
Process 22976 resuming
Process 22976 stopped
* thread #<!-- -->1, name = 'test.exe', stop reason = breakpoint 1.1
    frame #<!-- -->0: 0x00007ff68b264248 test.exe`main at test.c:1
-&gt; 1    int main() { return 0; }

Note that I'm using localhost here but I don't think that makes much difference at least on Windows. If anything I'd expect it to work better being on localhost.

The same setup works on Linux with same or different paths. I think the reason is that the DYLD plugin on Linux manages to get the load offset of the program file and realise that they are the same file, even if they're different copies of it.

On Windows, it gets as far as asking for the load offset but lldb-server can only look that up by path. Here I have enabled gdb-remote packet and lldb dyld logs:

 DynamicLoaderDarwin::UseDYLDSPI: Use new DynamicLoader plugin
 DynamicLoaderDarwin::UseDYLDSPI: Use new DynamicLoader plugin
 DynamicLoaderWindowsDYLD::virtual lldb_private::DynamicLoaderWindowsDYLD::DidAttach()
 &lt;  65&gt; send packet: $qFileLoadAddress:433a2f55736572732f746377672f746573742e657865#<!-- -->7c
 &lt;   7&gt; read packet: $E45#ae

433a2f55736572732f746377672f746573742e657865 is C:/Users/tcwg/test.exe in ASCII. You can see it fails to find the load address.

https://lldb.llvm.org/resources/lldbgdbremote.html#qfileloadaddress-file-path

If I use the same paths:

 DynamicLoaderDarwin::UseDYLDSPI: Use new DynamicLoader plugin
 DynamicLoaderDarwin::UseDYLDSPI: Use new DynamicLoader plugin
 DynamicLoaderWindowsDYLD::virtual lldb_private::DynamicLoaderWindowsDYLD::DidAttach()
 &lt;  43&gt; send packet: $qFileLoadAddress:433a2f746573742e657865#<!-- -->8b
 &lt;  20&gt; read packet: $00007ff7c2540000#b8
 &lt;  18&gt; send packet: $qShlibInfoAddr#<!-- -->6a
 &lt;   4&gt; read packet: $#<!-- -->00
 &lt;  13&gt; send packet: $qSymbol::#<!-- -->5b
 &lt;   4&gt; read packet: $#<!-- -->00

433a2f746573742e657865 is C:/test.exe in ASCII.

So clearly it can work, which means that it's not something silly like the / instead of Windows backslash.

On the Linux side I gave lldb /tmp/test.o and lldb-server ~/test.o:

lldb             &lt;  20&gt; read packet: $0000000000000000#<!-- -->00
lldb             &lt;  12&gt; send packet: $qOffsets#<!-- -->4b
lldb             &lt;   4&gt; read packet: $#<!-- -->00
lldb             &lt;  26&gt; send packet: $qStructuredDataPlugins#<!-- -->02
lldb             &lt;   4&gt; read packet: $#<!-- -->00
lldb             DynamicLoaderDarwin::UseDYLDSPI: Use new DynamicLoader plugin
lldb             DynamicLoaderDarwin::UseDYLDSPI: Use new DynamicLoader plugin
lldb             DYLDRendezvous::UpdateExecutablePath exe module executable path set: '/tmp/test.o'
lldb             DynamicLoaderPOSIXDYLD::DidAttach() pid 2224803
lldb             &lt;  29&gt; send packet: $qXfer:auxv:read::0,131071#d7
lldb             &lt; 341&gt; read packet: $...0#<!-- -->76
lldb             DynamicLoaderPOSIXDYLD::DidAttach pid 2224803 reloaded auxv data
lldb             DynamicLoaderPOSIXDYLD::ResolveExecutableModule - got executable by pid 2224803: /home/d
avid.spickett/test.o
lldb             &lt;  13&gt; send packet: $qSymbol::#<!-- -->5b
lldb             &lt;   4&gt; read packet: $#<!-- -->00
lldb             DYLDRendezvous::UpdateExecutablePath exe module executable path set: '/home/david.spicke
tt/test.o'
lldb             DynamicLoaderPOSIXDYLD::DidAttach pid 2224803 executable '/home/david.spickett/test.o', 
load_offset 0xaaaaaaaaa000
lldb             DynamicLoaderPOSIXDYLD::DidAttach pid 2224803 added executable '/home/david.spickett/tes
t.o' to module load list
lldb             &lt;  34&gt; send packet: $qMemoryRegionInfo:fffff7ffb000#<!-- -->07
lldb             &lt;  73&gt; read packet: $start:fffff7ffb000;size:1000;permissions:rx;flags:;name:5b7664736f5
d;#<!-- -->47
lldb             &lt; 104&gt; send packet: $qSupported:xmlRegisters=i386,arm,mips,arc;multiprocess+;fork-events
+;vfork-events+;swbreak+;hwbreak+#cd
lldb             &lt; 260&gt; read packet: $PacketSize=131072;QStartNoAckMode+;qEcho+;native-signals+;QThreadSu
ffixSupported+;QListThreadsInStopReply+;qXfer:features:read+;QNonStop+;QPassSignals+;qXfer:auxv:read+;qXf
er:libraries-svr4:read+;qXfer:siginfo:read+;multiprocess+;fork-events+;vfork-events+#<!-- -->82
lldb             &lt;   8&gt; send packet: $x0,0#<!-- -->04
lldb             &lt;   6&gt; read packet: $OK#<!-- -->9a
lldb             &lt;  22&gt; send packet: $xfffff7ffb000,1000#<!-- -->58
lldb             &lt;4114&gt; read packet: $...#fd
lldb             DYLDRendezvous::Resolve address size: 8, padding 4
lldb             &lt;  18&gt; send packet: $qShlibInfoAddr#<!-- -->6a
lldb             &lt;   4&gt; read packet: $#<!-- -->00
lldb             &lt;  39&gt; send packet: $qXfer:libraries-svr4:read::0,131071#<!-- -->8c
lldb             &lt;  54&gt; read packet: $Eff;496e76616c696420725f64656275672061646472657373#<!-- -->5b
lldb             ResolveRendezvousAddress info_location = 0xffffffffffffffff
lldb             ResolveRendezvousAddress resolved via direct object file approach to 0xaaaaaaabae70
lldb             ResolveRendezvousAddress reading pointer (8 bytes) from 0xaaaaaaabae70
lldb             &lt;  21&gt; send packet: $xaaaaaaabae00,200#<!-- -->65
lldb             &lt; 516&gt; read packet: $...
lldb             ResolveRendezvousAddress FAILED - the rendezvous address contained at 0xaaaaaaabae70 ret
urned a null value
lldb             DYLDRendezvous::Resolve cursor = 0xffffffffffffffff
lldb             DynamicLoaderPOSIXDYLD::LoadAllCurrentModules unable to resolve POSIX DYLD rendezvous ad
dress
lldb             DynamicLoaderPOSIXDYLD::DidAttach told the target about the modules that loaded:
lldb             -- [module] /home/david.spickett/test.o (pid 2224803)
lldb             Rendezvous structure is not set up yet. Trying to locate rendezvous breakpoint in the in
terpreter by symbol name.
lldb             &lt;  34&gt; send packet: $qMemoryRegionInfo:fffff7fc2000#d4
lldb             &lt; 158&gt; read packet: $start:fffff7fc2000;size:2b000;permissions:rx;flags:;name:2f7573722f
6c69622f616172636836342d6c696e75782d676e752f6c642d6c696e75782d616172636836342e736f2e31;#<!-- -->98
lldb             &lt;  21&gt; send packet: $Z0,fffff7fc4c20,4#<!-- -->0d
lldb             &lt;   6&gt; read packet: $OK#<!-- -->9a
lldb             Successfully set rendezvous breakpoint at address 0xfffff7fc4c20 for pid 2224803

(lldb)

Now it doesn't ask for a load offset for a specific file, I think it's looking for the dyld structure to effectively have lldb-server tell lldb what is loaded.

Plus it's doing a PID lookup and finding the program that way, which means it's actually ignored the file I gave it and used the one lldb-server loaded:

(lldb) image list
[  0] 385F7D75-67D0-132C-D09E-FF0A95AFBDF4-EF428CB0 0x0000aaaaaaaaa000 /home/david.spickett/test.o 
[  1] 92523E33-43D6-9A71-50DE-A7FE322D3B11-E29C895E 0x0000fffff7ffb000 [vdso] (0x0000fffff7ffb000)
[  2] 46771E9A-54FF-F825-3560-60BC1C8B4A44-C1108BF8 0x0000fffff7fc2000 /usr/lib/aarch64-linux-gnu/ld-linu
x-aarch64.so.1 
      /usr/lib/debug/.build-id/46/771e9a54fff825356060bc1c8b4a44c1108bf8.debug

This happens because despite using gdb-remote we like to assume localhost is special, which is fine in this case but not always.

So I'm not sure which of those tactics Windows could in fact do.

Perhaps we could do some UUID/Build ID check on the file that we're asked for the load address of, but we wouldn't be able to do that if this was truly a remote connection. We'd need a qFileLoadAddress:<some_unique_id> packet for that.

[DavidSpickett]

The original reporter says that platform mode works, and I can understand that being the case for local debug as it would start the gdbserver for you using the same path as the created target.

I suspect that if you were truly remote, the program file would end up somewhere else and cause the same issue. Though, we have a Linux -> Windows remote debugging buildbot that seems to be fine, so maybe I'm wrong about that.

[DavidSpickett]

OP confirms that as long as the path is the same on the (actually remote separate machine), it will work - https://discourse.llvm.org/t/does-anyone-know-why-the-breakpoint-is-unresolved-in-gdb-remote-debug/86647/5.

The two exes are identical. After copying it to the target machine, I briefly checked md5. And if I move both the local and remote exes to C: , lldb work properly. Therefore, this should not be caused by file differences.