Breakpoint is not mapped to any trace

[zwilcox]

I'm connecting to gdb server with Ghidra 11.4.
I'm able to connect. However, when I try to enable a break point i get the following error:

A breakpoint is not mapped to any trace.  Cannot enable it.  Is there a target?  Check your module map.

The module tab is empty.

I've tried to manually add them. However, clicking apply doesn't save. There's no way for me to save it :(

[d-millar]

@zwilcox Is anything visible the Dynamic Listing window? If not, from the Regions window, use the top right pull-down to “Force full memory”. If that works (i.e. you see bytes), try adding the module by hand again.

[zwilcox]

There's nothing in the Dynamic Listing window. I didn't see a Force full memory. I clicked on Force Full View

[zwilcox]

Actually, I restarted the debugger and server after adding the dynamic view.
Still can't add the addresses

[d-millar]

Oops - apologies. Don’t have my computer in front of me. Yes re “Force Full View”. Looking at the Static Mapping view in your snapshot, it looks like the ranges are backwards. (Again, can’t access my computer at the moment, so possibly I’m confused, but….)

[d-millar]

In other words, an assuming your program (in Listing) has a base address of 0x400000. I haven’t done this in a while, so possibly a remembering the wrong order.

[zwilcox]

The base is 0x400000
I'm pretty sure I don't have things backwards.

[zwilcox]

I'm also getting this exception in the GDB terminal. Not sure if it's related:

Using the DATA64 compiler map
Traceback (most recent call last):
  File "/home/user/ghidra_11.4_PUBLIC/Ghidra/Debug/Debugger-agent-gdb/py
pkg/src/ghidragdb/hooks.py", line 169, in _func
    return func(*args, **kwargs)
  File "/home/user/ghidra_11.4_PUBLIC/Ghidra/Debug/Debugger-agent-gdb/py
pkg/src/ghidragdb/hooks.py", line 336, in on_stop
    state.record("Stopped")
  File "/home/user/ghidra_11.4_PUBLIC/Ghidra/Debug/Debugger-agent-gdb/py
pkg/src/ghidragdb/hooks.py", line 115, in record
    commands.put_modules()
  File "/home/user/ghidra_11.4_PUBLIC/Ghidra/Debug/Debugger-agent-gdb/py
pkg/src/ghidragdb/commands.py", line 1283, in put_modules
    modobj.set_value('Range', base_addr.extend(m.max - m.base))
  File "/home/user/ghidra_11.4_PUBLIC/Ghidra/Debug/Debugger-rmi-trace/py
pkg/src/ghidratrace/client.py", line 217, in set_value
    return self.trace._set_value(self, span, key, value, schema, resolut
ion)
  File "/home/user/ghidra_11.4_PUBLIC/Ghidra/Debug/Debugger-rmi-trace/py
pkg/src/ghidratrace/client.py", line 421, in _set_value
    return self.client._set_value(self.id, object, span, key, value, sch
ema,
  File "/home/user/ghidra_11.4_PUBLIC/Ghidra/Debug/Debugger-rmi-trace/py
pkg/src/ghidratrace/client.py", line 1225, in _set_value
    self._write_value(root.request_set_value.value.value, value, schema)
  File "/home/user/ghidra_11.4_PUBLIC/Ghidra/Debug/Debugger-rmi-trace/py
pkg/src/ghidratrace/client.py", line 829, in _write_value
    Client._write_range(to.range_value, value)
  File "/home/user/ghidra_11.4_PUBLIC/Ghidra/Debug/Debugger-rmi-trace/py
pkg/src/ghidratrace/client.py", line 750, in _write_range
    to.extend = range.length() - 1
ValueError: Value out of range: -1
Python Exception <class 'ValueError'>: Value out of range: -1
Error occurred in Python: Value out of range: -1
(gdb) Python Exception <class 'gdb.MemoryError'>: Cannot access memory a
t address 0x10000
Python Exception <class 'gdb.MemoryError'>: Cannot access memory at addr
ess 0x0
Python Exception <class 'gdb.MemoryError'>: Cannot access memory at addr
ess 0x0
Python Exception <class 'gdb.MemoryError'>: Cannot access memory at addr
ess 0x0

[nsadeveloper789]

The Value out of range: -1 error should be fixed in 92708ef. The Cannot access memory at address 0x0 ones are very common. Sometimes it's because part of the UI is transiently positioned at page 0. Sometimes it's because the UI is evaluating a Watch that tries to deref a null. Etc. They're nothing to worry about, usually.

[d-millar]

OK, but am pretty sure you want that entered in the Static Range

[nsadeveloper789]

I concur with @d-millar. I think you have the Static Range and the Dynamic Range swapped.

[d-millar]

The error is telling you you’re trying to read memory at address 0x0, which is presumably invalid

[zwilcox]

So I've tried the other way around too. Still doesn't allow me to save. No error message.

[nsadeveloper789]

Okay. That's interesting.

1. The Apply button there should have worked. (Honestly, it should have worked even when swapped, even though the result would not have been what you wanted.)
2. Can you try the original breakpoint again after hitting Apply? Even though the entry is not displaying in the Static Mappings window, I wonder if it's active. We've had another recent issue where we think the mapping is actually added to the database, but for some reason or another, the window doesn't update. I'd be curious to see if any component sees the new mapping.

[zwilcox]

Here's the proc mappings:

00400000-00420000 r-xp 00000000 08:01 785010     /root/squashfs-root/usr/bin/tmpServer
00420000-00421000 rwxp 00020000 08:01 785010     /root/squashfs-root/usr/bin/tmpServer
00421000-00427000 rwxp 00000000 00:00 0          [heap]
77e3f000-77e42000 r-xp 00000000 08:01 783527     /root/squashfs-root/lib/libdl-0.9.33.2.so
77e42000-77e51000 ---p 00000000 00:00 0 
77e51000-77e52000 r-xp 00002000 08:01 783527     /root/squashfs-root/lib/libdl-0.9.33.2.so
77e52000-77e53000 rwxp 00003000 08:01 783527     /root/squashfs-root/lib/libdl-0.9.33.2.so
77e53000-77ee1000 r-xp 00000000 08:01 783383     /root/squashfs-root/lib/libuClibc-0.9.33.2.so
77ee1000-77ef0000 ---p 00000000 00:00 0 
77ef0000-77ef1000 r-xp 0008d000 08:01 783383     /root/squashfs-root/lib/libuClibc-0.9.33.2.so
77ef1000-77ef2000 rwxp 0008e000 08:01 783383     /root/squashfs-root/lib/libuClibc-0.9.33.2.so
77ef2000-77ef8000 rwxp 00000000 00:00 0 
77ef8000-77f0b000 r-xp 00000000 08:01 783405     /root/squashfs-root/lib/libpthread-0.9.33.2.so
77f0b000-77f1a000 ---p 00000000 00:00 0 
77f1a000-77f1b000 r-xp 00012000 08:01 783405     /root/squashfs-root/lib/libpthread-0.9.33.2.so
77f1b000-77f1c000 rwxp 00013000 08:01 783405     /root/squashfs-root/lib/libpthread-0.9.33.2.so
77f1c000-77f1e000 rwxp 00000000 00:00 0 
77f1e000-77f31000 r-xp 00000000 08:01 783750     /root/squashfs-root/lib/libgcc_s.so.1
77f31000-77f41000 ---p 00000000 00:00 0 
77f41000-77f42000 rwxp 00013000 08:01 783750     /root/squashfs-root/lib/libgcc_s.so.1
77f42000-77f57000 r-xp 00000000 08:01 783471     /root/squashfs-root/lib/libm-0.9.33.2.so
77f57000-77f66000 ---p 00000000 00:00 0 
77f66000-77f67000 rwxp 00014000 08:01 783471     /root/squashfs-root/lib/libm-0.9.33.2.so
77f67000-77f91000 r-xp 00000000 08:01 784931     /root/squashfs-root/usr/lib/liblua.so.5.1.4
77f91000-77fa1000 ---p 00000000 00:00 0 
77fa1000-77fa2000 rwxp 0002a000 08:01 784931     /root/squashfs-root/usr/lib/liblua.so.5.1.4
77fa2000-77fab000 r-xp 00000000 08:01 783536     /root/squashfs-root/lib/libubox.so
77fab000-77fba000 ---p 00000000 00:00 0 
77fba000-77fbb000 rwxp 00008000 08:01 783536     /root/squashfs-root/lib/libubox.so
77fbb000-77fbf000 r-xp 00000000 08:01 783776     /root/squashfs-root/lib/libubus.so
77fbf000-77fce000 ---p 00000000 00:00 0 
77fce000-77fcf000 rwxp 00003000 08:01 783776     /root/squashfs-root/lib/libubus.so
77fcf000-77fd8000 r-xp 00000000 08:01 783791     /root/squashfs-root/lib/libuci.so
77fd8000-77fe7000 ---p 00000000 00:00 0 
77fe7000-77fe8000 rwxp 00008000 08:01 783791     /root/squashfs-root/lib/libuci.so
77fe8000-77fef000 r-xp 00000000 08:01 783456     /root/squashfs-root/lib/ld-uClibc-0.9.33.2.so
77ffa000-77ffc000 rwxp 00000000 00:00 0 
77ffc000-77ffd000 r--p 00000000 00:00 0          [vvar]
77ffd000-77ffe000 r-xp 00000000 00:00 0          [vdso]
77ffe000-77fff000 r-xp 00006000 08:01 783456     /root/squashfs-root/lib/ld-uClibc-0.9.33.2.so
77fff000-78000000 rwxp 00007000 08:01 783456     /root/squashfs-root/lib/ld-uClibc-0.9.33.2.so
7ffde000-7ffff000 rwxp 00000000 00:00 0          [stack]
7ffff000-80000000 rwxp 00000000 00:00 0 
root@debian-mips:~# ^C

[d-millar]

If you enter 0x400000 for both, does it accept it? Or easier, if you hit the button in Modules for map the current program identically (i.e. at the same address range), does that create the mapping?

[nsadeveloper789]

The Modules, Sections, and Regions tables would not be affected by a Static Mapping. The only evidence of a static mapping existing (other than it being in the table) is UI behavior, e.g., listing synchronization, breakpoint placement.

[d-millar]

Following up, I think the root of the problem here is a mismatch between the output returned by your target for “info proc mappings” and “maintenance info sections —all-objects”, assuming these work at all, and the pattern templates in Ghidra/Debug/Debugger-agent-gdb/src/main/py/ghidragdb/util.py. It’s a pretty easy fix to change these as a one-off for your target. Does the mapping list above correspond to “info proc mappings”? And does the “maintenance” command work?

[zwilcox]

I think the root of the problem here is a mismatch between the output returned by your target for “info proc mappings”

When I run info proc mappings, I get the following error:

(gdb) info proc mapping
process 377
warning: unable to open /proc file '/proc/377/maps'

The listing above was done by reading /proc/377/mappings

And does the “maintenance” command work

(gdb) maintenance info sections —all-objects
Exec file: 
        `<path/to/the/binary/on/local/system/tmpServer', file type elf32-tradbigmips.

[d-millar]

OK, that's interesting. Normally, I assume gdbstub authors may or may not choose to support various commands, but in your case it looks like both are supported, sort of. In the first case, it looks like the gdbserver is not running with sufficient privilege to access /proc. Could be a lot of reasons for that, but in some sense "Force Full View" sidesteps that issue. You may want to play around with the "maintenance info sections" command. If there is some variant/option that actually lists the section info, you could replace the invocation in "util.py" and get the Module list for free.

That said, creating the mappings by hand should work. I am putting in a ticket to look at the "Add Static Mapping" dialog. I've always found it a little flaky / unintuitive, and at a minimum it should display an error when you enter invalid memory ranges. Have you gotten either method of making the identity mapping to work?

[zwilcox]

In the first case, it looks like the gdbserver is not running with sufficient privilege to access /proc.

It had to do with how I had my emulation environment setup. I've created a hacky work around and now info proc mapping works:

info proc mapping
process 11935
Mapped address spaces:

        Start Addr   End Addr       Size     Offset  Perms   objfile
          0x400000   0x420000    0x20000        0x0  r-xp   /root/squash
fs-root/usr/bin/tmpServer
          0x420000   0x421000     0x1000    0x20000  rwxp   /root/squash
fs-root/usr/bin/tmpServer
          0x421000   0x427000     0x6000        0x0  rwxp   [heap]
        0x77e3f000 0x77e42000     0x3000        0x0  r-xp   /root/squash
fs-root/lib/libdl-0.9.33.2.so
        0x77e42000 0x77e51000     0xf000        0x0  ---p   
        0x77e51000 0x77e52000     0x1000     0x2000  r-xp   /root/squash
fs-root/lib/libdl-0.9.33.2.so
        0x77e52000 0x77e53000     0x1000     0x3000  rwxp   /root/squash
fs-root/lib/libdl-0.9.33.2.so
        0x77e53000 0x77ee1000    0x8e000        0x0  r-xp   /root/squash
fs-root/lib/libuClibc-0.9.33.2.so
        0x77ee1000 0x77ef0000     0xf000        0x0  ---p   
        0x77ef0000 0x77ef1000     0x1000    0x8d000  r-xp   /root/squash
fs-root/lib/libuClibc-0.9.33.2.so
        0x77ef1000 0x77ef2000     0x1000    0x8e000  rwxp   /root/squash
fs-root/lib/libuClibc-0.9.33.2.so
        0x77ef2000 0x77ef8000     0x6000        0x0  rwxp   
        0x77ef8000 0x77f0b000    0x13000        0x0  r-xp   /root/squash
fs-root/lib/libpthread-0.9.33.2.so
        0x77f0b000 0x77f1a000     0xf000        0x0  ---p   
        0x77f1a000 0x77f1b000     0x1000    0x12000  r-xp   /root/squash
fs-root/lib/libpthread-0.9.33.2.so
        0x77f1b000 0x77f1c000     0x1000    0x13000  rwxp   /root/squash
fs-root/lib/libpthread-0.9.33.2.so
        0x77f1c000 0x77f1e000     0x2000        0x0  rwxp   
        0x77f1e000 0x77f31000    0x13000        0x0  r-xp   /root/squash
fs-root/lib/libgcc_s.so.1
        0x77f31000 0x77f41000    0x10000        0x0  ---p   
        0x77f41000 0x77f42000     0x1000    0x13000  rwxp   /root/squash
fs-root/lib/libgcc_s.so.1
        0x77f42000 0x77f57000    0x15000        0x0  r-xp   /root/squash
fs-root/lib/libm-0.9.33.2.so
        0x77f57000 0x77f66000     0xf000        0x0  ---p   
        0x77f66000 0x77f67000     0x1000    0x14000  rwxp   /root/squash
fs-root/lib/libm-0.9.33.2.so
        0x77f67000 0x77f91000    0x2a000        0x0  r-xp   /root/squash
fs-root/usr/lib/liblua.so.5.1.4
        0x77f91000 0x77fa1000    0x10000        0x0  ---p   
        0x77fa1000 0x77fa2000     0x1000    0x2a000  rwxp   /root/squash
fs-root/usr/lib/liblua.so.5.1.4
        0x77fa2000 0x77fab000     0x9000        0x0  r-xp   /root/squash
fs-root/lib/libubox.so
        0x77fab000 0x77fba000     0xf000        0x0  ---p   
--Type <RET> for more, q to quit, c to continue without paging--
        0x77fba000 0x77fbb000     0x1000     0x8000  rwxp   /root/squash
fs-root/lib/libubox.so
        0x77fbb000 0x77fbf000     0x4000        0x0  r-xp   /root/squash
fs-root/lib/libubus.so
        0x77fbf000 0x77fce000     0xf000        0x0  ---p   
        0x77fce000 0x77fcf000     0x1000     0x3000  rwxp   /root/squash
fs-root/lib/libubus.so
        0x77fcf000 0x77fd8000     0x9000        0x0  r-xp   /root/squash
fs-root/lib/libuci.so
        0x77fd8000 0x77fe7000     0xf000        0x0  ---p   
        0x77fe7000 0x77fe8000     0x1000     0x8000  rwxp   /root/squash
fs-root/lib/libuci.so
        0x77fe8000 0x77fef000     0x7000        0x0  r-xp   /root/squash
fs-root/lib/ld-uClibc-0.9.33.2.so
        0x77ffa000 0x77ffc000     0x2000        0x0  rwxp   
        0x77ffc000 0x77ffd000     0x1000        0x0  r--p   [vvar]
        0x77ffd000 0x77ffe000     0x1000        0x0  r-xp   [vdso]
        0x77ffe000 0x77fff000     0x1000     0x6000  r-xp   /root/squash
fs-root/lib/ld-uClibc-0.9.33.2.so
        0x77fff000 0x78000000     0x1000     0x7000  rwxp   /root/squash
fs-root/lib/ld-uClibc-0.9.33.2.so
        0x7ffde000 0x7ffff000    0x21000        0x0  rwxp   [stack]
        0x7ffff000 0x80000000     0x1000        0x0  rwxp

That said, creating the mappings by hand should work. I am putting in a ticket to look at the "Add Static Mapping" dialog

Thank you

Have you gotten either method of making the identity mapping to work?

No. Unfortunately. I just ended up using GDB and IDA. I'd much rather use Ghidra though

[d-millar]

Hmmm, what happens when you click the red double-arrow button in Modules?

[zwilcox]

Nothing. No change, no error.

[d-millar]

OK, some I'm clearly not understanding something and probably need to discuss with @nsadeveloper789. I thought "Force Full View" would create the necessary memory block, but I may be wrong about that. Really the only way "Map the current trace..." could fail is for there to be no memory, I think.

One other thing to try if you're game: in the Regions view, use "Add Region" to create a region from 0-7FFFFFFFFFFF (or any number that covers the /proc space), verify something shows up in Regions, and then try the "Map the current trace..." exercise.

[zwilcox]

When I start gdb-server with my hack, I see this in the region

Note, that path is what the OS and sees, not what the process sees.

so for example, the first path:
The OS sees:
/root/squash-fs/usr/<rest/of/path>
The process thinks it's accessing
/usr/<rest/of/path>

Fixed the paths. Same issue:

I'm unable to change the above mappings. I can delete them though

When I start gdb-server without my hack, I see:

Clicking on both versions of Mapt o region for <binary> throw's a null pointer :(
Clicking on Map to Region gives the following error:

[d-millar]

Actually, that’s better for us. The stack from the null pointer exception and/or the error message at least helps us circumscribe the error. With your hack for the regions in place, did you hair to try “Map the current trace to the current program” from Modules (rather than the Regions options)?

[d-millar]

There's also a gdb scipt in Ghidra/Debug/Debugger-agent-gdb/data/scripts called fallback_info_proc_mappings.gdb that implements a dummy version of "info proc mappings". Running that will also add default mappings, much like "Add Region".

[zwilcox]

Would it be easier for me to zip up my qemu environment and give you steps on how to reproduce so we don't have to have these back and forths??

[d-millar]

That would be awesome!

[zwilcox]

I'm assuming you know most of the details so I'm going to go a bit light on details but I'm more than happy to go into details if needed.
Preq, install qemu-system-mips and gdb-multiarch

1. untar file.
2. import squashFS dir into ghidra
3. start qemu system by running start.sh script
4. login root/root
5. run the start_gdb_server script. It chroots into a to the squashFS dir and starts gdb server on port 12345 for tmpServer binary
6. If need the hack for gdb-server to see /proc/<pid>/mapping , then copy the dir of /pid/<pid_gdb_server_give>' to /root/squashfs/pid`
7. open ghidra's debug tool on tmpServer
8. here's the debug config:

9. I put a breakpoint in main to get out of the loader:

b *0x401ee0
c

The tar file is too large for github for so I put it on dropbox:
https://www.dropbox.com/scl/fi/3ehhcir5vn0nwpg16r7ry/gdb-issue.tar.gz?rlkey=yw3l980on4wl22sleqi3zgg1x&e=1&st=eqtr6slb&dl=0

Let me know if there's any issues

[d-millar]

Cool - am away from my computer for a couple of hours, but will take a stab right after that!

[d-millar]

Got it!

[d-millar]

OK, on the (slightly) good new front, I can exactly reproduce your results. On the less good front, I haven't yet figured out what's going on. A couple of notes in the meantime:

(1) Am putting "mips:5000" in for "Architecture" in the launch dialog. If I don't do this, the Registers view has only two skeleton values because we didn't anticipate "show architecture" returning only "mips". Adding a ticket for this.
(2) When I launch, the current thread is not getting selected in the Model tree. I have to manually open Inferiors[1].Threads[1].Stack and double-click the value to get the Toolbar buttons enabled and the Registers view to populate. Also not ideal.
(3) Regions is populating with 0-0xFFFFFFFF (once I do step 2) , which is probably ok for the moment. I haven't tried your /proc hack yet, although I get what your're doing - nice hack.
(4) Attempting to set a breakpoint in the Static Listing doesn't work because the Module map is missing.
(5) Attempting to set a breakpoint in the Dynamic Listing does work, but doesn't automatically update the Breakpoints view as it should (and does for most targets). I have to step or refresh the tree to see it.
(6) Same for setting breakpoints in the Terminal - they show up eventually and work fine, but aren't visible in the Static Listing.
(7) As you have noted, there appears to be no way to add a new mapping to Modules. I had it succeed exactly once and haven't been able to repeat what I did.

My current guess is that the timeout for the connection needs to be increased for this target, i.e. responses may be getting randomly dropped on the floor. More as I know more....

[zwilcox]

I have an x86 binary that has the same (or at least similar) problem with it without gdb server. Admittedly there is some anti-debugging stuff going on in the binary. However, I would expect Ghidra to be able to attach to it before hitting that part of the code. I can step through it in normal gdb.

Would you like this binary as well?

[d-millar]

You bet!

[zwilcox]

debugger_wont_work.tar.gz

[d-millar]

Same problem on that one as the server case: it appears none of these executables list sections for "maintenance info sections -all-objects". This triggers the "range" exception you've posted (@nsadeveloper789 fixing this as I type). Then, of course, the inability to add the mapping bites you (also looking at that!).

[zwilcox]

@nsadeveloper789 fixing this as I type

Then, of course, the inability to add the mapping bites you (also looking at that!).

You guys are awesome! However, I'm a bit annoyed because you guys are making my Ghidra Plugin obsolete :P

[nsadeveloper789]

Okay. I have a theory. I think, oddly enough, it may be related to the Value out of range: -1 error way above. I should check and see if that fix is scheduled for 11.4.1... There are two things I'd be curious to check:

1. After you launch (and you presumably see the -1 error, check the Connections window. Does the connection have a red lock like this ? If so, then it looks like our gdb plugin failed to close a transaction, and that will hold up most event processing in the UI for that target. I'm able to reproduce what seems to be the same behavior by manually opening a transaction
   and never closing it.
2. Referring to 92708ef, you should be able to apply the same change in your installation. Just look for commands.py somewhere within the Ghidra/Debug/Debugger-agent-gdb/ directory. Then retry launching and see if that resolves the issue.