Skip to content

Commit

Permalink
Auto merge of #33612 - royalstream:royalstream-enc-enum-ptr, r=michae…
Browse files Browse the repository at this point in the history
…lwoerister

gdb Pretty Print: generic encoded was failing on reference/pointer types

If you debug this program using **gdb**
```rust
fn main() {
    let x = 10;
    let y = Some(&x);
    // additional code
}
```
And you try to print **y**'s value from the debugger, you get the following:
```
(gdb) print y
Python Exception <class 'gdb.error'> Cannot convert value to int.:
$1 = {RUST$ENCODED$ENUM$0$None = Some = {0x7fff5fbff97c}}
```
What happens is that inside **debugger_pretty_printers_common.py** the method `is_null_variant` doesn't have any special handling for pointer values so it ends up calling `.as_integer()` on `discriminant_val` (which holds a pointer) and fails.

Considering it needs to handle pointers and return _true_ when the pointer is _null_, I modified the `.as_integer()` method in **gdb_rust_pretty_printing.py** to take pointers into consideration.

After this modification **gdb** prints **y** like this:
```
(gdb) print y
$1 = Some = {0x7fff5fbff97c}
```
Now, it would be nice to print something useful (instead of a pointer address) but the pretty printer doesn't currently handle references/pointers so that's a completely different subject.
  • Loading branch information
bors committed May 15, 2016
2 parents 1a26d23 + 04baf25 commit b358319
Showing 1 changed file with 2 additions and 0 deletions.
2 changes: 2 additions & 0 deletions src/etc/gdb_rust_pretty_printing.py
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,8 @@ def get_child_at_index(self, index):
return child

def as_integer(self):
if self.gdb_val.type.code == gdb.TYPE_CODE_PTR:
return int(str(self.gdb_val), 0)
return int(self.gdb_val)

def get_wrapped_value(self):
Expand Down

0 comments on commit b358319

Please sign in to comment.