Android library loading messes up DR handling

[derekbruening]

On Android on any app we see:

<get_memory_info mismatch! (can happen if os combines entries in /proc/pid/maps)
        os says: 0xb6f0f000-0xb6f15000 prot=0x00000003
        cache says: 0xb6f0f000-0xb6f21000 prot=0x00000003
>

It's libc:

b6ea8000-b6f0e000 r-xp 00000000 b3:16 956        /system/lib/libc.so
b6f0e000-b6f0f000 ---p 00000000 00:00 0 
b6f0f000-b6f15000 rw-p 00066000 b3:16 956        /system/lib/libc.so
b6f15000-b6f1e000 rw-p 00000000 00:00 0 
b6f1e000-b6f21000 rw-p 00000000 00:00 0          [anon:linker_alloc]

First it seems to map the 1st page to read the headers I guess,
coincidentally sits where the linker anon is later put:

syscall: mmap2 addr=0x00000000 size=0x1000 prot=0x1 flags=0x2 offset=0x0 fd=3
process_mmap(0xb6f1e000,0x00001000,0x2,r--,mmap)
mmap 0xb6f1e000: partial

Then it does the final map:

syscall: mmap2 addr=0x00000000 size=0x76000 prot=0x0 flags=0x22 offset=0x0 fd=-1
process_mmap(0xb6ea8000,0x00076000,0x22,---,mmap)
mmap 0xb6ea8000: anon
syscall: mmap2 addr=0xb6ea8000 size=0x65960 prot=0x5 flags=0x12 offset=0x0 fd=3
process_mmap(0xb6ea8000,0x00065960,0x12,r-x,mmap)
mmap 0xb6ea8000: partial
syscall: mmap2 addr=0xb6f0f000 size=0x553c prot=0x3 flags=0x12 offset=0x66 fd=3
process_mmap(0xb6f0f000,0x0000553c,0x12,rw-,mmap)
mmap 0xb6f0f000: partial
syscall: mmap2 addr=0xb6f15000 size=0x9000 prot=0x3 flags=0x32 offset=0x0 fd=-1
process_mmap(0xb6f15000,0x00009000,0x32,rw-,mmap)
mmap 0xb6f15000: anon
process_mmap(0xb6f1e000,0x00001000,0x22,rw-,mmap)
mmap 0xb6f1e000: anon

And removes the 1-page header:

syscall: munmap addr=0xb6f1e000 size=0x00001000

So it thinks the main mmap is partial.

More info:

b6f31000-b6f97000 r-xp 00000000 b3:16 956        /system/lib/libc.so
b6f97000-b6f98000 ---p 00000000 00:00 0 
b6f98000-b6f9e000 rw-p 00066000 b3:16 956        /system/lib/libc.so
b6f9e000-b6fa7000 rw-p 00000000 00:00 0 
b6fa7000-b6faa000 rw-p 00000000 00:00 0          [anon:linker_alloc]
<get_memory_info mismatch! (can happen if os combines entries in /proc/pid/maps)
        os says: 0xb6f98000-0xb6f9e000 prot=0x00000003
        cache says: 0xb6f98000-0xb6faa000 prot=0x00000003
>
module_is_partial_map: 0xb6f31000 size 0x65960 vs 0x154
module_is_partial_map: 0xb6f31000-0xb6f96960 vs seg 0x00000000-0x00076000
(gdb) p/x 0xb6f31000+0x76000
$2 = 0xb6fa7000

It's b/c its whole-image mmap is anon, and so the first file-backed one
with an ELF header is just the code segment.

On Linux the loader's 1st +rx mmap with file backing covers the entire file:

syscall: mmap2 addr=0x00000000 size=0x1cba7c prot=0x5 flags=0x802 offset=0x0 fd=3
process_mmap(0xef502000,0x001cba7c,0x802,r-x,mmap)
mmap 0xef502000: elf header
module_is_partial_map: 0xef502000 size 0x1cba7c vs 0x174
module_is_partial_map: 0xef502000-0xef6cda7c vs seg 0x00000000-0x001cc000
syscall: mmap2 addr=0xef6c8000 size=0x3000 prot=0x3 flags=0x812 offset=0x1c6 fd=3
process_mmap(0xef6c8000,0x00003000,0x812,rw-,mmap)
mmap 0xef6c8000: overlaps image
syscall: mmap2 addr=0xef6cb000 size=0x2a7c prot=0x3 flags=0x32 offset=0x0 fd=-1
process_mmap(0xef6cb000,0x00002a7c,0x32,rw-,mmap)
mmap 0xef6cb000: anon

All the UNIX DR code for handling libraries assumes the Linux model.

We have another problem: a fault trying to read the dynamic section.

module_fill_os_data() assumes the whole file is mapped in, but it's not, so
the dynamic section is off the end:

<CURIOSITY : (0) && "crashed while walking dynamic header" in file /work/dr/git/src/core/unix/module_elf.c line 456

> objdump -p ../../binaries/libc.so 
../../binaries/libc.so:     file format elf32-little
Program Header:
    PHDR off    0x00000034 vaddr 0x00000034 paddr 0x00000034 align 2**2
         filesz 0x00000120 memsz 0x00000120 flags r--
  INTERP off    0x00000154 vaddr 0x00000154 paddr 0x00000154 align 2**0
         filesz 0x00000013 memsz 0x00000013 flags r--
    LOAD off    0x00000000 vaddr 0x00000000 paddr 0x00000000 align 2**12
         filesz 0x00065960 memsz 0x00065960 flags r-x
    LOAD off    0x00066260 vaddr 0x00067260 paddr 0x00067260 align 2**12
         filesz 0x000052dc memsz 0x0000ed86 flags rw-
 DYNAMIC off    0x000683a0 vaddr 0x000693a0 paddr 0x000693a0 align 2**2
         filesz 0x000000d8 memsz 0x000000d8 flags rw-

1st segment size is 0x65960.

So do we:

A) Mmap the whole file ourselves?
B) Delay this data init until the segment containing .dynamic is loaded.
The client lib load event is delayed anyway. Will this mess up some
internal part of DR?

Plus, how are any app modules being added? Wasn't drcov working?

[derekbruening]

Plus, how are any app modules being added? Wasn't drcov working?

Because of #1760 where we check for a new lib on new exec in a new vm_area.

[derekbruening]

C) Delay adding the module until 1st exec, when i#1760 kicks in.
Use some other mechanism to avoid the merge: split data-backed from anon
in memcache?

[derekbruening]

D) Modify the loader's initial mmap to be file-backed instead of anon? It
may be fragile to identify though.

Interactions with pcaches are probably the primary concern.

[derekbruening]

0b58677 implemented solution B