early and comprehensive injection on Linux

[derekbruening]

From derek.br...@gmail.com on February 24, 2009 11:25:45

for LD_PRELOAD we'll start by using -z initfirst. for that we need libc
independence ( issue #48 /PR 206369) and to directly read our env vars off the
stack.
we should also directly read the elf aux vector (PR 289138).

xref issue #37 /PR 248204: ptrace injection

Original issue: http://code.google.com/p/dynamorio/issues/detail?id=47

[derekbruening]

From derek.br...@gmail.com on February 24, 2009 08:28:33

this was PR 204554

[derekbruening]

From qin.zhao@gmail.com on March 23, 2009 09:42:21

If we want to have an early injection in Linux, vdso might not being present at init
time, so process_mmap should have code for handling vdso load.

[derekbruening]

From derek.br...@gmail.com on March 23, 2009 09:44:43

expanding on prev comment: xref issue #89 where vdso is now inside ld.so

[derekbruening]

From derek.br...@gmail.com on April 02, 2009 13:01:52

being loaded before libc should also let us preempt libc global symbols to provide
our own versions for clients (xref issue #30 )

[derekbruening]

From bruen...@google.com on November 23, 2011 07:00:09

xref a user hitting a conflict with DR's preferred base when running firefox

[derekbruening]

From rnk@google.com on April 22, 2012 17:12:33

I started using ptrace to accomplish this over the weekend because I thought it would be fun. I was right! =D

We should talk about some of the design points in our Monday meeting. Here's sort of a summary of what I'm doing.

I have a new drdeploy_inject binary on Linux (to be renamed later when it's the default injection method) that implements parts of the dr_inject.h API using a typical fork, PTRACE_TRACEME, exec(app, ...) pattern. Currently I'm just linking this code into libdynamorio.so and initializing DR as a standalone library, since that's easier. drdeploy_inject just links against libdyanamorio.so and implements the typical dr_inject_* API call pattern.

When we cross execve, we're at _start. At this point, I want to perform a bunch of syscalls to set up DR in the injectee's address space. I do this by generating PIC ilists in the injector, encoding them, and using ptrace to copy them over the injectee's current code. I terminate it with an int3, and set it running. When I hit the int3, I copy xax back to the injector, and put back the registers and memory to the original state.

At first, I was just doing two raw syscalls:

• dr_fd = open("libdynamorio.so", O_RDONLY, 0)
• dr_base = mmap(PREFERRED_BASE, ... RWX, fd, 0)

After that, I can set XIP to (fn_ptr - get_dynamorio_dll_base()) + dr_base and let it execute in the injectee, without going through the syscall ilist generation and injection.

I was hoping that would let me run enough code to initialize DynamoRIO and call into the private loader to finish the job, but it wasn't.

Currently I am adding indirection to the Linux (only Linux!) private loader to hook the mmap/munmap/mprotect syscalls it issues and translate them into injected syscalls.

This has worked fairly well and I can execute more code, but anything depending on libc still doesn't work. In particular, std_->_fileno in STDOUT broke, so I replaced it with STD__FILENO from unistd.h, which has the typical in->0, out->1, err->2 mapping.

I still can't get dynamorio_app_init() to succeed yet, so I'll look into it and possibly round out more private loader support for injection.

Based on this work, it should be easy to solve external and internal attaching on Linux, it's just a question of where and how we ptrace.

Status: Started
Owner: rnk@google.com

[derekbruening]

From bruen...@google.com on April 23, 2012 07:23:48

first we need to decide whether we ever want to rely on ptrace. if so, the first use of ptrace should be issue #37 where we use ptrace plus the regular loader and don't need to implement early injection just yet which requires processing imports w/o using any imports. there are other ways to do that that we should consider. we'll discuss offline.

[derekbruening]

From bruen...@google.com on July 11, 2012 17:20:49

we would like to support running DR on static executables, which the injector this case represents would solve

[derekbruening]

From bruen...@google.com on July 11, 2012 17:22:57

xref issue #840

[derekbruening]

From zhao...@google.com on July 11, 2012 17:34:30

Owner: zhao...@google.com

[derekbruening]

From bruen...@google.com on July 12, 2012 07:37:12

changed title to take into account that this is no longer a tweak to be earlier (via -z initfirst, though note that only the final LD_PRELOAD lib w/ that flag will be the first) but represents creating an injection method that works on static executables as well as taking over from the first app instruction.

Summary: early and comprehensive injection on Linux

[derekbruening]

From bruen...@google.com on July 13, 2012 06:37:46

for transparency in the application name that's displayed by various tools:

sprintf(argv[0], "/tmp/x");
getchar();

% ./original arg1 arg2

% cat /proc/pgrep original/cmdline
/tmp/xnalarg1arg2

% ps ax | grep tmp
39 ? S 0:00 [kdevtmpfs]
14787 pts/3 S+ 0:00 /tmp/x nal arg1 arg2
14817 pts/2 S+ 0:00 grep -d skip tmp

% pgrep original
14787

So pgrep is getting the real name from somewhere, but the regular ps
listing and /proc/pid/cmdline are both getting it from the arg data on the
stack (not the argv pointers but what they start out pointing at).

The original name is in /proc/self/comm and /proc/self/status and others:

for i in ls -1d /proc/14787/* | grep -v mem; do grep -q original $i && (echo -e "\n$i"; grep original $i;); done
grep: /proc/14787/clear_refs: Permission denied

/proc/14787/comm
original

/proc/14787/environ
Binary file /proc/14787/environ matches

/proc/14787/maps
08048000-08049000 r-xp 00000000 fd:09 2119216 /work/dr/test/original
08049000-0804a000 rw-p 00000000 fd:09 2119216 /work/dr/test/original

/proc/14787/numa_maps
08048000 default file=/work/dr/test/original mapped=1 N0=1
08049000 default file=/work/dr/test/original anon=1 dirty=1 N0=1

/proc/14787/sched
original (14787, #threads: 1)

/proc/14787/smaps
08048000-08049000 r-xp 00000000 fd:09 2119216 /work/dr/test/original
08049000-0804a000 rw-p 00000000 fd:09 2119216 /work/dr/test/original

/proc/14787/stat
14787 (original) S 2716 14787 2716 34819 14787 4194304 140 0 0 0 0 0 0 0 20 0 1 0 64330625 2072576 65 18446744073709551615 134512640 134514700 4294658880 4294658488 4152116272 0 0 0 0 18446744071582421497 0 0 17 7 0 0 0 0 0 134518796 134519060 139001856

/proc/14787/status
Name: original

strings /proc/14787/environ | grep original
_=./original

[derekbruening]

From bruen...@google.com on July 13, 2012 06:38:50

strace pgrep original 2>&1 | grep /proc/14787
stat("/proc/14787", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
open("/proc/14787/status", O_RDONLY) = 4
open("/proc/14787/cmdline", O_RDONLY) = 4
read(4, "grep\0-d\0skip\0/proc/14787\0", 2047) = 25

[derekbruening]

From bruen...@google.com on July 13, 2012 06:44:59

*** TODO proposal for transparency

1. Rename file in some temp dir prior to loading it, to get the right
   basename for /proc/self/{comm,sched,stat,status}

2. Change the first arg string to get the right path and filename
   for the app itself and for /proc/self/cmdline

3. Change $_ in the env string to get the right path for
   /proc/self/environ and for the app

The one thing we will not easily change is the full path in
/proc/self/*maps. We could intercept the app reading from that file though
we can't do much about other processes.

[derekbruening]

From rnk@google.com on July 13, 2012 07:01:58

I tried changing the "_" environment variable, and that was not reflected in /proc/pid/status either. I thought maybe the kernel was getting it from there instead of argv[0]. It must store it somewhere.

[derekbruening]

From rnk@google.com on July 13, 2012 07:06:32

Oh, hello: http://www.kernel.org/doc/man-pages/online/pages/man2/prctl.2.html Look at PR_SET_NAME. Available since Linux 2.6.9. I bet that does what we want.

[derekbruening]

From bruen...@google.com on July 13, 2012 08:48:09

that just sets a name, not a path, so it doesn't seem like it solves the remaining issue of the mapped path

[derekbruening]

From rnk@google.com on July 13, 2012 09:38:36

I'm OK with libdynamorio.so showing up in /proc/pid/maps, because most people use that for debugging. I just want to make sure that "killall myapp" still works. Not sure if killall uses the kernel name or the argv[0] name.

[derekbruening]

From bruen...@google.com on July 13, 2012 10:00:21

PR_SET_NAME can replace step 1) in comment 14. the next 2 steps are still required, and the paths in the maps remain. for PR_SET_NAME xref drmem's -prctl_whitelist option

[derekbruening]

From rnk@google.com on July 15, 2012 12:10:35

Qin, how far along are you on this? This was something I looked into using ptrace for a bit in April, but then we backed off because we didn't want to use ptrace. I think this is a fun project, so we both want to do it. :) OTOH, it would be silly to duplicate efforts.

I spent a few hours looking into the approach I outlined in my email:
"""

• finish issue libc independence on Linux #46
• define _start in libdynamorio.so
• remove the elf interp bits from libdynamorio.so
• now libdynamorio.so is both an exe and so, just like ld.so
• the injector is libdynamrio.so
• the kernel loads us at our preferred address (is that possible, or is a completely static binary loaded at a fixed address?)
• we get argc, argv, and env from _start, and change it to match the app
• initialize
• map in ld.so to load app
• set mcontext pointing at ld.so _start with new argc, argv, and env
  """

To "finish" libc independence I used the stdio isolation patch I sent out and ifdef'd out the rest of our libc usage (it's mostly in stackdump.c, the private loader, and the environment).

I added "-static -e _start" to our link flags and took out -lc -lm -ldl. This gets rid of the PT_INTERP phdr and sets e_entry to point at _start. I defined _start in x86.asm to mov xsp into ARG1 and call a C routine.

At this point, I could dump argc, argv, envp as expected, but the PLT and GOT have not been set up, so I could not make any references to exported functions or globals. Global data exports were actually really painful, because our_std* is exported. A lot of our functions have internal and external names, but some of them like the reg_* helpers do not.

IMO it would be best if we could find some linker flag to avoid the GOT for internal references to exported data, but I couldn't find one. I thought -Bsymbolic was supposed to do this for functions, but it seems to still rely on the GOT/PLT.

[derekbruening]

From rnk@google.com on July 15, 2012 12:16:41

I forgot to mention: If we can't find such a flag, that means we'll probably need a very specialized bootstrapping loader, like the one for Windows.

[derekbruening]

From zhao...@google.com on July 15, 2012 21:33:15

A few more notes

• /proc/pid/cmdline is control/changed by memory pointed by argv[0] and after. change of pointer argv[0] or argv does not change cmdline,
• /proc/pid/stat and /proc/pid/status is not affected by argv changes
• prctl PR_GET_NAME will get app name not from argv
  prctl PR_SET_NAME will change value at /proc/pid/stat and /proc/pid/status, which is read by ps, pgrep, and killall

my suggestion would be:
original app cmd: app arg1 arg2
actual cmd: drrun -ops "..." -- drinjector app arg1 arg2

By doing so app becomes the first arg1, then we can change argc and either change argv or shift the argv on stack, whichever is simpler.

For this issue, I think a simple drinjector would be simpler than converting libdynamorio.so into lib/exec, which is more for attach.

[derekbruening]

From rnk@google.com on July 16, 2012 06:04:24

Last night I got to the point where libdynamorio.so could be used both as a lib and an exe, so that's more or less already done. I got our loader to relocate DR, but then I didn't need it! I tweaked the ld flags a bit and all the GOT references ended up getting resolved internally.

I called dynamorio_app_init() from start and it mostly works, except for the parts that try to interpret libc's TLS segment, which makes sense.

---

w.r.t. the app name and command line, the execve syscall takes a filename and a full argv[] array. I wonder what happens if we pass the filename of libdynamorio.so, and the original argv[]? Maybe that will fix the command line in a simpler way.

We can write a simple exe in C that just calls execve("libdynamorio.so", argv, envp) to test that.

[derekbruening]

From bruen...@google.com on July 16, 2012 06:33:23

We should also think about MacOS and other *nix ports: a more general injector that uses fewer Linux-specific or even ELF-specific features will save effort later. Although in some cases things are just too different to share.

[derekbruening]

From rnk@google.com on August 10, 2012 12:50:10

When launching a process under DR with early injection from gdb, I found that it disables ASLR. This actually has weird consequences for ET_DYN ELFs, and DR gets mapped at 0x555555554000. Things crash soon afterwards, because the kernel does not perform relocations on the GOT, which is filled out by default to assume we were loaded at our preferred base.

If you always debug using attach, then this won't happen, but if you start the app under gdb this will come up. The solution is to run 'set disable-randomization off'. When ELF early injection is more prevalent, this should go into our .gdbinit file and HowToDebug.

Here's the kernel source code that decides where to load ELFs:
linux/fs/binfmt_elf.c:783:

        vaddr = elf_ppnt->p_vaddr;
        if (loc->elf_ex.e_type == ET_EXEC || load_addr_set) {
            elf_flags |= MAP_FIXED;
        } else if (loc->elf_ex.e_type == ET_DYN) {
            /* Try and get dynamic programs out of the way of the
             * default mmap base, as well as whatever program they
             * might try to exec.  This is because the brk will
             * follow the loader, and is not movable.  */
#ifdef CONFIG_ARCH_BINFMT_ELF_RANDOMIZE_PIE
            /* Memory randomization might have been switched off
             * in runtime via sysctl.
             * If that is the case, retain the original non-zero
             * load_bias value in order to establish proper
             * non-randomized mappings.
             */
            if (current->flags & PF_RANDOMIZE)
                load_bias = 0;
            else
                load_bias = ELF_PAGESTART(ELF_ET_DYN_BASE - vaddr);
#else
            load_bias = ELF_PAGESTART(ELF_ET_DYN_BASE - vaddr);
#endif
        }

        error = elf_map(bprm->file, load_bias + vaddr, elf_ppnt,
                elf_prot, elf_flags, 0);

[derekbruening]

From bruen...@google.com on September 05, 2012 22:49:41

reminder to implement control maintenance across execve (and augment suite to detect, ideally)

[derekbruening]

From rnk@google.com on September 11, 2012 14:54:18

Another proc file that we haven't mentioned yet: /proc/self/exe

This is a symlink to the filename passed to exec. I don't think there are any good ways to fake this. We may need to have some special casing around syscalls. For example, I believe Chrome calls exec on /proc/self/exe.

Might be worth splitting this and early follow children as separate issues.

[derekbruening]

From zhao...@google.com on October 25, 2012 07:56:24

Owner: rnk@google.com

[derekbruening]

From zhao...@google.com on December 28, 2013 02:48:23

I just investigated the possibilities of using -z initfirst.
$ readelf -d ~/Workspace/DynamoRIO/builds/build_x64_dbg.git/lib64/debug/libdrpreload.so

Dynamic section at offset 0x3e38 contains 23 entries:
Tag Type Name/Value
0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
0x000000000000000e (SONAME) Library soname: [libdrpreload.so]
...
0x000000006ffffffb (FLAGS_1) Flags: INITFIRST
0x000000006ffffffe (VERNEED) 0x518

It won't work with normal program, because the loader only honors one library for initfirst.
code from _dl_map_object_from_fd
/* Remember whether this object must be initialized first. */
if (l->l_flags_1 & DF_1_INITFIRST)
GL(dl_initfirst) = l;

If there are more than one libraries has flag initfirst, the last one will be remembered and called first on initialization.
In my test, libdrpreload will first be loaded due to LD_PRELOAD and pointed by GL(dl_initfirst), later libpthread will be loaded, and GL(dl_initfirst) is replaced with pthread library instead.

$ readelf -d /lib/x86_64-linux-gnu/libpthread-2.17.so

Dynamic section at offset 0x17d50 contains 31 entries:
Tag Type Name/Value
0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
0x0000000000000001 (NEEDED) Shared library: [ld-linux-x86-64.so.2]
...
0x000000000000001e (FLAGS) STATIC_TLS
0x000000006ffffffb (FLAGS_1) Flags: NODELETE INITFIRST

later in call_init, pthread's init will be called first, and no other libraries' initfirst flags are remembered.
(gdb) x/i $pc
=> 0x7f069a26480e <call_init+46>: callq *%rax
(gdb) x/4i $rax
0x7f0698cf1360 <_init>: sub $0x8,%rsp
0x7f0698cf1364 <_init+4>: callq 0x7f0698cf2920 <__pthread_initialize_minimal_internal>
0x7f0698cf1369 <_init+9>: add $0x8,%rsp
0x7f0698cf136d <_init+13>: retq

Owner: zhao...@google.com

[derekbruening]

From bruen...@google.com on December 28, 2013 07:07:31

For the record, this is in comment #11 above: "note that only the final LD_PRELOAD lib w/ that flag will be the first". Though for today's single-app-target DR usage, we don't need libdrpreload.

pthread having initfirst must be new, as it was not the case when we were investigating this in the past.

[derekbruening]

I'm making a push to get this on by default. Xref recently completed #909, #907, #1227, #1004.