When attaching with --pid=, shared libraries are not covered

[VincentGibert]

Hi (again),

When I use kcov with the --pid flag, I don't manage to get coverage results.

Environment

• Operating system: CentOS 6.6 x86_64
• kcov version: git snapshot from revision 157d013 which fixes Coverage collection from shared libraries with only build-id debug info does not work #104.
• Processus executed: /usr/sbin/snmpd from net-snmp-5.5 compiled using -g -O0 (snmpd binary is not stripped, debug symbols are not in a separate build id).

Steps to reproduce

1. Launch the snmpd daemon

   $ /usr/sbin/snmpd
   

2. Attach kcov to the running snmpd instance

   $ ps -ef | grep snmpd
   root      8307      1  0 16:03 ?        00:00:00 /usr/sbin/snmpd
   $ kcov -p 8307 /tmp/kcov_results /usr/sbin/snmpd &
   

3. Make sure snmpd does something and kill it

   $ snmpwalk -v 2c -c public localhost # make sure snmpd does something
   $ kill 8307
   

4. Coverage results generated by kcov are void

   $ ls /tmp/kcov_results/snmpd
   amber.png  bcov.css  cobertura.xml  coverage.db  data  glass.png  index.html  index.json  metadata  summary.db
   

Additional information

• If the snmp daemon is executed directly by kcov, coverage results are correctly generated.

  $ kcov /tmp/kcov_results /usr/sbin/snmpd &
  $ pkill snmpd
  kcov: Process exited with signal 15 (SIGTERM) at 0x7ffff77d4d4d
  $ ls /tmp/kcov_results/snmpd/ # 628 results
  agent_handler.c.df3bdac7.html agent_handler.c.df3bdac7.json agent_index.c.4976cbcb.html agent_index.c.4976cbcb.json [...] snmp.c.1eaa0f6.html snmp.c.1eaa0f6.json ...
  

• This issue also occurs when snmpd is launched using -f flag which prevents snmpd from forking.

• When running kcov with --debug=15, the following message appears:

  $ kcov -p 8283 --debug=15  /tmp/kcov_results /usr/sbin/snmpd
  PT stopped PID 8283 0x0000137f
  PT BP at 0x7f15c61e8432:-1 for 8283
  Can't find breakpoint at 0x7f15c61e8432
  PT continuing 8283 with signal 0
  

What does it mean ? Maybe some debug information are missing on my system ?

[SimonKagstrom]

Sorry, I've completely missed this bug report.

It might have something to do with shared library support I think, which perhaps isn't working well together with the --pid= option. Shared libraries are handled via a LD_PRELOAD override, but for an already started process, the LD_PRELOAD wasn't set at startup time. Therefore kcov doesn't detect the loaded libraries when it starts.

You should still get hits from snmpd itself, but not from the other libraries loaded by it.

[VincentGibert]

It makes sense, I will try with a static binary to confirm your analysis.

[VincentGibert]

I just tested with a static version of busybox and it worked well so it seems you guessed right.

[SimonKagstrom]

Great, thanks for testing!

I think I'll resolve this issue by adding a few lines to the documentation - I don't think it will be very easy to fix the core issue.

[VincentGibert]

You may probably retrieve the list of loaded shared libraries by the process through /proc//maps (sadly it is Linux specific) but I don't know if it is sufficient to trace them.

[SimonKagstrom]

Yes, maybe that's a possibility. I'll keep the bug report open for further investigation.

[SimonKagstrom]

I took a look at this bug again, and I believe it should be doable to parse /proc//maps for the shared library-when-tracing-via-pid case.

It will still not cover all cases since it will be difficult to detect dlopen():ing shared libraries this way (which is done via the LD_PRELOAD stuff above as well).

Anyway, it will be Linux-only, but the shared library handling depends on dl_iterate_phdr() as it is today, so it's already Linux-specific.