Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dtrace probes #875

Open
steveklabnik opened this issue Feb 16, 2015 · 10 comments
Open

dtrace probes #875

steveklabnik opened this issue Feb 16, 2015 · 10 comments
Labels
T-dev-tools Relevant to the development tools team, which will review and decide on the RFC.

Comments

@steveklabnik
Copy link
Member

Issue by graydon
Wednesday May 29, 2013 at 23:45 GMT

For earlier discussion, see rust-lang/rust#6816

This issue was labelled with: A-instrumentation, A-libs, I-enhancement, P-low in the Rust repository


I think rust should expose dtrace userland probes

Links:
http://www.solarisinternals.com/wiki/index.php/DTrace_Topics_USDT
https://wikis.oracle.com/display/DTrace/Statically+Defined+Tracing+for+User+Applications
https://wiki.freebsd.org/DTrace/userland

Some specific providers (including some dynamic):
https://github.com/chrisa/libusdt
https://github.com/chrisa/node-dtrace-provider
http://prefetch.net/projects/apache_modtrace/index.html
https://bugzilla.mozilla.org/show_bug.cgi?id=370906

Probably at least some for pervasive / standard-library actions:

  • per-task CPU time, coroutine switches, lifecycle events
  • heap allocations, GC, segmented stack growth
  • message passing, IO ops / byte volumes
  • conditions raised

Note that the "USDT" mechanism provides static probes whereas things like libusdt (above) provide an API for dynamically registering probes at runtime. This is probably important, possibly even a good place to start, though it's slightly more expensive; ideally of course we should be able to provide both.

See also #6810 concerning metrics. I think there's some potential design-informing between the two. Possibly libusdt as a reporting sink for metrics?

@alexcrichton alexcrichton added T-dev-tools Relevant to the development tools team, which will review and decide on the RFC. and removed T-dev-tools Relevant to the development tools team, which will review and decide on the RFC. labels May 18, 2015
@tromey
Copy link

tromey commented Apr 1, 2016

Linux has its own form of static probe: https://sourceware.org/systemtap/wiki/UserSpaceProbeImplementation. They are very low cost and various tools, like gdb and systemtap, already know how to access them.

@cuviper
Copy link
Member

cuviper commented Apr 1, 2016

@tromey see my probe crate for SystemTap SDT support, and in theory it could support implementations for other platforms too. It needs asm! though, which makes it nightly-only.

@Fiedzia
Copy link

Fiedzia commented Oct 24, 2018

Its been a while, I looked at it recently and I can't see any way to get it working on stable.
My attempts is here: https://github.com/Fiedzia/rust-usdt. It works, but only on nightly due to being compiler plugin and using asm, and even nightly removed some features I had before (access to mir pass). The only way for now I'd see would be to integrate it with compiler.

@jonhoo
Copy link
Contributor

jonhoo commented Mar 4, 2019

With the up-and-coming support for eBPF-based program tracing using tools like bpftrace, it would be really cool to see this being picked up again. From this article, it looks like any kind of USDT probe would be immediately recognizable by eBPF-based tools. @cuviper can you comment any more on the state of libprobe since rust-lang/rust#14031?

@cuviper
Copy link
Member

cuviper commented Mar 4, 2019

@jonhoo AFAIK my probe crate still works, but only on nightly for inline asm (rust-lang/rust#29722). I don't know how to implement this otherwise, but rumor from the recent all-hands meeting is that asm might now have a path toward stabilization.

@jonhoo
Copy link
Contributor

jonhoo commented Mar 4, 2019

I wonder whether libprobe also works on Linux with eBPF now... I suspect it should. Have you tried that at all?

@cuviper
Copy link
Member

cuviper commented Mar 4, 2019

I haven't tried with eBPF, but it ought to work -- I implemented it the same way as sys/sdt.h.

@jonhoo
Copy link
Contributor

jonhoo commented Mar 5, 2019

I can confirm that libprobe works with eBPF. Specifically, I added

#![feature(asm)]
#[macro_use]
#[no_link]
extern crate probe;

to inferno here. I then added annotations here:

if line.is_empty() {
    probe!(inferno_collapse_perf, post_stack);
    self.after_event();
} else {
    probe!(inferno_collapse_perf, in_stack);
    self.on_line(line.trim_end());
}

This then let me do this:

$ sudo bpftrace -e "usdt:$CARGO_TARGET_DIR/release/inferno-collapse-perf:inferno_collapse_perf:post_stack /@start[pid]/ { @ns[comm]= hist(nsecs - @start[pid]); delete(@start[pid]); } usdt:$CARGO_TARGET_DIR/release/inferno-collapse-perf:inferno_collapse_perf:post_stack { @start[pid] = nsecs; }"
Attaching 2 probes...
^C

@ns[inferno-collaps]:
[1K, 2K)           24014 |@@@@                                                |
[2K, 4K)          108620 |@@@@@@@@@@@@@@@@@@                                  |
[4K, 8K)          129485 |@@@@@@@@@@@@@@@@@@@@@                               |
[8K, 16K)         310007 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[16K, 32K)        244627 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@           |
[32K, 64K)         69413 |@@@@@@@@@@@                                         |
[64K, 128K)        21566 |@@@                                                 |
[128K, 256K)         152 |                                                    |
[256K, 512K)           1 |                                                    |
[512K, 1M)             2 |                                                    |

That's really neat!

The command is a little awkward since bpftrace requires that the name of the binary matches the program name (bpftrace/bpftrace#328 (comment)), and Rust identifiers cannot have - in them. We get around that by giving an explicit application name following bpftrace/bpftrace#413.

@jonhoo
Copy link
Contributor

jonhoo commented Mar 6, 2019

Also note that with bpftrace/bpftrace#444 (which is really bpftrace/bpftrace#280), the above can (?) be simplified greatly to:

$ sudo bpftrace -p $(pgrep inferno-collapse-perf) -e "\
usdt::post_stack /@start[pid]/ { @ns[comm]= hist(nsecs - @start[pid]); delete(@start[pid]); }\
usdt::post_stack { @start[pid] = nsecs; }"

@dalehamel
Copy link

Weird to come across an issue, linking my own pull requests. I'd love to see support for dtrace probes in Rust!

Should also check out libstapsdt, which can be used for dynamic support.

I'm not as familiar with rust as I am other runtimes, but it would be great to at least have an --enable-dtrace option when building Rust, offering functionality similar to the tools Sasha Goldshtein put together, (eg, ucalls.py to support "rustcalls.sh" sort of things).

I'm planning on doing a lot more eBPF USDT work in the next few months, I'd love to help however I can.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
T-dev-tools Relevant to the development tools team, which will review and decide on the RFC.
Projects
None yet
Development

No branches or pull requests

8 participants