-
Notifications
You must be signed in to change notification settings - Fork 27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Question: how to avoid hang during library instrumentation #177
Comments
omnitrace -v -1 --print-instrumented functions -o /ccs/home/jjhu/crusher/libs-instrumented/libteuchosnumerics.so.13 This doesn’t look right. You need to provide the library you want to instrument after a double-hyphen: omnitrace -v -1 --print-instrumented functions -o /ccs/home/jjhu/crusher/libs-instrumented/libteuchosnumerics.so.13 -- /path/to/original/lib I'm surprised it hung instead of throwing an error that you didn't provide a exe/lib. Binary rewrites don't take much time (couple seconds to a minute), you shouldn't need to run them in a job. Also, you may want to try the omnitrace-sample <options> -- <command-to-run> |
@jrmadsen Cut and paste error on my side. My command indeed looks like what you said it should:
|
Ah I wondered if that might be the case. Can you try running it interactively and when you think it has hung, hit control-C. I put a new feature in 1.7 that should print out some log messages about what it was doing when it fails or gets interrupted |
Oh yeah, I actually have those handy :). Here you go. (I'm not sure how long it ran before I broke it, but I can let it run for however long you think if it would produce better logging information.) log
|
Here's a log after I let omnitrace run for about 6 minutes:
|
Hi @jhux2, thanks for providing the second log. Yeah, it looks like Dyninst is hanging in the exact same spot both times. Let me look into this and get back to you in a couple hours. |
Hi @jhux2 if you install the (newly released) v1.7.1 and interrupt the job as before, there should be line info in the backtrace -- knowing the line number it is hanging on will (hopefully) significantly help me track down the reason for the hang. |
@jrmadsen Here's a backtrace from v1.7.1.
|
There are performance and other issues noted in these which help explain what is going on here |
@jrmadsen Thanks for the information. |
@jhux2 Has your issue been resolved? If so, please close the ticket. Thanks! |
When I attempt to instrument a particular library in the Trilinos project, the process doesn't finish, even running overnight.
This is with omnitrace release 1.7 on crusher. The library in question is
libteuchosnumerics.so.13
and the command isThe documentation presents a few options -- are there any that you'd recommend that I try?
I'm using Trilinos develop
a76c1c4a9
, and my module environment isThe text was updated successfully, but these errors were encountered: