Support Caliper based profiling #741
Conversation
|
Oh, I also normalized the commit messages with a |
|
This is looking like a good first cut to me. Possibly the commits could be massaged a bit to split the build system and hook additions, and squash incremental development, but if it's a pain or if we want to preserve @trws's line of thought to make it easier for him to review later, I"m OK with leaving it as is - it's just not that much change overall. Does this still require a non-released version of caliper to build? |
Yeah, I think I could try to do that and still preserve @trws's authorship. If not, I propose leaving as is (though I could squash my typo fix into his) I will test with other versions of Caliper, but AFAIK, it does require something recent from Caliper git master. I did get that building in Travis a little while ago so we could at least ensure the caliper support builds and works in Travis. |
Add an --enable-caliper option to ./configure to check for Caliper[1] support for use in profiling/tracing. Define caliper libs and flags accordingly. [1] https://github.com/LLNL/Caliper
Switch to the use of pkg-config for libcaliper, and link only to libcaliper-stub by default. Full tracing can be enabled by preloading the libcaliper full symbols with LD_PRELOAD=libcaliper.so
Add initial profiling support using libcaliper, creating a context for the broker main thread as well as module threads. Additionally, messages are traced via the dispatch code.
Unset CALI_SERVICES_ENABLE and CALI_CONFIG_PROFILE in flux-broker after profiling initialization to avoid tracing of child processes.
This update adds tracing for various properties of messages, RPCs, events, etc. This will not give us all we need to get matches for all messages, but it's on the way. As an FYI, be aware that the profiling_msg_snapshot routine can be called due to messaging activity inside connector_init, so the profiling object is not ready, and potentially cause uninitialized nodes to be emitted. This is currently addressed by a check and return in the snapshot routine, but if at some point those code-paths become threaded something fancier may be required.
Abstract broker and module caliper initialization into their own functions.
Add --caliper-profile=PROFILE option to flux-start, which causes LD_PRELOAD=libcaliper.so and CALI_CONFIG_PROFILE=$PROFILE to be set in the environment before invoking brokers. If Flux was not built with --enable-caliper, then this option issues an error. Unless already set in environment, CALI_LOG_VERBOSITY=0 is also set by default to quiet the Caliper logs to stderr.
In flux-start(1), document the --caliper-profile option
|
Ok, I reworked this PR by removing some of the intermediate state and splitting the build changes from the actual code. Some commits were left separate to preserve authorship. @garlick, let me know if this one is better. |
|
This does look good. Shall we merge? |
|
Sure -- I'm working on some related testing changes which could be a future PR. |
|
OK, thanks! |
|
Thanks guys! Sorry I dropped off, Sunday was the day. =) Sent from Boxer | http://getboxer.comhttp://bxr.io/PBI3C On August 3, 2016 at 1:02:31 PM PDT, Jim Garlick notifications@github.com wrote: You are receiving this because you were mentioned. |
|
Congratulations, Tom! Tapasya showed us the picture of Baby Kimberly and she is beautiful. We are all very excited for you and Britany and not the least worried that you dropped off flux threads for a while. Take your time and enjoy this special family time. When you do return please bring pictures! Becky Sent with Good (www.good.com) From: Tom Scogland Thanks guys! Sorry I dropped off, Sunday was the day. =) Sent from Boxer | http://getboxer.comhttp://bxr.io/PBI3C On August 3, 2016 at 1:02:31 PM PDT, Jim Garlick notifications@github.com wrote: You are receiving this because you were mentioned. — |
|
I certainly will, thanks very much Becky! -Tom On August 6, 2016 at 9:42:36 AM PDT, Becky Springmeyer notifications@github.com wrote: Becky Sent with Good (www.good.com) From: Tom Scogland Thanks guys! Sorry I dropped off, Sunday was the day. =) Sent from Boxer | http://getboxer.comhttp://bxr.io/PBI3C On August 3, 2016 at 1:02:31 PM PDT, Jim Garlick notifications@github.com wrote: You are receiving this because you were mentioned. You are receiving this because you are subscribed to this thread. You are receiving this because you were mentioned. |
This PR is a continuation of #672, which extends the work of @garlick and @trws with
pkg-configargument when building withlibcaliper-stub--caliper-profile=ARGtoflux start, which exports properLD_PRELOAD,CALI_CONFIG_PROFILE=ARGandCALI_LOG_VERBOSITY=0to brokers (CALI_*vars are not overwritten)Also, when I was working another angle I abstracted the caliper init routines for broker and modules into their own functions. I left that in here, but could easily remove it as it isn't necessary right now (thought I could get runtime initialization working)
I also realize I need to document the
--caliper-profileoption to flux-start.