-
Notifications
You must be signed in to change notification settings - Fork 52
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Build Profiling, main branch (2024.09.22.) #713
Build Profiling, main branch (2024.09.22.) #713
Conversation
e6f5acb
to
de1473c
Compare
@krasznaa just to point it out again: I literally wrote a tool that does this exact thing for you given a compilation database. It does structured printout and gives you nice csv. I would encourage you to at least give it a try before reimplementing it. |
I did have a look Paul. But that one still doesn't do any linking steps, does it? And unfortunately our build can be expensive during linking as well. As you can see, this PR is literally just a couple of lines long. So collecting the information is really not the complicated part here. But if we can re-use your code's logic for organizing the information, that would be very useful indeed. |
When using CTEST_USE_LAUNCHERS=TRUE, the build now sends each command through a specific script, which would profile those commands using the "time" executable. Saving all collected output into a file called "traccc_build_performance.log" in the build directory.
de1473c
to
1d6f7b8
Compare
Co-authored-by: Stephen Nicholas Swatman <stephenswat@gmail.com>
Also added a warning in case somebody tries to use the build profiling on something other than Linux or macOS.
1c9f666
to
1ac8524
Compare
|
As we've been discussing between a few of us, the build of the project is getting a bit out of hand by now. 😦 Building generally takes a long time, and also a surprisingly large amount of memory.
To help with this, first we need to understand exactly which steps of the build take the longest time and the largest amount of memory. To do this, I hijacked the CTEST_USE_LAUNCHERS feature of ctest. Which is the technique we use also in AtlasCMake for saving "package specific" build logs in ATLAS offline builds.
What happens is that if one specifies
-DCTEST_USE_LAUNCHERS=TRUE
in the CMake configuration command, the newly introducedtraccc-ctest.sh
script gets set up to intercept each and every build command. Including all linking, and any other technical commands. The script then runs the commands through GNU time, to get detailed statistics about every build command. It saves the the results of this into a file calledtraccc_build_performance.log
in the build directory. Which would have entries like:Leading up to some of the really "heavy" commands, like:
This PR just writes such a log file, we will still need to write a bit of scripting to extract useful information out of these log files. I imagine that it would be relatively doable to produce let's say a CSV file from this log file with the info we're interested in, and then we could use even something as simple as a spreadsheet application to understand where we need to concentrate our efforts. 🤔