Build Profiling, main branch (2024.09.22.) #713
Conversation
krasznaa
force-pushed
the
BuildProfiling-main-20240922
branch
from
e6f5acb to
de1473c
Compare
|
@krasznaa just to point it out again: I literally wrote a tool that does this exact thing for you given a compilation database. It does structured printout and gives you nice csv. I would encourage you to at least give it a try before reimplementing it. |
|
I did have a look Paul. But that one still doesn't do any linking steps, does it? And unfortunately our build can be expensive during linking as well. As you can see, this PR is literally just a couple of lines long. So collecting the information is really not the complicated part here. But if we can re-use your code's logic for organizing the information, that would be very useful indeed. |
When using CTEST_USE_LAUNCHERS=TRUE, the build now sends each command through a specific script, which would profile those commands using the "time" executable. Saving all collected output into a file called "traccc_build_performance.log" in the build directory.
krasznaa
force-pushed
the
BuildProfiling-main-20240922
branch
from
de1473c to
1d6f7b8
Compare
Co-authored-by: Stephen Nicholas Swatman <stephenswat@gmail.com>
Also added a warning in case somebody tries to use the build profiling on something other than Linux or macOS.
krasznaa
force-pushed
the
BuildProfiling-main-20240922
branch
from
1c9f666 to
1ac8524
Compare
|
As we've been discussing between a few of us, the build of the project is getting a bit out of hand by now. 😦 Building generally takes a long time, and also a surprisingly large amount of memory.
To help with this, first we need to understand exactly which steps of the build take the longest time and the largest amount of memory. To do this, I hijacked the CTEST_USE_LAUNCHERS feature of ctest. Which is the technique we use also in AtlasCMake for saving "package specific" build logs in ATLAS offline builds.
What happens is that if one specifies
-DCTEST_USE_LAUNCHERS=TRUEin the CMake configuration command, the newly introducedtraccc-ctest.shscript gets set up to intercept each and every build command. Including all linking, and any other technical commands. The script then runs the commands through GNU time, to get detailed statistics about every build command. It saves the the results of this into a file calledtraccc_build_performance.login the build directory. Which would have entries like:Leading up to some of the really "heavy" commands, like:
This PR just writes such a log file, we will still need to write a bit of scripting to extract useful information out of these log files. I imagine that it would be relatively doable to produce let's say a CSV file from this log file with the info we're interested in, and then we could use even something as simple as a spreadsheet application to understand where we need to concentrate our efforts. 🤔