Fix: CUDA Host-Side -O3 with CMake #2078
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ax3l
added
bug
`-O3` was only set for CUDA if we ran CMake at least twice. This was an intialization logic bug and should be set from the beginning, but we initialized CUDA too late / applied the transform too early. We now change the default build type to `Release` to avoid changing defaults and avoid further surprises.
WeiqunZhang
approved these changes
Jul 12, 2021
ax3l
commented
Jul 12, 2021
| list(TRANSFORM CMAKE_C_FLAGS_RELWITHDEBINFO REPLACE "-O2" "-O3") | ||
| list(TRANSFORM CMAKE_CXX_FLAGS_RELWITHDEBINFO REPLACE "-O2" "-O3") | ||
| # FIXME: due to the "AMReX inits CUDA first" logic we will first see this with -O2 in output | ||
| list(TRANSFORM CMAKE_CUDA_FLAGS_RELWITHDEBINFO REPLACE "-O2" "-O3") |
There was a problem hiding this comment.
This line was called before we set enable_language(cuda).
The result was an empty CUDA flag list, which as a result compiled the host with -O0 (GCC) and the device with -O3 (NVCC) for CUDA language translation units (TUs):
nvcc [no -O flags] [includes] [file].cpp
When calling cmake -S <src_dir> or ccmake again on the build directory, the CUDA language was enabled and then this was initialized correctly to -O3 ... for nvcc-compiled TUs.
9 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
-O3was only set for CUDA if we ran CMake at least twice. This was an intialization logic bug and should be set from the beginning, but we initialized CUDA too late / applied the transform too early.We now change the default build type to
Releaseto avoid changing defaults and avoid further surprises.Similar implications as in #1203: here the host-side default optimization was taken (GCC:
-O0but NVCC device-side is still defaulting to-O3). We saw ~30% performance regressions for GPU runs with this unoptimized host code path for CUDA translation units.Thanks to @NeilZaim for raising this! Thanks to @WeiqunZhang for debugging this with me!
To Do
Releaseto avoid any future issues-gexplicitly for any build mode