Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set NVHPC maximum optimization level to -O1 for now #3800

Merged
merged 3 commits into from
Nov 1, 2023
Merged

Set NVHPC maximum optimization level to -O1 for now #3800

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
cbafdcc
Set NVHPC maximum optimization level to -O1 for now
jhendersonHDF Oct 30, 2023
ff9d26b
Disable nvhpc Java testing in CMake and amend known issues
jhendersonHDF Nov 1, 2023
edde9ef
Re-enable testing of Autotools nvhpc
jhendersonHDF Nov 1, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions .github/workflows/nvhpc-auto.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,11 +67,11 @@ jobs:

# RUN TESTS
# NORMAL
# - name: Autotools Run Tests
# run: |
# export PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/23.7/comm_libs/openmpi4/bin:/opt/nvidia/hpc_sdk/Linux_x86_64/23.7/compilers/bin:$PATH
# make check -j
# working-directory: ${{ runner.workspace }}/build
- name: Autotools Run Tests
run: |
export PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/23.7/comm_libs/openmpi4/bin:/opt/nvidia/hpc_sdk/Linux_x86_64/23.7/compilers/bin:$PATH
make check -j
working-directory: ${{ runner.workspace }}/build

# INSTALL (note that this runs even when we don't run the tests)
- name: Autotools Install
Expand Down
12 changes: 6 additions & 6 deletions .github/workflows/nvhpc-cmake.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ jobs:
-DLIBAEC_USE_LOCALCONTENT=OFF \
-DZLIB_USE_LOCALCONTENT=OFF \
-DHDF5_BUILD_FORTRAN:BOOL=ON \
-DHDF5_ENABLE_ASSERTS:BOOL=ON \
-DHDF5_BUILD_JAVA:BOOL=OFF \
-DMPIEXEC_MAX_NUMPROCS:STRING="2" \
$GITHUB_WORKSPACE
cat src/libhdf5.settings
Expand All @@ -69,8 +69,8 @@ jobs:
working-directory: ${{ runner.workspace }}/build

# RUN TESTS
# - name: CMake Run Tests
# shell: bash
# run: |
# ctest . --parallel 2 -C ${{ inputs.build_mode }} -V
# working-directory: ${{ runner.workspace }}/build
- name: CMake Run Tests
shell: bash
run: |
ctest . --parallel 2 -C ${{ inputs.build_mode }} -V
working-directory: ${{ runner.workspace }}/build
38 changes: 38 additions & 0 deletions config/cmake/HDFCompilerFlags.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,44 @@ if (CMAKE_C_COMPILER_ID STREQUAL "NVHPC" )
else ()
set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Mbounds -g")
endif ()

# With at least NVHPC 23.5 - 23.9, compiling with -O2 or higher and -DNDEBUG
# appears to have issues that manifest in the tests as incorrect metadata
# checksums being read or memory being corrupted. Compiling without -DNDEBUG
# does not appear to have these issues, but is not ideal due to compiling in
# asserts and other library debug code. Compiling with -O1 also does not appear
# to have these issues, so set maximum optimization level to -O1 for now until
# it can be determined whether these issues are compiler-specific or issues
# in the library.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have a keyword that we can add to this to indicate we should revisit the issue at a later date?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought about creating an issue to track this, but open to other suggestions as well.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@derobins , we probably need to develop a procedure indicating workarounds that should be revisited at a future date. Otherwise, they will be forgotten about.

set (cmake_c_flags_minsizerel_edited "${CMAKE_C_FLAGS_MINSIZEREL}")
string (REPLACE "-O2" "" cmake_c_flags_minsizerel_edited "${cmake_c_flags_minsizerel_edited}")
string (REPLACE "-O3" "" cmake_c_flags_minsizerel_edited "${cmake_c_flags_minsizerel_edited}")
string (REPLACE "-O4" "" cmake_c_flags_minsizerel_edited "${cmake_c_flags_minsizerel_edited}")
string (REPLACE "-Ofast" "" cmake_c_flags_minsizerel_edited "${cmake_c_flags_minsizerel_edited}")
string (REPLACE "-fast" "" cmake_c_flags_minsizerel_edited "${cmake_c_flags_minsizerel_edited}")
string (STRIP "${cmake_c_flags_minsizerel_edited}" cmake_c_flags_minsizerel_edited)
string (PREPEND cmake_c_flags_minsizerel_edited "-O1 ")
set (CMAKE_C_FLAGS_MINSIZEREL "${cmake_c_flags_minsizerel_edited}")

set (cmake_c_flags_release_edited "${CMAKE_C_FLAGS_RELEASE}")
string (REPLACE "-O2" "" cmake_c_flags_release_edited "${cmake_c_flags_release_edited}")
string (REPLACE "-O3" "" cmake_c_flags_release_edited "${cmake_c_flags_release_edited}")
string (REPLACE "-O4" "" cmake_c_flags_release_edited "${cmake_c_flags_release_edited}")
string (REPLACE "-Ofast" "" cmake_c_flags_release_edited "${cmake_c_flags_release_edited}")
string (REPLACE "-fast" "" cmake_c_flags_release_edited "${cmake_c_flags_release_edited}")
string (STRIP "${cmake_c_flags_release_edited}" cmake_c_flags_release_edited)
string (PREPEND cmake_c_flags_release_edited "-O1 ")
set (CMAKE_C_FLAGS_RELEASE "${cmake_c_flags_release_edited}")

set (cmake_c_flags_relwithdebinfo_edited "${CMAKE_C_FLAGS_RELWITHDEBINFO}")
string (REPLACE "-O2" "" cmake_c_flags_relwithdebinfo_edited "${cmake_c_flags_relwithdebinfo_edited}")
string (REPLACE "-O3" "" cmake_c_flags_relwithdebinfo_edited "${cmake_c_flags_relwithdebinfo_edited}")
string (REPLACE "-O4" "" cmake_c_flags_relwithdebinfo_edited "${cmake_c_flags_relwithdebinfo_edited}")
string (REPLACE "-Ofast" "" cmake_c_flags_relwithdebinfo_edited "${cmake_c_flags_relwithdebinfo_edited}")
string (REPLACE "-fast" "" cmake_c_flags_relwithdebinfo_edited "${cmake_c_flags_relwithdebinfo_edited}")
string (STRIP "${cmake_c_flags_relwithdebinfo_edited}" cmake_c_flags_relwithdebinfo_edited)
string (PREPEND cmake_c_flags_relwithdebinfo_edited "-O1 ")
set (CMAKE_C_FLAGS_RELWITHDEBINFO "${cmake_c_flags_relwithdebinfo_edited}")
endif ()

if (CMAKE_COMPILER_IS_GNUCC)
Expand Down
6 changes: 4 additions & 2 deletions config/nvidia-flags
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,8 @@ if test "X-nvc" = "X-$cc_vendor" -o "X-nvcc" = "X-$cc_vendor"; then
##############

# NDEBUG is handled explicitly by the configure script
PROD_CFLAGS="-fast"
#PROD_CFLAGS="-fast"
PROD_CFLAGS="" # -fast implies -O2 and -O2+ currently has test failures.

#########
# Debug #
Expand Down Expand Up @@ -106,7 +107,8 @@ if test "X-nvc" = "X-$cc_vendor" -o "X-nvcc" = "X-$cc_vendor"; then
################

HIGH_OPT_CFLAGS="-O1" # -O2+ currently has test failures.
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like we already did this previously for Autotools builds, which is why the NVHPC action passes tests for Autotools builds, but the tests were disabled for CMake builds

DEBUG_OPT_CFLAGS="-gopt -O2"
#DEBUG_OPT_CFLAGS="-gopt -O2"
DEBUG_OPT_CFLAGS="-gopt -O1" # -O2+ currently has test failures.
NO_OPT_CFLAGS="-O0"

#################
Expand Down
22 changes: 22 additions & 0 deletions release_docs/RELEASE.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1158,6 +1158,28 @@ Platforms Tested
Known Problems
==============

When HDF5 is compiled with NVHPC versions 23.5 - 23.9 (additional versions may
also be applicable) and with -O2 (or higher) and -DNDEBUG, test failures occur
in the following tests:

H5PLUGIN-filter_plugin
H5TEST-flush2
H5TEST-testhdf5-base
MPI_TEST_t_filters_parallel

Since these tests pass with an optimization level of -O1 (and -O0) and it is
currently unclear whether the test failures are due to issues in HDF5 or issues
in the 'nvc' compiler, the maximum optimization level for NVHPC has been set
to -O1 until the test failures can be resolved. Note that even at -O1 optimization
level, there still appears to be a sporadic test failure in the Java JUnit tests
that has occasionally been seen in JUnit-TestH5Pfapl and JUnit-TestH5D. It is also
unclear whether this is an issue in HDF5 or with the 'nvc' compiler. Finally, note
that NVHPC 23.9 will fail to compile the test/tselect.c test file with a compiler
error of 'use of undefined value' when the optimization level is -O2 or higher.
Nvidia is aware of this issue and has suggested lowering the optimization level to
-O1 for the time being:
https://forums.developer.nvidia.com/t/hdf5-no-longer-compiles-with-nv-23-9/269045.

IEEE standard arithmetic enables software to raise exceptions such as overflow,
division by zero, and other illegal operations without interrupting or halting
the program flow. The HDF5 C library intentionally performs these exceptions.
Expand Down