master_june24: 'error: unhandled SGPR spill to memory' in gg_ttggg builds on LUMI/HIP

[valassi]

There is 'error: unhandled SGPR spill to memory' in gg_ttggg builds on LUMI/HIP

I got this in large scale tests so I am not sure where it came from. I am doing more tests to find out.

[valassi]

This seems to have come from

./throughputX.sh -makeonly -makej -ggttggg -fltonly -bldall

and also

./throughputX.sh -makeonly -makej -ggttggg -fltonly -bridge -bldall

So only in FPTYE=f builds?

[valassi]

I reproduced this very simply in a manual build on the logiin node

[valassia@uan03 bash] ~/GPU2024/madgraph4gpu/epochX/cudacpp/gg_ttggg.mad/SubProcesses/P1_gg_ttxggg > make  BACKEND=hip -f cudacpp.mk USEBUILDDIR=1 FPTYPE=f
BACKEND='hip'
OMPFLAGS=
FPTYPE='f'
HELINL='0'
HRDCOD='0'
HASCURAND=hasNoCurand
HASHIPRAND=hasHiprand
Building in BUILDDIR=build.hip_f_inl0_hrd0 for tag=hip_f_inl0_hrd0_hasNoCurand_hasHiprand (USEBUILDDIR == 1)
mkdir -p build.hip_f_inl0_hrd0
make -C ../../src  -f cudacpp_src.mk
make[1]: Entering directory '/pfs/lustrep3/scratch/project_465001114/valassia/GPU2024/madgraph4gpu/epochX/cudacpp/gg_ttggg.mad/src'
mkdir -p ../lib/build.hip_f_inl0_hrd0
mkdir -p build.hip_f_inl0_hrd0
ccache CC --cray-bypass-pkgconfig -craype-verbose  -I. -O3  -std=c++17 -Wall -Wshadow -Wextra -ffast-math   -DMGONGPU_FPTYPE_FLOAT -DMGONGPU_FPTYPE2_FLOAT -fPIC -c read_slha.cc -o build.hip_f_inl0_hrd0/read_slha_cpp.o
g++ -march=znver3 -D__CRAY_X86_TRENTO -D__CRAY_AMD_GFX90A -D__CRAYXT_COMPUTE_LINUX_TARGET -D__TARGET_LINUX__ -O3 -std=c++17 -Wall -Wshadow -Wextra -ffast-math -fPIC -I. -DMGONGPU_FPTYPE_DOUBLE -DMGONGPU_FPTYPE2_DOUBLE -DMGONGPU_CHANNELID_DEBUG -c -o build.hip_d_inl0_hrd0/read_slha_cpp.o read_slha.cc 
ccache /opt/rocm/bin/hipcc  -I.   -O3 --offload-arch=gfx90a -target x86_64-linux-gnu -DHIP_PLATFORM=amd -DHIP_FAST_MATH -I/opt/rocm/include/ -std=c++17 -DMGONGPU_FPTYPE_FLOAT -DMGONGPU_FPTYPE2_FLOAT  -fPIC -c -x hip Parameters_sm.cc -o build.hip_f_inl0_hrd0/Parameters_sm_hip.o
ccache /opt/rocm/bin/hipcc -shared -o ../lib/build.hip_f_inl0_hrd0/libmg5amc_common_hip.so build.hip_f_inl0_hrd0/read_slha_cpp.o build.hip_f_inl0_hrd0/Parameters_sm_hip.o 
make[1]: Leaving directory '/pfs/lustrep3/scratch/project_465001114/valassia/GPU2024/madgraph4gpu/epochX/cudacpp/gg_ttggg.mad/src'
ccache /opt/rocm/bin/hipcc  -I. -I../../src   -O3 --offload-arch=gfx90a -target x86_64-linux-gnu -DHIP_PLATFORM=amd -DHIP_FAST_MATH -I/opt/rocm/include/ -std=c++17 -DMGONGPU_FPTYPE_FLOAT -DMGONGPU_FPTYPE2_FLOAT  -fPIC  -DMGONGPU_HAS_NO_CURAND -c -x hip check_sa.cc -o build.hip_f_inl0_hrd0/check_sa_hip.o
ccache /opt/rocm/bin/hipcc  -I. -I../../src   -O3 --offload-arch=gfx90a -target x86_64-linux-gnu -DHIP_PLATFORM=amd -DHIP_FAST_MATH -I/opt/rocm/include/ -std=c++17 -DMGONGPU_FPTYPE_FLOAT -DMGONGPU_FPTYPE2_FLOAT  -fPIC -c -x hip CPPProcess.cc -o build.hip_f_inl0_hrd0/CPPProcess_hip.o
error: unhandled SGPR spill to memory
error: unhandled SGPR spill to memory
error: unhandled SGPR spill to memory
error: unhandled SGPR spill to memory
error: unhandled SGPR spill to memory
error: unhandled SGPR spill to memory
error: unhandled SGPR spill to memory
error: unhandled SGPR spill to memory
error: unhandled SGPR spill to memory
error: unhandled SGPR spill to memory
error: unhandled SGPR spill to memory
error: unhandled SGPR spill to memory
error: unhandled SGPR spill to memory
error: unhandled SGPR spill to memory
error: unhandled SGPR spill to memory
error: unhandled SGPR spill to memory
error: unhandled SGPR spill to memory
error: unhandled SGPR spill to memory
error: unhandled SGPR spill to memory
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated when compiling for gfx90a.
make: *** [cudacpp.mk:741: build.hip_f_inl0_hrd0/CPPProcess_hip.o] Error 1

Note that this took 3 hours before failing... another FPTYPE=d build is continuing after 3 hours.

This is only happening on the master_june24 derived branch with #882 after #830. I assume that the present code has grown too complex, and LUMI does not build it anymore.

As a workaround, I will simply disable all ggttggg tests in LUMI in my manual tput/tmad scripts.

[valassi]

Workarounds for this are in PR #882. Essentially I skip the ggttggg HIP builds on LUMI, as they run out of memory. Closing.