Kokkos::complex<double> atomic deadlocks with Clang 6 Cuda build with -O0

[crtrott]

The following code deadlocks with O0:

#include<Kokkos_Core.hpp>


int main(int argc, char* argv[]) {
  Kokkos::initialize(argc,argv);
  {
     Kokkos::View<Kokkos::complex<double>> data("Data");
     Kokkos::parallel_for(2, KOKKOS_LAMBDA(const int& i) {
       Kokkos::atomic_fetch_add( &data(), (Kokkos::complex<double>) 1 );
     });
  }
  Kokkos::finalize();
}

It doesn't deadlock for a single loop iteration.

Build command:

make KOKKOS_DEVICES=Cuda KOKKOS_ARCH=Volta70 -j KOKKOS_PATH=/home/crtrott/Kokkos/kokkos -j CXX=clang++ CXXFLAGS="-O0"

[crtrott]

This caused the Apollo Debug tests to time out.

[crtrott]

It also deadlocks with O1, but works with O2 and O3.

[crtrott]

Found the issue. Effectively __activemask() doesn't do what we think it does, in particular

unsigned mask1 = __activemask();
{
  stuff
}
unsigned mask2 = __activemask();

In this case mask1 != mask2 is possible.