sycl/build/bin/opt: error: expected top-level entity

[j-stephan]

I am currently trying to integrate the sycl/unified/next compiler into alpaka's (https://github.com/alpaka-group/alpaka) CMake infrastructure. Compiling a simple test case (axpy) works fine but once the link stage is reached it fails with the following error:

[100%] Linking CXX executable axpy
cd /home/jan/workspace/zoo/alpaka/build/test/integ/axpy && /snap/cmake/703/bin/cmake -E cmake_link_script CMakeFiles/axpy.dir/link.txt --verbose=1
/home/jan/sycl-workspace/sycl/build/bin/clang++ -O3 -DNDEBUG -fsycl -fsycl-targets=fpga64-xilinx-unknown-sycldevice CMakeFiles/axpy.dir/src/axpy.cpp.o -o axpy  ../../common/libcommon.a -lpthread /usr/lib/x86_64-linux-gnu/librt.so /usr/lib/x86_64-linux-gnu/libOpenCL.so -lsycl ../../catch_main/libCatchMain.a 
++ pwd
+ CWD=/home/jan/workspace/zoo/alpaka/build/test/integ/axpy
+ export VERBOSE=1
+ VERBOSE=1
+ [[ -z /home/jan/software/xilinx/Vitis/2020.1/bin ]]
+ [[ -z /home/jan/sycl-workspace/sycl/build/bin ]]
+ [[ -z axpy.cpp-37c5af ]]
+ [[ -z /tmp/axpy.cpp-37c5af.txt ]]
+ [[ -z /tmp ]]
+ [[ -z /tmp/axpy-e9e831.bc ]]
+ [[ -z /tmp/sycl-xocc-args2712-cbcee1 ]]
+ [[ -z /tmp/sycl-xocc-args2716-8d3ef5 ]]
+ [[ -z sw_emu ]]
+ SDX_BIN_PATH_DIR=/home/jan/software/xilinx/Vitis/2020.1/bin
+ [[ ! -d /home/jan/software/xilinx/Vitis/2020.1/bin ]]
+ SDX_LIB_SPIR=/home/jan/software/xilinx/Vitis/2020.1/bin/../lnx64/lib/libspir64-39-hls.bc
+ [[ ! -f /home/jan/software/xilinx/Vitis/2020.1/bin/../lnx64/lib/libspir64-39-hls.bc ]]
+ SDX_CLANG_LLVM_BIN=/home/jan/software/xilinx/Vitis/2020.1/bin/../lnx64/tools/clang-3.9-csynth/bin
+ [[ ! -f /home/jan/software/xilinx/Vitis/2020.1/bin/../lnx64/tools/clang-3.9-csynth/bin/llvm-as ]]
+ SDX=/home/jan/software/xilinx/Vitis/2020.1/bin/v++
+ [[ ! -f /home/jan/software/xilinx/Vitis/2020.1/bin/v++ ]]
+ DRIVER_PATH_DIR=/home/jan/sycl-workspace/sycl/build/bin
+ [[ ! -d /home/jan/sycl-workspace/sycl/build/bin ]]
+ OPT=/home/jan/sycl-workspace/sycl/build/bin/opt
+ [[ ! -f /home/jan/sycl-workspace/sycl/build/bin/opt ]]
+ LLVM_LINK=/home/jan/sycl-workspace/sycl/build/bin/llvm-link
+ [[ ! -f /home/jan/sycl-workspace/sycl/build/bin/llvm-link ]]
+ [[ ! -d /tmp ]]
+ SOURCE_FILE_NAME=axpy.cpp-37c5af
+ INPUT_FILE_NAME=/tmp/axpy.cpp-37c5af.txt
++ mktemp /tmp/sycl-xocc.XXXXXX
+ TMP_PATH=/tmp/sycl-xocc.s4Ko3U
+ OUTPUT_FILE_NAME=/tmp/axpy-e9e831.bc
++ cat /tmp/sycl-xocc-args2712-cbcee1
+ ADDITIONAL_COMPILE_ARGS=
++ cat /tmp/sycl-xocc-args2716-8d3ef5
+ ADDITIONAL_LINK_ARGS=
+ XCL_EMULATION_MODE=sw_emu
+ KERNELPROP=/tmp/sycl-xocc.s4Ko3U_KernelProperties_axpy.cpp-37c5af.bash
+ cp /tmp/axpy.cpp-37c5af.txt /tmp/sycl-xocc.s4Ko3U_axpy.cpp-37c5af_kernels.bc
+ /home/jan/sycl-workspace/sycl/build/bin/opt -asfix -globaldce -O3 -globaldce --infer-address-spaces -flat-address-space=4 -globaldce -inSPIRation -globaldce -kernelPropGen --sycl-kernel-propgen-output /tmp/sycl-xocc.s4Ko3U_KernelProperties_axpy.cpp-37c5af.bash /tmp/axpy.cpp-37c5af.txt -o /tmp/sycl-xocc.s4Ko3U_axpy.cpp-37c5af_kernels-optimized.bc
/home/jan/sycl-workspace/sycl/build/bin/opt: /tmp/axpy.cpp-37c5af.txt:1:1: error: expected top-level entity
/tmp/axpy-4e596d-e13720.devo
^
clang-12: error: sycl-link-xocc command failed with exit code 1 (use -v to see invocation)
make[3]: *** [test/integ/axpy/CMakeFiles/axpy.dir/build.make:107: test/integ/axpy/axpy] Error 1
make[3]: Leaving directory '/home/jan/workspace/zoo/alpaka/build'
make[2]: *** [CMakeFiles/Makefile2:1710: test/integ/axpy/CMakeFiles/axpy.dir/all] Error 2
make[2]: Leaving directory '/home/jan/workspace/zoo/alpaka/build'
make[1]: *** [CMakeFiles/Makefile2:1717: test/integ/axpy/CMakeFiles/axpy.dir/rule] Error 2
make[1]: Leaving directory '/home/jan/workspace/zoo/alpaka/build'
make: *** [Makefile:610: axpy] Error 2

Is there anything I can do to prevent or further investigate this error?

[j-stephan]

For the record: Switching to Vitis 2020.2 doesn't change anything.

[Ralender]

From my understanding, you are trying split compilation meaning compile and link steps are separated compiler invocations.
with the current compile flow we don't yet support this. but i agree we should because it is the build flow used by most build systems.
I am going to look at how we can support that.

[j-stephan]

From my understanding, you are trying split compilation meaning compile and link steps are separated compiler invocations.

I believe that is what CMake does internally, yes. We didn't change this behaviour. Running with make VERBOSE=1 shows that the files are first compiled individually (using clang's -c flag) and later linked together with a separate command.

I am going to look at how we can support that.

Great, thank you!

[Ralender]

I have a prototype for split compilation at https://github.com/Ralender/sycl/tree/SplitCompilation. if you want to test it.

[j-stephan]

Nice to see that this is being worked on! Unfortunately this doesn't solve the problem yet:

~/sycl-workspace/SplitCompilation/sycl:$ git branch --show-current
SplitCompilation

~/workspace/zoo/alpaka/build:$ which clang
/home/jan/sycl-workspace/SplitCompilation/sycl/build/bin/clang

~/workspace/zoo/alpaka/build:$ make axpy
Linking CXX executable axpy
++ pwd
+ CWD=/home/jan/workspace/zoo/alpaka/build/test/integ/axpy
+ export VERBOSE=1
+ VERBOSE=1
+ [[ -z /home/jan/software/xilinx/Vitis/2020.2/bin ]]
+ [[ -z /home/jan/sycl-workspace/SplitCompilation/sycl/build/bin ]]
+ [[ -z axpy.cpp-557f97 ]]
+ [[ -z /tmp/axpy.cpp-557f97.txt ]]
+ [[ -z /tmp ]]
+ [[ -z /tmp/axpy-2a4851.bc ]]
+ [[ -z /tmp/sycl-xocc-args2712-64dd6d ]]
+ [[ -z /tmp/sycl-xocc-args2716-f4c90f ]]
+ [[ -z sw_emu ]]
+ SDX_BIN_PATH_DIR=/home/jan/software/xilinx/Vitis/2020.2/bin
+ [[ ! -d /home/jan/software/xilinx/Vitis/2020.2/bin ]]
+ SDX_LIB_SPIR=/home/jan/software/xilinx/Vitis/2020.2/bin/../lnx64/lib/libspir64-39-hls.bc
+ [[ ! -f /home/jan/software/xilinx/Vitis/2020.2/bin/../lnx64/lib/libspir64-39-hls.bc ]]
+ SDX_LIB_SPIR=/home/jan/software/xilinx/Vitis/2020.2/bin/../../../Vitis_HLS/2020.2/lnx64/lib/libspir64-39-hls.bc
+ [[ ! -f /home/jan/software/xilinx/Vitis/2020.2/bin/../../../Vitis_HLS/2020.2/lnx64/lib/libspir64-39-hls.bc ]]
+ SDX_CLANG_LLVM_BIN=/home/jan/software/xilinx/Vitis/2020.2/bin/../lnx64/tools/clang-3.9-csynth/bin
+ [[ ! -f /home/jan/software/xilinx/Vitis/2020.2/bin/../lnx64/tools/clang-3.9-csynth/bin/llvm-as ]]
+ SDX_CLANG_LLVM_BIN=/home/jan/software/xilinx/Vitis/2020.2/bin/../../../Vitis_HLS/2020.2/lnx64/tools/clang-3.9-csynth/bin
+ [[ ! -f /home/jan/software/xilinx/Vitis/2020.2/bin/../../../Vitis_HLS/2020.2/lnx64/tools/clang-3.9-csynth/bin/llvm-as ]]
+ SDX=/home/jan/software/xilinx/Vitis/2020.2/bin/v++
+ [[ ! -f /home/jan/software/xilinx/Vitis/2020.2/bin/v++ ]]
+ DRIVER_PATH_DIR=/home/jan/sycl-workspace/SplitCompilation/sycl/build/bin
+ [[ ! -d /home/jan/sycl-workspace/SplitCompilation/sycl/build/bin ]]
+ OPT=/home/jan/sycl-workspace/SplitCompilation/sycl/build/bin/opt
+ [[ ! -f /home/jan/sycl-workspace/SplitCompilation/sycl/build/bin/opt ]]
+ LLVM_LINK=/home/jan/sycl-workspace/SplitCompilation/sycl/build/bin/llvm-link
+ [[ ! -f /home/jan/sycl-workspace/SplitCompilation/sycl/build/bin/llvm-link ]]
+ [[ ! -d /tmp ]]
+ SOURCE_FILE_NAME=axpy.cpp-557f97
+ INPUT_FILE_NAME=/tmp/axpy.cpp-557f97.txt
++ mktemp /tmp/sycl-xocc.XXXXXX
+ TMP_PATH=/tmp/sycl-xocc.cZqmFT
+ OUTPUT_FILE_NAME=/tmp/axpy-2a4851.bc
++ cat /tmp/sycl-xocc-args2712-64dd6d
+ ADDITIONAL_COMPILE_ARGS=
++ cat /tmp/sycl-xocc-args2716-f4c90f
+ ADDITIONAL_LINK_ARGS=
+ XCL_EMULATION_MODE=sw_emu
+ KERNELPROP=/tmp/sycl-xocc.cZqmFT_KernelProperties_axpy.cpp-557f97.bash
+ cp /tmp/axpy.cpp-557f97.txt /tmp/sycl-xocc.cZqmFT_axpy.cpp-557f97_kernels.bc
+ /home/jan/sycl-workspace/SplitCompilation/sycl/build/bin/opt -asfix -globaldce -O3 -globaldce --infer-address-spaces -flat-address-space=4 -globaldce -inSPIRation -globaldce -kernelPropGen --sycl-kernel-propgen-output /tmp/sycl-xocc.cZqmFT_KernelProperties_axpy.cpp-557f97.bash /tmp/axpy.cpp-557f97.txt -o /tmp/sycl-xocc.cZqmFT_axpy.cpp-557f97_kernels-optimized.bc
/home/jan/sycl-workspace/SplitCompilation/sycl/build/bin/opt: /tmp/axpy.cpp-557f97.txt:1:1: error: expected top-level entity
/tmp/axpy-98f09c-970d9d.devo
^
clang-12: error: sycl-link-xocc command failed with exit code 1 (use -v to see invocation)
make[3]: *** [test/integ/axpy/CMakeFiles/axpy.dir/build.make:107: test/integ/axpy/axpy] Error 1
make[2]: *** [CMakeFiles/Makefile2:1710: test/integ/axpy/CMakeFiles/axpy.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:1717: test/integ/axpy/CMakeFiles/axpy.dir/rule] Error 2
make: *** [Makefile:610: axpy] Error 2

[Ralender]

Could you provide me with guidance on how i can reproduce this locally ?

[j-stephan]

Sure! Just follow the steps below. Dependencies are clang from your SplitCompilation branch, Vitis (with all the environment flags set), Boost, CMake >= 3.15 and the ComputeCpp SDK from here: https://github.com/codeplaysoftware/computecpp-sdk

git clone -b sycl git@github.com:j-stephan/alpaka.git
cd alpaka
mkdir build && cd build
CXX=clang cmake \
    -DALPAKA_CXX_STANDARD=17 \
    -DCMAKE_BUILD_TYPE=Release \
    -DBUILD_TESTING=ON \
    -DALPAKA_ACC_SYCL_ENABLE=ON \
    -DALPAKA_SYCL_PLATFORM_ONEAPI=OFF \
    -DALPAKA_SYCL_PLATFORM_XILINX=ON \
    -DALPAKA_COMPUTECPP_SDK_DIR=/path/to/computecpp-sdk \
    ..
cmake --build . --target axpy

[Ralender]

Thank you I was able to reproduce it. I will look into it.

[Ralender]

The issue you were seeing is fixed on my branch but now there are double pointer that make XOCC unhappy.

[j-stephan]

You mean pointers of type double?

[Ralender]

no a pointer on a pointer, something like A**

[j-stephan]

Do you know if those are coming from the compiler? There shouldn't be any pointers to pointers in the AXPY test case (only simple float pointers) or the alpaka code base, but you never know...

[Ralender]

The pointer on pointer restriction is a restriction of v++.

Generally speaking is is hard to know that there are no pointers on pointers because:

• it depends on what optimization the sycl toolchain can perform before invoking v++.
• many C++ construct have implicit pointer not necessarily obvious to the user.

for example:

int i = 0;
int* p = &i;

without optimization, this will contain a pointer on pointer because this would be put the on stack(in IR).
but obviously with optimizations this will not contain pointers on pointers.

In this instance the pointer on pointer comes from:
the kernel lambda at alpaka/include/alpaka/kernel/TaskKernelGenericSycl.hpp:160:13
capturing a utility::tuple::Tuple<unsigned long, float, float *, float *> that contains some float*
and the kernel lambda being passed to the kernel function by pointer.

I think the best fix is to pass accessors to the data pointed to by the various float*.
there is some special handling for accessors that should make it work.

hope this helps.