[SYCL] xclbinutil crashes with segmentation fault in sw_emu mode

[j-stephan]

I thought I'd give the n-body example from #28 another shot and see how things improved. This caused a segmentation fault in xclbinutil during linking in sw_emu mode. Versions used: up-to-date master (c3c6023) for the compiler, up-to-date master for XRT (Xilinx/XRT@feb6849), SDAccel 2019.1.1, Ubuntu 19.04.

Environment:

$: echo $XCL_EMULATION_MODE
sw_emu
$: echo $XILINX_PLATFORM
xilinx_u200_xdma_201830_2
$: apt search xdma
xilinx-u200-xdma/now 201830.2-2580015 all [installed,local]
xilinx-u200-xdma-dev/now 201830.2-2580015 amd64 [installed,local]

Command line:

clang++ -std=c++2a -fsycl -fsycl-targets=fpga64-xilinx-unknown-sycldevice nbody.cpp -o nbody.sw_emu -lOpenCL -I/opt/xilinx/xrt/include

Error message:

[previous messages snipped]
****** xocc v2019.1.1 (64-bit)
  **** SW Build 2580384 on Sat Jun 29 08:04:45 MDT 2019
    ** Copyright 1986-2019 Xilinx, Inc. All Rights Reserved.

Attempting to get a license: ap_opencl
Feature available: ap_opencl
INFO: [XOCC 60-1306] Additional information associated with this xocc link can be found at:
	Reports: /home/jan/workspace/sycl-nbody/_x/reports/link
	Log files: /home/jan/workspace/sycl-nbody/_x/logs/link
INFO: [XOCC 60-629] Linking for software emulation target
INFO: [XOCC 60-1316] Initiating connection to rulecheck server, at Thu Aug 15 15:55:15 2019
Running Rule Check Server on port:43061
INFO: [XOCC 60-1315] Creating rulecheck session with output '/home/jan/workspace/sycl-nbody/_x/reports/link/xocc_link_nbody-0e0111_guidance.html', at Thu Aug 15 15:55:16 2019
INFO: [XOCC 60-895]   Target platform: /opt/xilinx/platforms/xilinx_u200_xdma_201830_2/xilinx_u200_xdma_201830_2.xpfm
INFO: [XOCC 60-423]   Target device: xilinx_u200_xdma_201830_2
ERROR: [XOCC 60-399] xclbinutil failed, please see log file for detail: '/home/jan/workspace/sycl-nbody/_x/link/int/nbody-0e0111_xclbinutil.log'
ERROR: [XOCC 60-626] Kernel link failed to complete
ERROR: [XOCC 60-703] Failed to finish linking
/home/jan/software/sycl/xilinx/bin/sycl-xocc: line 179: 23871 Segmentation fault      (core dumped) $XILINX_XRT/bin/xclbinutil --info --input "$OUTPUT_FILE_NAME" &> "$OUTPUT_FILE_NAME.xclbinutil.dump"
/usr/bin/ld: /tmp/nbody-3310d8.o: file not recognized: file truncated
clang-9: error: sycl-link-xocc command failed with exit code 139 (use -v to see invocation)
clang-9: error: linker command failed with exit code 1 (use -v to see invocation)

Unfortunately the log file mentioned in the output is present but empty.

[agozillon]

I'll take a look into it when I get a chance, thank you for the report!

In future, if you could dump the whole compiler spew that would be very helpful as sometimes the script will fail at a prior stage and carry on through till it dies.

The segfault in this case seems to be one of those occasions, it looks like it's dying because it's dumping some debug info out after it's attempted to link a binary: https://github.com/triSYCL/sycl/blob/sycl/unified/master/sycl/tools/sycl-xocc/bin/sycl-xocc#L176

Which may be because the link stage is failing and not outputting a file (likely, but you'd hope you'd get an error message and not a segfault from the xclbinutil invocation) or because the xclbinutil XRT has installed isn't the same version as the xclbinutil that SDAccel 2019.1.1 is using and the binaries are incompatible. As sadly SDAccel comes packaged with it's own version which I don't think is always forward compatible to the XRT master version. We should probably be using the SDAccel packaged version instead in this case rather than the XRT version to dump the log information.

Also just as an aside to make sure you define export DISABLE_INFER_AS=1 as part of your environment at the moment, in case you're missing it! it's not likely to solve the issue in this particular case as sw_emu usually doesn't die over address spaces, but at the moment that needs to be defined in the environment for hw/hw_emu.

[j-stephan]

Full compiler output:

warning: Linking two modules of different data layouts: '/tools/Xilinx/SDx/2019.1/bin/../lnx64/lib/libspir64-39-hls.bc' is 'e-m:e-i64:64-i128:128-i256:256-i512:512-i1024:1024-i2048:2048-i4096:4096-n8:16:32:64-S128-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024' whereas 'llvm-link' is 'e-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024'

warning: Linking two modules of different target triples: /tools/Xilinx/SDx/2019.1/bin/../lnx64/lib/libspir64-39-hls.bc' is 'fpga64-xilinx-none' whereas 'llvm-link' is 'spir64'

Invoking xocc Kernel Compilation
xocc: /tools/Xilinx/SDx/2019.1/bin/xocc
--target: sw_emu
--platform: xilinx_u200_xdma_201830_2
Compiling kernel: xSYCL17780616878999318149
Outputting file to: /tmp/xSYCL17780616878999318149.xo
Input file is: /tmp/nbody_kernels-linked.xpirbc
Option Map File Used: '/tools/Xilinx/SDx/2019.1/data/sdx/xocc/optMap.xml'

****** xocc v2019.1.1 (64-bit)
  **** SW Build 2580384 on Sat Jun 29 08:04:45 MDT 2019
    ** Copyright 1986-2019 Xilinx, Inc. All Rights Reserved.

Attempting to get a license: ap_opencl
Feature available: ap_opencl
INFO: [XOCC 60-1306] Additional information associated with this xocc compile can be found at:
	Reports: /home/jan/workspace/sycl-nbody/_x/reports/xSYCL17780616878999318149
	Log files: /home/jan/workspace/sycl-nbody/_x/logs/xSYCL17780616878999318149
INFO: [XOCC 60-585] Compiling for software emulation target
INFO: [XOCC 60-1316] Initiating connection to rulecheck server, at Thu Aug 15 20:48:52 2019
Running Rule Check Server on port:35787
INFO: [XOCC 60-1315] Creating rulecheck session with output '/home/jan/workspace/sycl-nbody/_x/reports/xSYCL17780616878999318149/xocc_compile_xSYCL17780616878999318149_guidance.html', at Thu Aug 15 20:48:53 2019
INFO: [XOCC 60-895]   Target platform: /opt/xilinx/platforms/xilinx_u200_xdma_201830_2/xilinx_u200_xdma_201830_2.xpfm
INFO: [XOCC 60-423]   Target device: xilinx_u200_xdma_201830_2
INFO: [XOCC 60-242] Creating kernel: 'xSYCL17780616878999318149'
INFO: [XOCC 60-594] Finished kernel compilation
INFO: [XOCC 60-586] Created /tmp/xSYCL17780616878999318149.xo
INFO: [XOCC 60-791] Total elapsed time: 0h 0m 8s
Option Map File Used: '/tools/Xilinx/SDx/2019.1/data/sdx/xocc/optMap.xml'

****** xocc v2019.1.1 (64-bit)
  **** SW Build 2580384 on Sat Jun 29 08:04:45 MDT 2019
    ** Copyright 1986-2019 Xilinx, Inc. All Rights Reserved.

Attempting to get a license: ap_opencl
Feature available: ap_opencl
INFO: [XOCC 60-1306] Additional information associated with this xocc link can be found at:
	Reports: /home/jan/workspace/sycl-nbody/_x/reports/link
	Log files: /home/jan/workspace/sycl-nbody/_x/logs/link
INFO: [XOCC 60-629] Linking for software emulation target
INFO: [XOCC 60-1316] Initiating connection to rulecheck server, at Thu Aug 15 20:49:03 2019
Running Rule Check Server on port:46137
INFO: [XOCC 60-1315] Creating rulecheck session with output '/home/jan/workspace/sycl-nbody/_x/reports/link/xocc_link_nbody-7e26cf_guidance.html', at Thu Aug 15 20:49:04 2019
INFO: [XOCC 60-895]   Target platform: /opt/xilinx/platforms/xilinx_u200_xdma_201830_2/xilinx_u200_xdma_201830_2.xpfm
INFO: [XOCC 60-423]   Target device: xilinx_u200_xdma_201830_2
ERROR: [XOCC 60-399] xclbinutil failed, please see log file for detail: '/home/jan/workspace/sycl-nbody/_x/link/int/nbody-7e26cf_xclbinutil.log'
ERROR: [XOCC 60-626] Kernel link failed to complete
ERROR: [XOCC 60-703] Failed to finish linking
/home/jan/software/sycl/xilinx/bin/sycl-xocc: line 179:  9090 Segmentation fault      (core dumped) $XILINX_XRT/bin/xclbinutil --info --input "$OUTPUT_FILE_NAME" &> "$OUTPUT_FILE_NAME.xclbinutil.dump"
/usr/bin/ld: /tmp/nbody-2cc506.o: file not recognized: file truncated
clang-9: error: sycl-link-xocc command failed with exit code 139 (use -v to see invocation)
clang-9: error: linker command failed with exit code 1 (use -v to see invocation)

Sorry for snipping before, the first pass seemed uninteresting which is why I wanted to keep it short. I also attached everything the compiler produced in the zip file below.

crashinfo.zip

Thanks for the hint with the environment variable, I missed that it was added to the instructions. It didn't change anything for this issue, though.

[agozillon]

Thank you very much! It's no problem, I just like the full log as I get the full context of the script from it and thank you for the crash information.

Yes, unfortunately or fortunately depending on how you look at it sw_emu seems to not care too much about address spaces, but the environment variable will prevent some hw_emu failures and probably some hw compilation problems. We have an idea to circumvent address spaces at some point which will hopefully dodge a lot of these issues, but I've not had a chance to look into it deeply yet sadly.

[j-stephan]

It seems this isn't related to the SYCL runtime. XRT's xclbinutil crashes with a segmentation fault if I call it directly and I assume this causes the error message above.

[j-stephan]

Closing for now, until I figured out where this crash is coming from.

[j-stephan]

Wasn't related to SYCL at all. xclbinutil needs special compilation flags for Ubuntu 18.04 and newer, which weren't applied for Ubuntu 19.04. See Xilinx/XRT#1833 for future reference.

[agozillon]

Awesome, thank you very much for reporting and fixing this!