[SYCL] [DOC] Prepare design-document for assert feature

[s-kanaev]

See extension document for SYCL describing assert behaviour

[HoppeMateusz]

I don;t think this is possible to achieve in asynchronous / non-blocking way in L0.

We dont have any communication between kernel and event - so we can;t signal events with "assert happened" information.

if we use global / program wide assert buffer - each kernel will be using the same assert happened flag - we do not have fine grain control to determine which kernel - and which connected event fired the assert.

Fences could be used - allowing to synchronize at cmdQueue level and not kernel - any kernel causing assert executed in cmd Queue can then make fence synchronize to return error:https://spec.oneapi.com/level-zero/latest/core/PROG.html#fences

[s-kanaev]

Is it still possible in OpenCL?
Can the OpenCL approach be reused in Level-Zero?

[s-kanaev]

Could you, please, provide more details about using fences?

[HoppeMateusz]

fences are decribed in L0 spec - they are similar to events, but directly connected to command queues: https://spec.oneapi.com/level-zero/latest/core/PROG.html#fences

In OpenCL the submission model is different - each enqueue is independent - single kernel is submitted ( queued) at a time. L0 operates on command lists that may contain multiple kernels - once cmd list is submitted to HW - we can;t control when a kernel in whole sequence is started completed.

OpenCL handles kernels with printf in a blocking way - enqueueNDRangeKErnel with printf makes this a blocking call - so we have fine control when specific kernel is completed - we can do the same for assert() message - output event will be created when the kernel has already finished. I L0 this is not possible - as we would have to synchronize whoel command list.

[AlexeySachkov]

Does it happen always or only without async_handler set?

[gmlueck]

I think it should always print the assertion message because:

• This would be consistent with the "safe" implementation (the one that depends on hardware support), which is defined to print the message even before notifying the host.

• This is also consistent with the way assert works on the host, which prints the assertion message even before raising SIGABRT.

[AlexeySachkov]

So, even if user set an async_handler in order to gracefully react to assert failure, we still print something to stderr? What for? It is not that bad as if we printed into stdout, but still seems unnecessary a bit.

[gmlueck]

I thought it was weird when I first read this spec also. But then I tried the following test:

#include <cassert>
#include <csignal>
#include <cstdlib>

void handle(int sig) {
  std::exit(0);  // Exit silently
}

int main() {
  std::signal(SIGABRT, handle);
  assert(false);
}

The results:

$ clang -std=c++17 -pedantic -o t t.cpp
$ ./t
t: t.cpp:11: int main(): Assertion `false' failed.
$ echo $?
0

Despite the fact that I catch the SIGABRT and exit without printing anything, I still get a message printed to stderr.

Therefore, it seems like the behavior defined in this spec is consistent with the way assert works on the host.

[s-kanaev]

That's kind of obvious due to the fact that assert(expr) on host is unwrapped into

if (!(expr)) {
  fprintf(stderr, ...);
  abort();
}

In device-code, assert(expr) unwraps to:

if (!(expr)) {
  __devicelib_assert_fail(#expr, __FILE__, __LINE__, __PRETTY_FUNCTION__, global ID, local ID);
}

[s-kanaev]

@kbobrovs , a friendly ping

[bader]

LGTM, just a few nits.

[gmlueck]

I'd suggest changing this paragraph in the extension specification now that we have the new aspect:

It is unspecified whether a failing assert() returns to its caller before the kernel terminates. If a failing call returns, the device code may need to continue execution without deadlocking for the assertion message to be printed or for std::abort() to be called.

Maybe something like this:

Some devices implement assert() natively while others use a fallback implementation, and the two implementations provide different guarantees. The native implementation is most similar to the way assert() works on the host. If an assertion fails in the native implementation, the assertion message is immediately printed to stderr and the program terminates by calling std::abort(). If an assertion fails with the fallback implementation, the failing assert() returns back to its caller and the device code must continue executing (without deadlocking) until the kernel completes. The implementation prints the assertion message to stderr and terminates with std::abort() only after the kernel completes execution. An application can determine which of the two mechanisms a device uses by testing the device aspect aspect::ext_oneapi_native_assert.

Note that this also defines the terms "native support" and "fallback implementation", which you use later in the description of ext_oneapi_native_assert.

[s-kanaev]

I'd suggest changing this paragraph in the extension specification now that we have the new aspect:

Done.

[gmlueck]

LGTM

[s-kanaev]

The failure in Jenkins/Precommit doesn't relate to the changes here. Reported the failure.

[bader]

The failure in Jenkins/Precommit doesn't relate to the changes here. Reported the failure.

Please, disable related tests and re-run the job.

[s-kanaev]

Created PR to disable the test: intel/llvm-test-suite#303

[olegmaslovatintel]

We had a many user reported issues after functionality is merged #3767 which seems caused by fall back design.
@s-kanaev is there possibility to NOT enable/define/link `__devicelib_assert_fail by default?

tagging @AlexeySachkov @gmlueck @kbobrovs