[SYCL][CUDA] Fix unexpected async memcpy (#1798)

When memory buffers are created with either of the following flags
* PI_MEM_FLAGS_HOST_PTR_USE
* PI_MEM_FLAGS_HOST_PTR_COPY
we copy the data to the device.

During memory buffer creation we do not have a CUDA stream and therefore call
cuMemCpyHtoD which operates on the CUDA default stream.

This fix synchronizes with the default stream to ensure that data copying is
finished before any other PI operation uses it on a non-default stream.

Signed-off-by: Bjoern Knafla <bjoern@codeplay.com>

Author: bjoernknafla

sycl/plugins/cuda/pi_cuda.cpp

@@ -1521,7 +1521,15 @@ pi_result cuda_piMemBufferCreate(pi_context context, pi_mem_flags flags,
       if (piMemObj != nullptr) {
         retMemObj = piMemObj.release();
         if (performInitialCopy) {
+          // Operates on the default stream of the current CUDA context.
           retErr = PI_CHECK_ERROR(cuMemcpyHtoD(ptr, host_ptr, size));
+          // Synchronize with default stream implicitly used by cuMemcpyHtoD
+          // to make buffer data available on device before any other PI call
+          // uses it.
+          if (retErr == PI_SUCCESS) {
+            CUstream defaultStream = 0;
+            retErr = PI_CHECK_ERROR(cuStreamSynchronize(defaultStream));
+          }
         }
       } else {
         retErr = PI_OUT_OF_HOST_MEMORY;