Question: Inference from CUDA allocated memory

[IvensaMDH]

Hi,

Is it possible to run inference (using CUDA provider) )from memory already allocated on CUDA without moving it from GPU to CPU and vice versa?

Could you provide an example of such implementation in C#?

Thanks,

/M

[yuslepukhin]

You may want to consider using IOBinding. Here is some docs.

This is the C# class that implements IOBinding.

Here is a couple of links that show IOBinding use in tests:

https://github.com/microsoft/onnxruntime/blob/master/csharp/test/Microsoft.ML.OnnxRuntime.Tests.Common/OrtIoBindingAllocationTest.cs

https://github.com/microsoft/onnxruntime/blob/master/csharp/test/Microsoft.ML.OnnxRuntime.Tests.Common/InferenceTest.cs#L1630

[IvensaMDH]

Thanks!

How would you go about using a pointer to GPU memory and bind it using IOBinding as input?

Currently the IOBinding.BindInput has two overloads; using a FixedBufferOnnxValue or OrtMemoryAllocation - I'm unable to identify any way to instantiate any of these using a pointer to unmanaged memory on the GPU?

Thanks in advance,

[yuslepukhin]

Thanks!

How would you go about using a pointer to GPU memory and bind it using IOBinding as input?

Currently the IOBinding.BindInput has two overloads; using a FixedBufferOnnxValue or OrtMemoryAllocation - I'm unable to identify any way to instantiate any of these using a pointer to unmanaged memory on the GPU?

Thanks in advance,

How do you represent your pointer to a GPU memory in your C# code?

[IvensaMDH]

I am interfacing with a Nvidia DeepStream application and retrieving the pointer as described in: API

Using cudaMemcpy2D (API) I am able to copy the buffer to host device without problems.

unsafe static extern int cudaMemcpy2D(IntPtr dst, uint dpitch, IntPtr src, uint spitch, uint width, uint height, cudaMemcpyKind kind);

Example:

using var buffer = info.Buffer;

buffer.Map(out MapInfo map, MapFlags.Read);

NvBufSurface nvBufSurface = Marshal.PtrToStructure<NvBufSurface>(map.DataPtr);
IntPtr cudaPtr = (IntPtr)nvBufSurface.surfaceList[0].dataPtr;

byte[] buff = new byte[nvBufSurface.surfaceList[0].width * nvBufSurface.surfaceList[0].height * *nvBufSurface.surfaceList[0].planeParams.bytesPerPix];
fixed (byte* dst = &buff[0])
{
    int ret = cudaMemcpy2D((IntPtr)dst, nvBufSurface.surfaceList[0].width, cudaPtr, nvBufSurface.surfaceList[0].pitch, nvBufSurface.surfaceList[0].width, nvBufSurface.surfaceList[0].height, cudaMemcpyKind.cudaMemcpyDeviceToHost);

}

buffer.Unmap(map);

I notice that OrtValue has a IntPtr option:
OrtValue ortValue = OrtValue.CreateTensorValueWithData(cudaMemoryInfo, TensorElementType.Float, new long[] { 1, 3, 640, 640 }, cudaPtr, nvBufSurface.surfaceList[0].width * nvBufSurface.surfaceList[0].height * *nvBufSurface.surfaceList[0].planeParams.bytesPerPix);

However I don't see any way of passing an OrtValue to the inference session Run() or bind it with IOBinding.

The Python API describes in the Docs (Scenario 2) binds the X_ortvalue with io_binding.bind_input(name='input', device_type=X_ortvalue.device_name(), device_id=0, element_type=np.float32, shape=X_ortvalue.shape(), buffer_ptr=X_ortvalue.data_ptr()) however I don't see a similar API in C#?

Thanks,

[stale]

This issue has been automatically marked as stale due to inactivity and will be closed in 7 days if no further activity occurs. If further support is needed, please provide an update and/or more details.