Varying performance based on slightly different data

[user01]

Hi,

I've been experimenting with pyopencl by implementing a specific function from h3.

During my tests, I've noticed very different performance based on small changes to the provided sets (~10x). The timings are quite repeatable based on multiple executions. Also interestingly, I don't see a pattern for which sets will perform differently.

This is a gist with a test harness and the .cl functions I modified from uber's codebase. clinfo is attached, but in short it's a debian os with a 3090

❯ python pycl_exp.py
OpenCL for 18,001,933 in 1.48301 seconds
OpenCL for 18,001,932 in 0.15895 seconds
OpenCL for 18,001,933 in 1.35726 seconds

Would very much appreciate ideas for why the read_buffer would vary so wildly based on the input data. Recommendations for how to approach the task or debug the process? Am I missing something in the process?

Thanks for your time on this question and the library overall,
Erik

[inducer]

• Which ICD are you using? (Straight Nvidia or Nvidia via pocl (conda install pocl-cuda)?)
• What happens if you try the respective other one?

[user01]

Straight NVIDIA.

I tried conda install pocl-cuda but noticed no new platforms in clinfo or examples/dump-properties.py. Interestingly, no error messages. Conda claims pocl-cuda-1.8 is installed with the transaction, but nothing appears for platforms.

I also grabbed the pocl source and tried to build it myself (cmake -DENABLE_CUDA=ON .. etc). The build failed with a missing clang header - perhaps wrong version of clang?

I'll give it another try when I can. Thanks for your reply and help so far

[inducer]

One thing to note is that clEnqueueReadBuffer (which you point to for having varying execution times) is, in principle, doing very little work: all it is supposed to do is add an operation to a queue, a quick, amortized O(1) operation.

Now, suppose that Nvidia has some "fill level" controls on their queues (not unrealistic given that they're possibly ring buffers under the hood), some of these clEnqueueReadBuffer may need to wait for operations at the front of the queue to complete. This may serve to explain what's going on, and on its own I think isn't necessarily indication of a problem.

The Nvidia profiler is likely to offer a better and more detailed picture of what's happening. Unfortunately, the Nvidia-provided OpenCL ICD does not permit its use, for whatever reason (only Nvidia would know). pocl-cuda allows you to use the Nvidia profiler, though, so that would be my suggestion.

[isuruf]

I tried conda install pocl-cuda but noticed no new platforms in clinfo or examples/dump-properties.py. Interestingly, no error messages. Conda claims pocl-cuda-1.8 is installed with the transaction, but nothing appears for platforms.

You need to install pyopencl from conda as well. Did you install pyopencl with pip?