repeat for Julia 1.6 and higher

[torfjelde]

This PR adds an implementation of Base.repeat for arrays of arbitrary dimensionality. It unfortunately makes use of Base._RepeatInnerOuter which is only available on ≥ 1.6, but AFAIK this simplifies the implementation (plus my motivation was usage on Julia v1.6).

I haven't looked into an implementation for < 1.6, but guessing we can at least re-use the kernel for those too if we want to support earlier versions.

[maleadt]

GPUArrays is 1.6+ only, so that shouldn't matter 🙂

[torfjelde]

GPUArrays is 1.6+ only, so that shouldn't matter

I thought this was the case because of the recent work you have done on GPUCompiler, but doesn't Project.toml says 1.5?

[torfjelde]

Tests are passing locally now, and I dropped the check for Julia v1.6 :)

[maleadt]

Oh whoops, I was confusing with GPUCompiler. Yeah we can probably just bump it to 1.6 here then.

[torfjelde]

Dope:) Did so now.

[maleadt]

Looking good. I'm still concerned by the multiple reads of the input data though. Would it not be better to flip the inner and outer repetition kernels around, launching one thread per element of the input, and having a for loop in the kernel looping over every output element corresponding to this input? That would also avoid the costly division that happens now to calculate the src_inds.

One potential issue is when the array to repeat is very small. A possible solution here would be to pass in the number of elements each thread has to process, so that it would still be possible to launch one thread per output element (with the number of element to process then being one). This is similar to https://developer.nvidia.com/blog/cuda-pro-tip-write-flexible-kernels-grid-stride-loops/, and would allow for a heuristic to determine a configuration (a simple one would be to ensure we have a couple of 100s of threads).

[torfjelde]

I'm still concerned by the multiple reads of the input data though. Would it not be better to flip the inner and outer repetition kernels around, launching one thread per element of the input, and having a for loop in the kernel looping over every output element corresponding to this input? That would also avoid the costly division that happens now to calculate the src_inds.

This was my first "naive" attempt, but then I thought "Given the parallizable nature of GPUs, surely it's better to parallelize over the largest of the two, i.e. output, rather than using a sequential for-loop?". But this might very likely be the wrong intuition as I have no idea how expensive memory-reads/thread spawns/etc. vs loops are on GPU:)

One potential issue is when the array to repeat is very small. A possible solution here would be to pass in the number of elements each thread has to process, so that it would still be possible to launch one thread per output element (with the number of element to process then being one). This is similar to https://developer.nvidia.com/blog/cuda-pro-tip-write-flexible-kernels-grid-stride-loops/, and would allow for a heuristic to determine a configuration (a simple one would be to ensure we have a couple of 100s of threads).

Lemme read this and I'll get back to you!

[mcabbott]

It should also be possible to implement repeat in terms of broadcasting. This gist has an attempt (which probably needs to be checked & tested):
https://gist.github.com/mcabbott/80ac43cca3bee8f57809155a5240519f
I wonder how this compares to this PR's version for speed?

[maleadt]

Superseded by #400