Use same GPU stream for all kernels

[MrBurmark]

Use the same GPU stream for all kernels

Use a specific GPU stream for all Cuda/Hip kernels. This is done by using the same resource for all kernels. By default this is the RAJA default stream, but can be changed to stream 0 with the --gpu_stream_0 command line argument.

• This PR is a feature
• It does the following:
  • Adds GPU stream control as requested in Use non-default stream with gpu kernels #294

[MrBurmark]

One thing I didn't think too deeply about while making this PR is what to do with calls to cudaMemcpy (non-async). I believe those API calls run on stream 0 which isn't a correctness problem as the RAJA default stream is synchronous with stream 0, but it isn't really in the spirit of the change in this PR either. I'm thinking about adding a resource argument to some of our memory helper functions to work around this.

[rhornung67]

@MrBurmark the mem copy calls are outside of the kernel timing regions, so does it matter?

[MrBurmark]

Most of them don't matter, but there are some that are used in the timed loop in reduction kernels. I've been trying it out and it looks like there is a performance penalty for using implicitly synchronized streams compared to a single stream. I'm going to rewrite those memory calls to explicitly call cuda|hipMemcpyAsync and streamSynchronize.

[MrBurmark]

I made the change for the kernels using memcpy and synchronize calls in the timed portion. They now use memcpyAsync+streamSynchronize or streamSynchronize to use their stream.

[rhornung67]

A lot of changes for something we should have thought about earlier, huh? 😄 Thank you for working through this.

[artv3]

Wonderful!

[rhornung67]

@MrBurmark let's merge this one next when it gets through tioga CI.

Also, we need to make sure that all new kernels are following this pattern.