Use separate along-step kernel for neutral particles for 25% performance boost

[sethrj]

This results in a ~25% speedup for GPU tracks on CMS+msc+field. Sorting the along-step tracks results in a slight (5-10% per kernel) performance increase for the geometry kernels but a substantial (~7% overall execution) increase in the pre-step kernel plus a penalty (~5% overall) for the sort. Sorting by both action and post-step action further increases the overall time.

NOTE: preliminary testing suggests this is terrible for AMD hardware by default: the along-step-field time does not change for TestEm3, and the along-step-neutral is simply added on top of it. Perhaps because AMD doesn't have separate hardware counters?

[esseivaju]

Other than the diagnostic test failing, everything looks good to me

[amandalund]

Nice speedup! Looks good to me too once the test is fixed. Interestingly, for cms2018+msc+field this decreased the mean number of steps per gamma track by ~10%.

[sethrj]

@amandalund Yeah, that might be related to the change in results for the diagnostic: I'm not sure how, but the

               "geo-propagation-limit e+",
               "geo-propagation-limit e-",

actions are no longer showing up. Maybe it's a lingering issue with positrons being biased to die near geometry boundaries?

[esseivaju]

Perhaps because AMD doesn't have separate hardware counters?

I suppose you're referring to Independent thread scheduling from Volta where each thread has its own PC and SP. If my understanding is correct, even then, the execution is still SIMT, so locksteps still happen, we can just interleave divergent code paths since each thread has its own stats. I suppose it still helps if threads are stalled due to memory transfer or fetching instruction (the two most common causes of stalling we have) then we can execute the other code path? I'd still expect AMD to be faster since it has less work to do.

[amandalund]

interesting, could be... I do see the geo-propagation-limit actions when I increase the number of steps in the tests, but not sure if it's statistical or something else causing the change.