Add CUDA/HIP MultiGPU Event Polling

[G-071]

While the [cuda,hip]_executors within HPX support MultiGPU scenarios themselves (by using a device index and cudaSetDevice), the event polling actually does not, forcing users to use the less performant callback version instead! To fix this, we need to make the CUDA/HIP event pool aware on which device the events it reuses are created, and add an index to always push/pop events on the correct device.

This PR implements this, by adding multiple event stacks (one per device) to the event pool singleton. Furthermore, the PR adds an additional device_id parameter to the appropriate get_future calls that use events. To keep things compatible with code that used previous HPX versions, 0 is used as a default where this parameter was added. For each device, there's always 128 events added in the beginning by default - in case one wants to avoid this overhead when only a single GPU is used, simply set the environment variable CUDA_VISIBLE_DEVICES=0.

[hkaiser]

@G-071 should we add a test that actually excersises this functionality, perhaps run it on the DGX partition only (e.g. enable it with a special CMake variable)?

[hkaiser]

May I suggest that you move the (HPX_CORE_EXPORTed) implementation of this function into a source file? The rationale is that otherwise on some platforms (Mac, Windows) each executable module (shared library, executable) that will contain code calling this function will have its own instance of the event_pool_ variable.

[G-071]

I have added a few changes/additions/fixes:

• The definition of the method get_event_pool is now in cuda_event.cpp. @hkaiser : Is this what you had in mind?
• I have added a test (cuda_multi_device_polling.cpp) which tests executors on different devices. This tests queries the number of devices, creates an executor per device and runs a dummy kernel on each executor. If the multi device polling does not work, the futures for those dummy kernel calls will never become ready and the test will time out. As it adapts to the number of devices present, it will test with all 8 GPUs on the DGX node. The test also verifies that detail::get_future_with_event uses the currently active device (previousely set in the user code via cudaSetDevice) by default if no other device is given.
• The thread that initializes the event pool now restores its original device ID afterwards. Without that, the thread used a different device ID that the other ones afterward (if multiple devices are present), causing problems in user code that only used 1 GPU. This is only done for the original initialization of the event pools for all devices. Subsequent events that are added on-demand for a device pool use the device ID that is passed by the user (either via detail::get_future_with_event directly or via the cuda executor). Meaning, if we get a non-zero device ID, the user code handles the setting of the devices already and we will simply use the passed device ID accordingly (saving us a bit of overhead by avoiding to call cudaGetDevice and cudaSetDevice). Edit: On second thought, with
  46b477a this is now also done when adding events on-demand in pop. Makes it more difficult to use it the wrong way, and the overhead seems negligible after all.
• Format fixes.

Overall, this seems to work fine on the NVIDIA nodes I have tested in on so far (using Octo-Tiger).

[hkaiser]

@G-071 could you please fix the inspect errors reported? After that I would be able to go ahead and merge the PR.

[G-071]

@hkaiser Done! I just forgot about the noascii check (which does not like my name due to the ß and thus complains about me being in the author list of a file) and was missing a few headers.

[hkaiser]

bors merge

[bors]

This PR was included in a batch that successfully built, but then failed to merge into master. It will not be retried.

Additional information:

Response status code: 422
{"message":"Changes must be made through a pull request.","documentation_url":"https://docs.github.com/articles/about-protected-branches"}