[core][compiled graphs] Controllably destroy CUDA events in GPUFutures

[AndyUB]

Why are these changes needed?

Currently, a GPUFuture contains a recorded CUDA event. When the GPUFuture is garbage-collected, its event is also garbage-collected, at which point cupy destroys that CUDA event.

This is problematic because the event might be destroyed after other CUDA resources. In particular, we found that the overlapping test in test_torch_tensor_dag.py consistently prints out an invalid cuda memory error that occurs during dag teardown. Our hypothesis is that: For an event, CUDA likely stores a pointer to the stream it recorded. Since the CUDA streams are destroyed before the events, the stream pointer is no longer valid when the event is destroyed.

This PR fixes the issue by caching an actor's unresolved GPUFutures in its serialization context. After the GPUFuture has been waited on, its event is manually destroyed. During teardown, the events inside all unresolved GPUFutures are destroyed before other CUDA resources.

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[AndyUB]

CC @dengwxn

[stephanie-wang]

Thanks for this fix, great find!

[hipudding]

Hi @AndyUB I'm trying to remove cupy for non-cuda accelerator(#51032 ). About destory_event, Is it safe for just deleting self._event?

    def destroy_event(self) -> None:
        """
        Destroy the CUDA event associated with this future.
        """
        if self._event is None:
            return

        del self._event # it may not necessary.
        self._event = None

I read source code in torch. When there's no reference to event. torch will destory event by destructor.