fix: L0_sequence_batcher_cudashm

[oandreeva-nv]

What does the PR do?

We were hiting the case, when cudaMemcpy was copying output to GPU in an async way without proper synchronization. Thus, when during the test client side was reading output result from cuda SHM, there were no guarantees that server side finished the copy. As a result, test failures.

Resolution: use cudaMemcpyAsync + sync on stream.

Documentation reference: https://docs.nvidia.com/cuda/cuda-c-programming-guide/#concurrent-execution-between-host-and-device

The following device operations are asynchronous with respect to the host:

* Kernel launches;
* Memory copies within a single device’s memory;
* Memory copies from host to device of a memory block of 64 KB or less; <------------------
* Memory copies performed by functions that are suffixed with Async;
* Memory set function calls.

Checklist

• PR title reflects the change and is of format <commit_type>: <Title>
• Changes are described in the pull request.
• Related issues are referenced.
• Populated github labels field
• Added test plan and verified test passes.
• Verified that the PR passes existing CI.
• Verified copyright is correct on all changed files.
• Added succinct git squash message before merging ref.
• All template sections are filled out.
• Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

Check the conventional commit type
box here and add the label to the github PR.

• build
• ci
• docs
• feat
• fix
• perf
• refactor
• revert
• style
• test

Related PRs:

Where should the reviewer start?

Test plan:

• CI Pipeline ID:

21063313 - relevant test: L0_sequence_batcher_cudashm--base

Caveats:

Background

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

• closes GitHub issue: #xxx