TL/MLX5: adding mcast interface

[MamziB]

What

TL/MLX5/MCAST: multicast-related changes to mlx5 files WIP

Why ?

Implementing HW MCAST in UCC

FYI @janjust @Sergei-Lebedev

[swx-jenkins3]

Can one of the admins verify this patch?

[Sergei-Lebedev]

ok to test

[janjust]

@Sergei-Lebedev how do you retrigger CI? specifically the codestyle CI

[Sergei-Lebedev]

@Sergei-Lebedev how do you retrigger CI? specifically the codestyle CI

it auto restarts after new commit or force push, note that codestyle doesn't check PR title but commit title

[MamziB]

@Sergei-Lebedev Thanks for the constructive comments. I have resolved all of them.

[samnordmann]

Hey Mamzi, thanks for the clean implementation that smoothly fits into the existing tl.
I left some minor comments, some of them might not be relevant at this stage where I can't see the full collective implementation, so feel free to tell me what you think! :)

This PR and my open PR will have quite many conflicts, we should think about how to properly merge/rebase them two.

[MamziB]

@Sergei-Lebedev @samnordmann Thank you all for the comments. I have resolved all of them. Please feel free to approve the changes.
There were also a lot of merge conflicts that needed to be addressed. Now it should be free of any conflict with the latest master branch.

@janjust FYI

[samnordmann]

Hey Mamzi! Thanks for addressing the comments, I'm approving the PR. I added another minor comment. Can you also make sure to fix the errors revealed by Linter-NVIDIA / clang-tidy check ?

[MamziB]

@samnordmann Thanks for pointing out the issue. I have updated the commit.

[manjugv]

@MamziB Since this is only part of PR. I know you will address some of this in your presentation to WG. Some design questions:

• How do you create multicast groups?
• How do you plan to support shared memory? Hierarchy?
• Do you plan to support NVLINK multicast or other SM multicast?
• What reliability protocol do you plan to use? Can you describe it?
• How do you support large messages? How do you plan to support GPU memory?

[MamziB]

@manjugv Please find the answers inline:

- How do you create multicast groups?
Mcast group is created during the TL MLX5 team creation. Rank 0 in the new team calls rdma_join_multicast() and retrieves the gid and lid of this mcast group. Then it calls bcast() to send this information to all other processes in the new team. Upon receiving this info, all can make sure that they are joining the same mcast group.

How do you plan to support shared memory? Hierarchy?
We are envisioning a two-level approach for our mcast based bcast design. Root does an Inter-node bcast using Mcast and then node-leaders use shared memory to propagate the data to all other processes in the same node.
Therefore, CL can be utilized to take advantage of TL UCP (intra-node step) + TL MLX5 (Inter-node step).

Do you plan to support NVLINK multicast or other SM multicast?
Absolutely. GPU memory bcast is on our roadmap and such technology can accelerate the intra-node step. If you know of any implementation that uses NVLINK multicast, please let us know.

What reliability protocol do you plan to use? Can you describe it?
We use a binomial tree to represent the mcast group. Each rank has max one parent and two children. Each rank sends ACK to the parent upon receiving the mcast packet and then waits until it receives two ACK packets from its children. If a rank does not receive a packet in a timely manner, it sends a NACK to its parent, and then the parent directly sends the missing packet.

How do you support large messages? How do you plan to support GPU memory?
The max mcast packet size is MTU. Therefore, for large messages, we chunk the buffer into MTU-size buffers and mcast these smaller chunks back-to-back to each other.
For GPU memory, we are planning to stage the GPU buffer into CPU memory, and then use mcast to propagate the data. To hide this overhead, the staging step of a smaller chunk can be overlapped with mcast operation of the previous packet.

[samnordmann]

Please can you return UCC_OK at team and context creation ?

[MamziB]

@samnordmann Thanks for the new comments. I just resolved all of them. @manjugv Please let me know if you have any other comments regarding this PR.

[manjugv]

Thanks, it looks good!

1. Mamzi, please add FAQ as we discussed.
2. I will let CI run, and we can merge

[manjugv]

Thanks, it looks good!

1. Mamzi, please add FAQ as we discussed.
2. I will let CI run, and we can merge

CC @Sergei-Lebedev

[MamziB]

@manjugv Thanks, I will add FAQ as suggested. I just also saw that CI tests have finished successfully.