Develop an UCCL plugin for NIXL

[praveingk]

During Inference, we see the following scenarios:

1. NIXL/UCX used for P2P comms for PD disaggregation (KV Cache transfers).
2. NVSHMEM used for AlltoAll comms for MOE (https://github.com/ppl-ai/pplx-kernels). However, underlying transport is via UCX.

UCCL's transport optimization could apply to the above scenarios too.

As part of enhancement, would like to explore how UCCL can be integrated into the inference transport stack.

One solution could be to develop UCX plugin (UCT) with UCCL transport, which can solve both P2P and AllToAll for inference.

[tjtanaa]

Amazing work🚀 Looking forward to see this integrated into vLLM, LMCache, and llm-d.

[praveingk]

On Further thoughts, I feel developing a NIXL plugin is easier, and would give us an alternate path to adoption. I will change this issue title to create a UCCL plugin for NIXL.

[lengrongfu]

If we use it as a Nixl plugin, we can only provide the P2P capabilities of uccl and cannot make good use of the all2all capabilities. @praveingk Is it better to integrate it independently into vllm?

[praveingk]

I feel both approaches can have integration paths as the community evolves.
I agree we should also have a direct vllm integration path as well as documented this issue : #116 (comment) . Currently, vLLM's KV Connector is only for P2P and uses NIXL.
And, vLLM's DeepEP integration and NIXL integration are completely independent. Hence, having NIXL plugin will not limit UCCL's capabilities, it will just provide integration paths. Additionally, we need to work on UCCL integration with vllm's DeepEP independently. @YangZhou1997 Thoughts?

[praveingk]

ai-dynamo/nixl#895