[Bug][NIXL][PD]: Revisiting NIXL slowness on CUDA_IPC -- installation problem

[kouroshHakha]

Hi all, I wanted to follow up on a issue that I have sporadically fixed with help from @robertgshaw2-redhat , but not sure what the long term fix is. Happy to make PRs to get this issue out of the way, just need some agreement on what needs to be done.

In 2025-07-03 NIXL Perf OH Runbook @robertgshaw2-redhat points out a performance issue with NIXL having a transfer launch overhead and analyzes the implication of that overhead when using CUDA_IPC vs. IB. There is also a proposed fix for that issue which implies changes to both Nixl c++ layer, and changes to vLLM NIXL connector.

I have separately validated that particular fix on a performance issue that I was seeing while serving gptoss-120B with P4TP1-D2TP2 on an 8xH100. My fixes follow the proposed fixes by Robert by stitching together a nixl and vllm patch as follows:

1. Install a custom branch of Nixl (from Rob’s fork)

mkdir -p ~/default/patches && cd ~/default/patches
git clone https://github.com/Pyngon/nixl.git
cd nixl
git remote add robertgshaw2 https://github.com/robertgshaw2-redhat/nixl.git
git fetch robertgshaw2 batched-workers
git checkout robertgshaw2/batched-workers

cd ~/default/patches/nixl

# if I don't do these replacements the installation of nixl will barf
sed -i "s|ucx_backend_dep = declare_dependency(link_with: ucx_backend_lib, include_directories: \[nixl_inc_dirs, '../../../../src/plugins/ucx'\])|ucx_backend_dep = declare_dependency(link_with: [ucx_backend_lib, ucx_utils_lib], include_directories: [nixl_inc_dirs, ucx_utils_inc_dirs, '../../../../src/plugins/ucx'])|" test/unit/plugins/ucx/meson.build test/unit/plugins/ucx_mo/meson.build
meson setup build --prefix=/usr/local/nixl/ -Ducx_path=/usr/local/ucx
cd build 
ninja 
sudo ninja install
cd .. 
pip install .

2. Install a custom branch of vllm that has Rob’s fixes (+ some other fixes):

cd ~/default/patches
pip uninstall vllm -y
git clone -b kh/fix-nixl-with-batch-xfer https://github.com/kouroshHakha/vllm.git
cd vllm
VLLM_USE_PRECOMPILED=1 pip install .  --extra-index-url https://download.pytorch.org/whl/cu128

Now I want to figure out a reliable packaging of this solution, ideally upstreaming the real fix, avoiding custom patches. My understanding was that NIXL team has shipped this fix with the intention to fix this issue here. And for this to be effective, Rob made this vLLM change to NIXL connector as well.

However after upgrading to vllm 0.11.0 and NiXL 0.6.0, I see that the transmission times are in the range of seconds resulting in massive TTFT hits when we do PD in a single node. (I don’t have a multi node setup to test out IB so we are limited to CUDA_IPC for now). When I use the patches above my TTFT goes back to a reasonable range.

Settings:

• GPTOSS-120B
• P: 4xTP1 D:2xTP2
• 8xH100
• ITL: 10000, OTL: 500
• Blocksize: 128

Vllm 0.11.0, nixl 0.6.0

{"date": "20251007-131635", "endpoint_type": "vllm", "backend": "vllm", "label": null, "model_id": "openai/gpt-oss-120b", "tokenizer_id": "openai/gpt-oss-120b", "num_prompts": 512, "request_rate": 8.0, "burstiness": 1.0, "max_concurrency": 256, "duration": 173.1752845250012, "completed": 512, "total_input_tokens": 5120000, "total_output_tokens": 256000, "request_throughput": 2.9565419881038677, "request_goodput": null, "output_throughput": 1478.270994051934, "total_token_throughput": 31043.69087509061, "max_output_tokens_per_s": 2283.0, "max_concurrent_requests": 271, "mean_ttft_ms": 49437.55960683285, "median_ttft_ms": 54130.73057449947, "std_ttft_ms": 23736.198817249224, "p99_ttft_ms": 82495.83750360207, "mean_tpot_ms": 9.974323081549954, "median_tpot_ms": 10.19100517832757, "std_tpot_ms": 1.388854618770527, "p99_tpot_ms": 12.426047975852768, "mean_itl_ms": 9.977955695804779, "median_itl_ms": 10.170783993089572, "std_itl_ms": 1.9778775679690934, "p99_itl_ms": 14.450422546360633}

Patched vllm and patched nixl

{"date": "20251006-235458", "endpoint_type": "openai", "label": null, "model_id": "openai/gpt-oss-120b", "tokenizer_id": "openai/gpt-oss-120b", "num_prompts": 512, "request_rate": 8.0, "burstiness": 1.0, "max_concurrency": 256, "duration": 72.13274197399733, "completed": 512, "total_input_tokens": 5120000, "total_output_tokens": 256000, "request_throughput": 7.098024918899764, "request_goodput": null, "output_throughput": 3549.012459449882, "total_token_throughput": 74529.26164844753, "max_output_tokens_per_s": 4445.0, "max_concurrent_requests": 103, "mean_ttft_ms": 1771.6487848570637, "median_ttft_ms": 1708.5792995058, "std_ttft_ms": 766.5458062734209, "p99_ttft_ms": 3491.9522748235613, "mean_tpot_ms": 16.349380812230752, "median_tpot_ms": 16.794258085175027, "std_tpot_ms": 1.7807793309110627, "p99_tpot_ms": 19.426186372438096, "mean_itl_ms": 16.35386197637269, "median_itl_ms": 15.602655497787055, "std_itl_ms": 8.4624046859869, "p99_itl_ms": 37.73036762679111}

The new issue on VLLM 0.11.0 + NIXL 0.6.0 manifests differently than the original issue that the above patches were trying to work around. In particular, the above patches were trying to address the launch time overhead (the time it takes to call self.nixl_wrapper.transfer(handle) ) by introducing concurrency and batching. But the high ttft on vLLM now is root caused by high transmission times (not high transfer init times), so it might be a new issue? Wondering if anyone has any prior on this problem? Is my understanding right about the intended fixes? Or there are more fixes missing from the stack to reach the high performance regime.

🐛 Describe the bug

N/A

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[kouroshHakha]

Update: The output of ucx_info -d
It seems like instead of CUDA_IPC, tcp is getting used which is ~350x slower. But why?

$ /usr/local/ucx/bin/ucx_info -d
#
# Memory domain: self
#     Component: self
#             register: unlimited, cost: 0 nsec
#           remote key: 0 bytes
#           rkey_ptr is supported
#         memory types: host (access,reg_nonblock,reg,cache)
#
#      Transport: self
#         Device: memory
#           Type: loopback
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 0.00/ppn + 19360.00 MB/sec
#              latency: 0 nsec
#             overhead: 10 nsec
#            put_short: <= 4294967295
#            put_bcopy: unlimited
#            get_bcopy: unlimited
#             am_short: <= 8K
#             am_bcopy: <= 8K
#               domain: cpu
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 0 bytes
#        iface address: 8 bytes
#       error handling: ep_check
#
#
# Memory domain: tcp
#     Component: tcp
#             register: unlimited, cost: 0 nsec
#           remote key: 0 bytes
#         memory types: host (access,reg_nonblock,reg,cache)
#
#      Transport: tcp
#         Device: eth0
#           Type: network
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 1130.02/ppn + 0.00 MB/sec
#              latency: 5258 nsec
#             overhead: 50000 nsec
#            put_zcopy: <= 18446744073709551590, up to 6 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 0
#             am_short: <= 8K
#             am_bcopy: <= 8K
#             am_zcopy: <= 64K, up to 6 iov
#   am_opt_zcopy_align: <= 1
#         am_align_mtu: <= 0
#            am header: <= 8037
#           connection: to ep, to iface
#      device priority: 0
#     device num paths: 1
#              max eps: 256
#       device address: 6 bytes
#        iface address: 2 bytes
#           ep address: 10 bytes
#       error handling: peer failure, ep_check, keepalive
#
#      Transport: tcp
#         Device: lo
#           Type: network
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 11.91/ppn + 0.00 MB/sec
#              latency: 10960 nsec
#             overhead: 50000 nsec
#            put_zcopy: <= 18446744073709551590, up to 6 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 0
#             am_short: <= 8K
#             am_bcopy: <= 8K
#             am_zcopy: <= 64K, up to 6 iov
#   am_opt_zcopy_align: <= 1
#         am_align_mtu: <= 0
#            am header: <= 8037
#           connection: to ep, to iface
#      device priority: 1
#     device num paths: 1
#              max eps: 256
#       device address: 18 bytes
#        iface address: 2 bytes
#           ep address: 10 bytes
#       error handling: peer failure, ep_check, keepalive
#
#
# Connection manager: tcp
#      max_conn_priv: 2064 bytes
#
# Memory domain: sysv
#     Component: sysv
#             allocate: unlimited
#           remote key: 12 bytes
#           rkey_ptr is supported
#         memory types: host (access,alloc,cache)
#
#      Transport: sysv
#         Device: memory
#           Type: intra-node
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 0.00/ppn + 15360.00 MB/sec
#              latency: 80 nsec
#             overhead: 10 nsec
#            put_short: <= 4294967295
#            put_bcopy: unlimited
#            get_bcopy: unlimited
#             am_short: <= 100
#             am_bcopy: <= 8256
#               domain: cpu
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 16 bytes
#        iface address: 8 bytes
#       error handling: ep_check
#
#
# Memory domain: posix
#     Component: posix
#             allocate: <= 469342940K
#           remote key: 32 bytes
#           rkey_ptr is supported
#         memory types: host (access,alloc,cache)
#
#      Transport: posix
#         Device: memory
#           Type: intra-node
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 0.00/ppn + 15360.00 MB/sec
#              latency: 80 nsec
#             overhead: 10 nsec
#            put_short: <= 4294967295
#            put_bcopy: unlimited
#            get_bcopy: unlimited
#             am_short: <= 100
#             am_bcopy: <= 8256
#               domain: cpu
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 16 bytes
#        iface address: 16 bytes
#       error handling: ep_check
#
#
# Memory domain: cuda_cpy
#     Component: cuda_cpy
#             allocate: unlimited
#             register: unlimited, cost: 0 nsec
#         memory types: host (reg), cuda (access,alloc,reg,detect,dmabuf), cuda-managed (access,alloc,reg,cache,detect)
#
#      Transport: cuda_copy
#         Device: cuda
#           Type: accelerator
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 10000.00/ppn + 0.00 MB/sec
#              latency: 8000 nsec
#             overhead: 0 nsec
#            put_short: <= 4294967295
#            put_zcopy: unlimited, up to 1 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 1
#            get_short: <= 4294967295
#            get_zcopy: unlimited, up to 1 iov
#  get_opt_zcopy_align: <= 1
#        get_align_mtu: <= 1
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 0 bytes
#        iface address: 8 bytes
#       error handling: none
#
#
# Memory domain: cuda_ipc
#     Component: cuda_ipc
#             register: unlimited, cost: 0 nsec
#           remote key: 200 bytes
#           memory invalidation is supported
#         memory types: cuda (access,reg,cache)
#
#      Transport: cuda_ipc
#         Device: cuda
#           Type: intra-node
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 400000.00/ppn + 0.00 MB/sec
#              latency: 1000 nsec
#             overhead: 7000 nsec
#            put_zcopy: unlimited, up to 1 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 1
#            get_zcopy: unlimited, up to 1 iov
#  get_opt_zcopy_align: <= 1
#        get_align_mtu: <= 1
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 9 bytes
#        iface address: 4 bytes
#       error handling: peer failure
#
# < failed to open connection manager rdmacm >
#
# Memory domain: cma
#     Component: cma
#             register: unlimited, cost: 9 nsec
#         memory types: host (access,reg_nonblock,reg,cache)
#
#      Transport: cma
#         Device: memory
#           Type: intra-node
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 0.00/ppn + 11145.00 MB/sec
#              latency: 80 nsec
#             overhead: 2000 nsec
#            put_zcopy: unlimited, up to 16 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 1
#            get_zcopy: unlimited, up to 16 iov
#  get_opt_zcopy_align: <= 1
#        get_align_mtu: <= 1
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 16 bytes
#        iface address: 16 bytes
#       error handling: peer failure, ep_check
#

[mkhazraee]

Are you building ucx yourself? What flags do you use? Also is it ucx 1.19? You can check them by "ucx_info -v" and these are the necessary flags:
https://github.com/ai-dynamo/nixl/blob/main/contrib/Dockerfile#L116
TCP being slower is expected, 350x is a tad surprising, but first let's figure out what's going on with IPC not being used.

[kouroshHakha]

yes. So I have built ucx in both patched and latest versions. In fact they are the exact same ucx installation (I interactively changed the installation of nixl and vllm to test things out).

Here are the output of ucx_info -v and configure flags used:

(base) ray@k-97a284fc6563a0000:~/default$ /usr/local/ucx/bin/ucx_info -v
# Library version: 1.19.0
# Library path: /usr/local/ucx/lib/libucs.so.0
# API headers version: 1.19.0
# Git branch '', revision e463614
# Configured with: --prefix=/usr/local/ucx --enable-shared --disable-static --disable-doxygen-doc --enable-optimizations --enable-cma --enable-devel-headers --with-cuda=/usr/local/cuda --with-dm --with-gdrcopy=/usr/local/gdrcopy --with-verbs --enable-mt --with-rdmacm --with-mlx5-dv --with-ib-hw-tm --enable-experimental-api

I also tried a version without experiemtnal flag

[kouroshHakha]

ok, I think I found the fix, but not sure why it fixes things. I also document the debugging procedure for later use.

1. Find what transport layer does NIXL ends up using by setting UCX_LOG_LEVEL=debug, UCX_PROTO_INFO=y, NIXL_LOG_LEVEL=DEBUG.

The log should say something like if it is using cuda_ipc

+--------------------------------+-------------------------------------------------------------------+
 ucp_context_0 intra-node cfg#2 | remote memory read by ucp_get*(multi) into cuda/GPU0 from cuda    |
+--------------------------------+---------------------------------------------------+---------------+
                         0..inf | zero-copy                                         | cuda_ipc/cuda |
+--------------------------------+---------------------------------------------------+---------------+

But for me it was using tcp/eth0.

2. So this was my setup during test

startup: my machine comes up with the following dependencies:

vllm 0.11.0
nixl 0.3.1

change: install nixl 0.6.0 against the existing ucx -- this uses tcp/eth0

But when I install ucx 0.19.0 and then nixl 0.6.0 it used cuda_ipc. The following configuration was used for building ucx:

./configure --prefix=/usr/local/ucx --enable-shared --disable-static --disable-doxygen-doc --enable-optimizations --enable-cma --enable-devel-headers --with-cuda=/usr/local/cuda --with-dm --with-gdrcopy=/usr/local/gdrcopy --with-verbs --enable-mt --with-rdmacm --with-mlx5-dv --with-ib-hw-tm

I still cannot explain why these exact set of installation steps fix the issue.

Before the ucx re-installation:

{"date": "20251008-111801", "endpoint_type": "vllm", "backend": "vllm", "label": null, "model_id": "openai/gpt-oss-120b", "tokenizer_id": "openai/gpt-oss-120b", "num_prompts": 512, "request_rate": 8.0, "burstiness": 1.0, "max_concurrency": 256, "duration": 162.87559812299878, "completed": 512, "total_input_tokens": 5120000, "total_output_tokens": 256000, "request_throughput": 3.143503421631968, "request_goodput": null, "output_throughput": 1571.7517108159839, "total_token_throughput": 33006.785927135665, "max_output_tokens_per_s": 2476.0, "max_concurrent_requests": 271, "mean_ttft_ms": 46869.47852490596, "median_ttft_ms": 49670.854847001465, "std_ttft_ms": 23515.7753340105, "p99_ttft_ms": 80572.00382677373, "mean_tpot_ms": 10.162605589820446, "median_tpot_ms": 10.370895449897235, "std_tpot_ms": 1.6052185938325811, "p99_tpot_ms": 12.483308922727991, "mean_itl_ms": 10.164754442035681, "median_itl_ms": 10.270560498611303, "std_itl_ms": 4.72731579118775, "p99_itl_ms": 14.620294751366597}

After:

{"date": "20251008-113940", "endpoint_type": "vllm", "backend": "vllm", "label": null, "model_id": "openai/gpt-oss-120b", "tokenizer_id": "openai/gpt-oss-120b", "num_prompts": 512, "request_rate": 8.0, "burstiness": 1.0, "max_concurrency": 256, "duration": 76.1194290170024, "completed": 512, "total_input_tokens": 5120000, "total_output_tokens": 256000, "request_throughput": 6.726272209499065, "request_goodput": null, "output_throughput": 3363.1361047495325, "total_token_throughput": 70625.85819974018, "max_output_tokens_per_s": 4396.0, "max_concurrent_requests": 170, "mean_ttft_ms": 1754.1380929375664, "median_ttft_ms": 1614.6994730006554, "std_ttft_ms": 807.0948556407902, "p99_ttft_ms": 3689.3645088211633, "mean_tpot_ms": 31.48565090822647, "median_tpot_ms": 33.05343330961084, "std_tpot_ms": 4.858415231306663, "p99_tpot_ms": 36.565555808826275, "mean_itl_ms": 31.669105258861887, "median_itl_ms": 30.128834998322418, "std_itl_ms": 14.867977892564271, "p99_itl_ms": 61.78514426297624}