[Bug]: slot_mapping of schedulined_tokens in tpu_model_runner should be an array (not tensor). #17969
Description
Your current environment
The output of python collect_env.py
INFO 05-11 18:34:35 [__init__.py:248] Automatically detected platform tpu.
Collecting environment information...
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A
OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 4.0.0
Libc version: glibc-2.35
Python version: 3.10.16 (main, Dec 11 2024, 16:24:50) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-6.5.0-1013-gcp-x86_64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 48 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 224
On-line CPU(s) list: 0-223
Vendor ID: AuthenticAMD
Model name: AMD EPYC 7B13
CPU family: 25
Model: 1
Thread(s) per core: 2
Core(s) per socket: 56
Socket(s): 2
Stepping: 0
BogoMIPS: 4899.99
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext invpcid_single ssbd ibrs ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr arat npt nrip_save umip vaes vpclmulqdq rdpid fsrm
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 3.5 MiB (112 instances)
L1i cache: 3.5 MiB (112 instances)
L2 cache: 56 MiB (112 instances)
L3 cache: 448 MiB (14 instances)
NUMA node(s): 2
NUMA node0 CPU(s): 0-55,112-167
NUMA node1 CPU(s): 56-111,168-223
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Mitigation; safe RET
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Versions of relevant libraries:
[pip3] numpy==2.2.5
[pip3] pyzmq==26.4.0
[pip3] torch==2.8.0.dev20250430+cpu
[pip3] torch-xla==2.8.0+gita7c3e85
[pip3] torchvision==0.22.0.dev20250430+cpu
[pip3] transformers==4.51.3
[pip3] triton==3.3.0
[conda] numpy 2.2.5 pypi_0 pypi
[conda] pyzmq 26.4.0 pypi_0 pypi
[conda] torch 2.8.0.dev20250430+cpu pypi_0 pypi
[conda] torch-xla 2.8.0+gita7c3e85 pypi_0 pypi
[conda] torchvision 0.22.0.dev20250430+cpu pypi_0 pypi
[conda] transformers 4.51.3 pypi_0 pypi
[conda] triton 3.3.0 pypi_0 pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.1.dev6496+g4d66549.d20250511 (git sha: 4d66549, date: 20250511)
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
Could not collect
NCCL_CUMEM_ENABLE=0
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
🐛 Describe the bug
When running examples/offline_inference/tpu.py, came up with the following issue:
Traceback (most recent call last):
File "/home/jcgu_google_com/anaconda3/envs/nixl/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/home/xxx/anaconda3/envs/nixl/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/xxx/vllm/vllm/v1/engine/core.py", line 425, in run_engine_core
raise e
File "/home/xxx/vllm/vllm/v1/engine/core.py", line 414, in run_engine_core
engine_core.run_busy_loop()
File "/home/xxx/vllm/vllm/v1/engine/core.py", line 438, in run_busy_loop
self._process_engine_step()
File "/home/xxx/vllm/vllm/v1/engine/core.py", line 463, in _process_engine_step
outputs = self.step_fn()
File "/home/xxx/vllm/vllm/v1/engine/core.py", line 226, in step
model_output = self.execute_model(scheduler_output)
File "/home/xxx/vllm/vllm/v1/engine/core.py", line 213, in execute_model
raise err
File "/home/xxx/vllm/vllm/v1/engine/core.py", line 207, in execute_model
return self.model_executor.execute_model(scheduler_output)
File "/home/xxx/vllm/vllm/v1/executor/abstract.py", line 86, in execute_model
output = self.collective_rpc("execute_model",
File "/home/xxx/vllm/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
answer = run_method(self.driver_worker, method, args, kwargs)
File "/home/xxx/vllm/vllm/utils.py", line 2529, in run_method
return func(*args, **kwargs)
File "/home/xxx/vllm/vllm/v1/worker/tpu_worker.py", line 208, in execute_model
output = self.model_runner.execute_model(scheduler_output)
File "/home/xxx/anaconda3/envs/nixl/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/home/xxx/vllm/vllm/v1/worker/tpu_model_runner.py", line 791, in execute_model
attn_metadata, logits_indices, padded_num_reqs = self._prepare_inputs(
File "/home/xxx/vllm/vllm/v1/worker/tpu_model_runner.py", line 535, in _prepare_inputs
np.add(block_numbers * self.block_size,
TypeError: return arrays must be of ArrayType
The slot_mapping tensor of the scheduled_tokens should be converted to an array. The issue was caused by #17483.
Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.