[Bug]: vLLM (AsyncLLMEngine, LLM) engine initialization fails when using runai_streamer

[jiangwu300]

Your current environment

The output of python collect_env.py

Collecting environment information...
==============================
        System Info
==============================
OS                           : Ubuntu 22.04.5 LTS (x86_64)
GCC version                  : (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version                : Could not collect
CMake version                : Could not collect
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.7.1+cu126
Is debug build               : False
CUDA used to build PyTorch   : 12.6
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.11.11 | packaged by conda-forge | (main, Mar  3 2025, 20:43:55) [GCC 13.3.0] (64-bit runtime)
Python platform              : Linux-6.1.140-154.222.amzn2023.x86_64-x86_64-with-glibc2.35

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.8.93
CUDA_MODULE_LOADING set to   : LAZY
GPU models and configuration : 
GPU 0: NVIDIA H200
GPU 1: NVIDIA H200
GPU 2: NVIDIA H200
GPU 3: NVIDIA H200
GPU 4: NVIDIA H200
GPU 5: NVIDIA H200
GPU 6: NVIDIA H200
GPU 7: NVIDIA H200

Nvidia driver version        : 570.133.20
cuDNN version                : Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.8.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.8.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.8.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.8.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.8.0
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.8.0
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.8.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.8.0
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Address sizes:                           46 bits physical, 48 bits virtual
Byte Order:                              Little Endian
CPU(s):                                  192
On-line CPU(s) list:                     0-191
Vendor ID:                               GenuineIntel
Model name:                              Intel(R) Xeon(R) Platinum 8488C
CPU family:                              6
Model:                                   143
Thread(s) per core:                      2
Core(s) per socket:                      48
Socket(s):                               2
Stepping:                                8
BogoMIPS:                                4800.00
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 wbnoinvd ida arat avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq rdpid cldemote movdiri movdir64b md_clear serialize amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities
Hypervisor vendor:                       KVM
Virtualization type:                     full
L1d cache:                               4.5 MiB (96 instances)
L1i cache:                               3 MiB (96 instances)
L2 cache:                                192 MiB (96 instances)
L3 cache:                                210 MiB (2 instances)
NUMA node(s):                            2
NUMA node0 CPU(s):                       0-47,96-143
NUMA node1 CPU(s):                       48-95,144-191
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Vulnerability Srbds:                     Not affected
Vulnerability Tsx async abort:           Not affected

==============================
Versions of relevant libraries
==============================
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.6.4.1
[pip3] nvidia-cuda-cupti-cu12==12.6.80
[pip3] nvidia-cuda-nvrtc-cu12==12.6.77
[pip3] nvidia-cuda-runtime-cu12==12.6.77
[pip3] nvidia-cudnn-cu12==9.5.1.17
[pip3] nvidia-cufft-cu12==11.3.0.4
[pip3] nvidia-cufile-cu12==1.11.1.6
[pip3] nvidia-curand-cu12==10.3.7.77
[pip3] nvidia-cusolver-cu12==11.7.1.2
[pip3] nvidia-cusparse-cu12==12.5.4.2
[pip3] nvidia-cusparselt-cu12==0.6.3
[pip3] nvidia-nccl-cu12==2.26.2
[pip3] nvidia-nvjitlink-cu12==12.6.85
[pip3] nvidia-nvtx-cu12==12.6.77
[pip3] pyzmq==27.0.1
[pip3] torch==2.7.1
[pip3] torchaudio==2.7.1
[pip3] torchvision==0.22.1
[pip3] transformers==4.55.0
[pip3] triton==3.3.1
[conda] numpy                                 1.26.4           pypi_0              pypi
[conda] nvidia-cublas-cu12                    12.6.4.1         pypi_0              pypi
[conda] nvidia-cuda-cupti-cu12                12.6.80          pypi_0              pypi
[conda] nvidia-cuda-nvrtc-cu12                12.6.77          pypi_0              pypi
[conda] nvidia-cuda-runtime-cu12              12.6.77          pypi_0              pypi
[conda] nvidia-cudnn-cu12                     9.5.1.17         pypi_0              pypi
[conda] nvidia-cufft-cu12                     11.3.0.4         pypi_0              pypi
[conda] nvidia-cufile-cu12                    1.11.1.6         pypi_0              pypi
[conda] nvidia-curand-cu12                    10.3.7.77        pypi_0              pypi
[conda] nvidia-cusolver-cu12                  11.7.1.2         pypi_0              pypi
[conda] nvidia-cusparse-cu12                  12.5.4.2         pypi_0              pypi
[conda] nvidia-cusparselt-cu12                0.6.3            pypi_0              pypi
[conda] nvidia-nccl-cu12                      2.26.2           pypi_0              pypi
[conda] nvidia-nvjitlink-cu12                 12.6.85          pypi_0              pypi
[conda] nvidia-nvtx-cu12                      12.6.77          pypi_0              pypi
[conda] pyzmq                                 27.0.1           py311hc251a9f_0     conda-forge
[conda] torch                                 2.7.1            pypi_0              pypi
[conda] torchaudio                            2.7.1            pypi_0              pypi
[conda] torchvision                           0.22.1           pypi_0              pypi
[conda] transformers                          4.55.0           pypi_0              pypi
[conda] triton                                3.3.1            pypi_0              pypi

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
Neuron SDK Version           : N/A
vLLM Version                 : 0.10.0
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV18    NV18    NV18    NV18    NV18    NV18    NV18    0-47,96-143     0               N/A
GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18    0-47,96-143     0               N/A
GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18    0-47,96-143     0               N/A
GPU3    NV18    NV18    NV18     X      NV18    NV18    NV18    NV18    0-47,96-143     0               N/A
GPU4    NV18    NV18    NV18    NV18     X      NV18    NV18    NV18    48-95,144-191   1               N/A
GPU5    NV18    NV18    NV18    NV18    NV18     X      NV18    NV18    48-95,144-191   1               N/A
GPU6    NV18    NV18    NV18    NV18    NV18    NV18     X      NV18    48-95,144-191   1               N/A
GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18     X      48-95,144-191   1               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
NVIDIA_VISIBLE_DEVICES=all
NVIDIA_REQUIRE_CUDA=cuda>=12.8 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=560,driver<561 brand=grid,driver>=560,driver<561 brand=tesla,driver>=560,driver<561 brand=nvidia,driver>=560,driver<561 brand=quadro,driver>=560,driver<561 brand=quadrortx,driver>=560,driver<561 brand=nvidiartx,driver>=560,driver<561 brand=vapps,driver>=560,driver<561 brand=vpc,driver>=560,driver<561 brand=vcs,driver>=560,driver<561 brand=vws,driver>=560,driver<561 brand=cloudgaming,driver>=560,driver<561 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566
NCCL_VERSION=2.25.1-1
NVIDIA_DRIVER_CAPABILITIES=compute,utility
NVIDIA_PRODUCT_NAME=CUDA
CUDA_VERSION=12.8.1
LD_LIBRARY_PATH=/usr/local/cuda/lib64
NCCL_CUMEM_ENABLE=0
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

🐛 Describe the bug

When I use vllm serve with runai_streamer, everything works as expected, however, trying to initialize the LLM or AsyncLLMEngine in python code with runai_streamer will lead to engine failure because it seems like it doesn't download the config.json/other model files to the tmp model directory created. It properly uses runai_streamer to first stream the model from s3, and then we get a failure regarding the HuggingFace validation of the tmp directory, and after some digging it is related to that directory not having the config.json. I think we need to similarly download the s3 files into the tmp directory like it is being done for online serve. Refer to the following code snippet and error:

from  vllm import SamplingParams, LLM, AsyncEngineArgs, AsyncLLMEngine
tests = ["hello what is your name?"]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
kwargs = {
    "model": "s3://path/to/model/qwen3-32b/",
    "tensor_parallel_size": 8,
    "max_model_len": 20000,
    "load_format": "runai_streamer",
}
engine_args = AsyncEngineArgs(
    **kwargs,
)

# Trying to launch AsyncLLMEngine fails
engine = AsyncLLMEngine.from_engine_args(engine_args)

## Trying to launch LLM also fails
llm = LLM(model="s3://path/to/model/qwen3-32b/", load_format="runai_streamer",
                       tensor_parallel_size=8, max_model_len=20000)

outputs = llm.generate(tests)

print(outputs)

INFO 08-13 10:16:08 [core.py:193] init engine (profile, create kv cache, warmup model) took 83.10 seconds
ERROR 08-13 10:16:08 [core.py:632] EngineCore failed to start.
ERROR 08-13 10:16:08 [core.py:632] Traceback (most recent call last):
ERROR 08-13 10:16:08 [core.py:632]   File "/home/ray/anaconda3/lib/python3.11/site-packages/transformers/utils/hub.py", line 479, in cached_files
ERROR 08-13 10:16:08 [core.py:632]     hf_hub_download(
ERROR 08-13 10:16:08 [core.py:632]   File "/home/ray/anaconda3/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 106, in _inner_fn
ERROR 08-13 10:16:08 [core.py:632]     validate_repo_id(arg_value)
ERROR 08-13 10:16:08 [core.py:632]   File "/home/ray/anaconda3/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 154, in validate_repo_id
ERROR 08-13 10:16:08 [core.py:632]     raise HFValidationError(
ERROR 08-13 10:16:08 [core.py:632] huggingface_hub.errors.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/tmp/tmpzrwp_51y'. Use `repo_type` argument if needed.
ERROR 08-13 10:16:08 [core.py:632] 
ERROR 08-13 10:16:08 [core.py:632] During handling of the above exception, another exception occurred:
ERROR 08-13 10:16:08 [core.py:632] 
ERROR 08-13 10:16:08 [core.py:632] Traceback (most recent call last):
ERROR 08-13 10:16:08 [core.py:632]   File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 623, in run_engine_core
ERROR 08-13 10:16:08 [core.py:632]     engine_core = EngineCoreProc(*args, **kwargs)
ERROR 08-13 10:16:08 [core.py:632]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-13 10:16:08 [core.py:632]   File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 441, in __init__
ERROR 08-13 10:16:08 [core.py:632]     super().__init__(vllm_config, executor_class, log_stats,
ERROR 08-13 10:16:08 [core.py:632]   File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 93, in __init__
ERROR 08-13 10:16:08 [core.py:632]     self.structured_output_manager = StructuredOutputManager(vllm_config)
ERROR 08-13 10:16:08 [core.py:632]                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-13 10:16:08 [core.py:632]   File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/v1/structured_output/__init__.py", line 51, in __init__
ERROR 08-13 10:16:08 [core.py:632]     self.tokenizer = init_tokenizer_from_configs(
ERROR 08-13 10:16:08 [core.py:632]                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-13 10:16:08 [core.py:632]   File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/transformers_utils/tokenizer_group.py", line 111, in init_tokenizer_from_configs
ERROR 08-13 10:16:08 [core.py:632]     return TokenizerGroup(
ERROR 08-13 10:16:08 [core.py:632]            ^^^^^^^^^^^^^^^
ERROR 08-13 10:16:08 [core.py:632]   File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/transformers_utils/tokenizer_group.py", line 24, in __init__
ERROR 08-13 10:16:08 [core.py:632]     self.tokenizer = get_tokenizer(self.tokenizer_id, **tokenizer_config)
ERROR 08-13 10:16:08 [core.py:632]                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-13 10:16:08 [core.py:632]   File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/transformers_utils/tokenizer.py", line 259, in get_tokenizer
ERROR 08-13 10:16:08 [core.py:632]     raise e
ERROR 08-13 10:16:08 [core.py:632]   File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/transformers_utils/tokenizer.py", line 238, in get_tokenizer
ERROR 08-13 10:16:08 [core.py:632]     tokenizer = AutoTokenizer.from_pretrained(
ERROR 08-13 10:16:08 [core.py:632]                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-13 10:16:08 [core.py:632]   File "/home/ray/anaconda3/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 1049, in from_pretrained
ERROR 08-13 10:16:08 [core.py:632]     tokenizer_config = get_tokenizer_config(pretrained_model_name_or_path, **kwargs)
ERROR 08-13 10:16:08 [core.py:632]                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-13 10:16:08 [core.py:632]   File "/home/ray/anaconda3/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 881, in get_tokenizer_config
ERROR 08-13 10:16:08 [core.py:632]     resolved_config_file = cached_file(
ERROR 08-13 10:16:08 [core.py:632]                            ^^^^^^^^^^^^
ERROR 08-13 10:16:08 [core.py:632]   File "/home/ray/anaconda3/lib/python3.11/site-packages/transformers/utils/hub.py", line 321, in cached_file
ERROR 08-13 10:16:08 [core.py:632]     file = cached_files(path_or_repo_id=path_or_repo_id, filenames=[filename], **kwargs)
ERROR 08-13 10:16:08 [core.py:632]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-13 10:16:08 [core.py:632]   File "/home/ray/anaconda3/lib/python3.11/site-packages/transformers/utils/hub.py", line 531, in cached_files
ERROR 08-13 10:16:08 [core.py:632]     resolved_files = [
ERROR 08-13 10:16:08 [core.py:632]                      ^
ERROR 08-13 10:16:08 [core.py:632]   File "/home/ray/anaconda3/lib/python3.11/site-packages/transformers/utils/hub.py", line 532, in <listcomp>
ERROR 08-13 10:16:08 [core.py:632]     _get_cache_file_to_return(path_or_repo_id, filename, cache_dir, revision, repo_type)
ERROR 08-13 10:16:08 [core.py:632]   File "/home/ray/anaconda3/lib/python3.11/site-packages/transformers/utils/hub.py", line 144, in _get_cache_file_to_return
ERROR 08-13 10:16:08 [core.py:632]     resolved_file = try_to_load_from_cache(
ERROR 08-13 10:16:08 [core.py:632]                     ^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-13 10:16:08 [core.py:632]   File "/home/ray/anaconda3/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 106, in _inner_fn
ERROR 08-13 10:16:08 [core.py:632]     validate_repo_id(arg_value)
ERROR 08-13 10:16:08 [core.py:632]   File "/home/ray/anaconda3/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 154, in validate_repo_id
ERROR 08-13 10:16:08 [core.py:632]     raise HFValidationError(
ERROR 08-13 10:16:08 [core.py:632] huggingface_hub.errors.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/tmp/tmpzrwp_51y'. Use `repo_type` argument if needed.
ERROR 08-13 10:16:09 [multiproc_executor.py:140] Worker proc VllmWorker-1 died unexpectedly, shutting down executor.
Process EngineCore_0:
Traceback (most recent call last):
  File "/home/ray/anaconda3/lib/python3.11/site-packages/transformers/utils/hub.py", line 479, in cached_files
    hf_hub_download(
  File "/home/ray/anaconda3/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 106, in _inner_fn
    validate_repo_id(arg_value)
  File "/home/ray/anaconda3/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 154, in validate_repo_id
    raise HFValidationError(
huggingface_hub.errors.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/tmp/tmpzrwp_51y'. Use `repo_type` argument if needed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ray/anaconda3/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/ray/anaconda3/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 636, in run_engine_core
    raise e
  File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 623, in run_engine_core
    engine_core = EngineCoreProc(*args, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 441, in __init__
    super().__init__(vllm_config, executor_class, log_stats,
  File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 93, in __init__
    self.structured_output_manager = StructuredOutputManager(vllm_config)
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/v1/structured_output/__init__.py", line 51, in __init__
    self.tokenizer = init_tokenizer_from_configs(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/transformers_utils/tokenizer_group.py", line 111, in init_tokenizer_from_configs
    return TokenizerGroup(
           ^^^^^^^^^^^^^^^
  File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/transformers_utils/tokenizer_group.py", line 24, in __init__
    self.tokenizer = get_tokenizer(self.tokenizer_id, **tokenizer_config)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/transformers_utils/tokenizer.py", line 259, in get_tokenizer
    raise e
  File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/transformers_utils/tokenizer.py", line 238, in get_tokenizer
    tokenizer = AutoTokenizer.from_pretrained(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ray/anaconda3/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 1049, in from_pretrained
    tokenizer_config = get_tokenizer_config(pretrained_model_name_or_path, **kwargs)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ray/anaconda3/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 881, in get_tokenizer_config
    resolved_config_file = cached_file(
                           ^^^^^^^^^^^^
  File "/home/ray/anaconda3/lib/python3.11/site-packages/transformers/utils/hub.py", line 321, in cached_file
    file = cached_files(path_or_repo_id=path_or_repo_id, filenames=[filename], **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ray/anaconda3/lib/python3.11/site-packages/transformers/utils/hub.py", line 531, in cached_files
    resolved_files = [
                     ^
  File "/home/ray/anaconda3/lib/python3.11/site-packages/transformers/utils/hub.py", line 532, in <listcomp>
    _get_cache_file_to_return(path_or_repo_id, filename, cache_dir, revision, repo_type)
  File "/home/ray/anaconda3/lib/python3.11/site-packages/transformers/utils/hub.py", line 144, in _get_cache_file_to_return
    resolved_file = try_to_load_from_cache(
                    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ray/anaconda3/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 106, in _inner_fn
    validate_repo_id(arg_value)
  File "/home/ray/anaconda3/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 154, in validate_repo_id
    raise HFValidationError(
huggingface_hub.errors.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/tmp/tmpzrwp_51y'. Use `repo_type` argument if needed.
Traceback (most recent call last):
  File "/home/ray/test.py", line 15, in <module>
    engine = AsyncLLMEngine.from_engine_args(engine_args)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/engine/async_llm_engine.py", line 653, in from_engine_args
    return async_engine_cls.from_vllm_config(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/v1/engine/async_llm.py", line 163, in from_vllm_config
    return cls(
           ^^^^
  File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/v1/engine/async_llm.py", line 117, in __init__
    self.engine_core = EngineCoreClient.make_async_mp_client(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/v1/engine/core_client.py", line 98, in make_async_mp_client
    return AsyncMPClient(*client_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/v1/engine/core_client.py", line 677, in __init__
    super().__init__(
  File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/v1/engine/core_client.py", line 408, in __init__
    with launch_core_engines(vllm_config, executor_class,
  File "/home/ray/anaconda3/lib/python3.11/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/v1/engine/utils.py", line 697, in launch_core_engines
    wait_for_engine_startup(
  File "/home/ray/anaconda3/lib/python3.11/site-packages/vllm/v1/engine/utils.py", line 750, in wait_for_engine_startup
    raise RuntimeError("Engine core initialization failed. "
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_0': 1}

t

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[graysonk546]

same, I'm seeing this issue as well!

[lengrongfu]

/path/to/model/qwen3-32b/ this host local path? if this local paht, why need use s3:// this use type, I don't understand this usage, could you give me some information? @jiangwu300

[jiangwu300]

@lengrongfu I store my models in s3, and I want to use runai_streamer to load it in from s3, and that's when we run into this bug. Loading from s3 using runai_streamer breaks the engine initialization.

[pwschuurman]

I tested with other combinations of vLLM / RunAI:

• v0.9.2 / runai-model-streamer=0.13.2,0.14.0
• v0.10.0 / runai-model-streamer=0.13.4,0.14.0

Initialization fails with these combinations. So this doesn't appear to be a recent regression. It appears that this method of model loading via the Python API may have some integration challenges.

So even with the fix for #23236, this issue will still need a change in vLLM, and potentially in the model streamer library as well.

[DarkLight1337]

Is this issue fixed by #23845?

[pwschuurman]

Is this issue fixed by #23845?

No, not fixed. From what I can tell this issue #22843 is pre-existing (engine initialization fails on versions prior to #21345), and needs further investigation

[lengrongfu]

The root cause is https://github.com/vllm-project/vllm/pull/23842/files#diff-dc4afb08b7a54bb237ee02180225d41d542166342c1f09d41978b7893f332dd0 this code updte. we need from s3 load config.json file.

[jiangwu300]

@DarkLight1337 can we assign someone to do the code fix suggested by @lengrongfu ? I've already made a PR for Ray that fixes the issue from the Ray end (ray-project/ray#55662), but still need this fixed on vllm side to use with Ray batch inference. Happy to submit a PR myself but I would just be copy pasting lengronfu's code so maybe he should take credit for this.