[Bug]: Prefix cache feature does not work with the Ascend Scheduler

[Yikun]

Your current environment

[Bug]: Prefix cache feature does not work with the Ascend Scheduler

🐛 Describe the bug

[Bug]: Prefix cache feature does not work with the Ascend Scheduler

[MengqingCao]

Reproduction script:

import os
import time
from vllm import LLM, SamplingParams

os.environ["VLLM_USE_V1"] = "1"

# A prompt containing a large markdown table. The table is randomly generated by GPT-4.
LONG_PROMPT = "You are a helpful assistant in recognizes the content of tables in markdown format. Here is a table as follows.\n# Table\n" + """
| ID  | Name          | Age | Occupation    | Country       | Email                  | Phone Number   | Address                       |
|-----|---------------|-----|---------------|---------------|------------------------|----------------|------------------------------|
| 1   | John Doe      | 29  | Engineer      | USA           | john.doe@example.com   | 555-1234       | 123 Elm St, Springfield, IL  |
| 2   | Jane Smith    | 34  | Doctor        | Canada        | jane.smith@example.com | 555-5678       | 456 Oak St, Toronto, ON      |
| 3   | Alice Johnson | 27  | Teacher       | UK            | alice.j@example.com    | 555-8765       | 789 Pine St, London, UK      |
| 4   | Bob Brown     | 45  | Artist        | Australia     | bob.b@example.com      | 555-4321       | 321 Maple St, Sydney, NSW    |
| 5   | Carol White   | 31  | Scientist     | New Zealand   | carol.w@example.com    | 555-6789       | 654 Birch St, Wellington, NZ |
| 6   | Dave Green    | 28  | Lawyer        | Ireland       | dave.g@example.com     | 555-3456       | 987 Cedar St, Dublin, IE     |
| 7   | Emma Black    | 40  | Musician      | USA           | emma.b@example.com     | 555-1111       | 246 Ash St, New York, NY     |
| 8   | Frank Blue    | 37  | Chef          | Canada        | frank.b@example.com    | 555-2222       | 135 Spruce St, Vancouver, BC |
| 9   | Grace Yellow  | 50  | Engineer      | UK            | grace.y@example.com    | 555-3333       | 864 Fir St, Manchester, UK   |
| 10  | Henry Violet  | 32  | Artist        | Australia     | henry.v@example.com    | 555-4444       | 753 Willow St, Melbourne, VIC|
| 11  | Irene Orange  | 26  | Scientist     | New Zealand   | irene.o@example.com    | 555-5555       | 912 Poplar St, Auckland, NZ  |
| 12  | Jack Indigo   | 38  | Teacher       | Ireland       | jack.i@example.com     | 555-6666       | 159 Elm St, Cork, IE         |
| 13  | Karen Red     | 41  | Lawyer        | USA           | karen.r@example.com    | 555-7777       | 357 Cedar St, Boston, MA     |
| 14  | Leo Brown     | 30  | Chef          | Canada        | leo.b@example.com      | 555-8888       | 246 Oak St, Calgary, AB      |
| 15  | Mia Green     | 33  | Musician      | UK            | mia.g@example.com      | 555-9999       | 975 Pine St, Edinburgh, UK   |
| 16  | Noah Yellow   | 29  | Doctor        | Australia     | noah.y@example.com     | 555-0000       | 864 Birch St, Brisbane, QLD  |
| 17  | Olivia Blue   | 35  | Engineer      | New Zealand   | olivia.b@example.com   | 555-1212       | 753 Maple St, Hamilton, NZ   |
| 18  | Peter Black   | 42  | Artist        | Ireland       | peter.b@example.com    | 555-3434       | 912 Fir St, Limerick, IE     |
| 19  | Quinn White   | 28  | Scientist     | USA           | quinn.w@example.com    | 555-5656       | 159 Willow St, Seattle, WA   |
| 20  | Rachel Red    | 31  | Teacher       | Canada        | rachel.r@example.com   | 555-7878       | 357 Poplar St, Ottawa, ON    |
| 21  | Steve Green   | 44  | Lawyer        | UK            | steve.g@example.com    | 555-9090       | 753 Elm St, Birmingham, UK   |
| 22  | Tina Blue     | 36  | Musician      | Australia     | tina.b@example.com     | 555-1213       | 864 Cedar St, Perth, WA      |
| 23  | Umar Black    | 39  | Chef          | New Zealand   | umar.b@example.com     | 555-3435       | 975 Spruce St, Christchurch, NZ|
| 24  | Victor Yellow | 43  | Engineer      | Ireland       | victor.y@example.com   | 555-5657       | 246 Willow St, Galway, IE    |
| 25  | Wendy Orange  | 27  | Artist        | USA           | wendy.o@example.com    | 555-7879       | 135 Elm St, Denver, CO       |
| 26  | Xavier Green  | 34  | Scientist     | Canada        | xavier.g@example.com   | 555-9091       | 357 Oak St, Montreal, QC     |
| 27  | Yara Red      | 41  | Teacher       | UK            | yara.r@example.com     | 555-1214       | 975 Pine St, Leeds, UK       |
| 28  | Zack Blue     | 30  | Lawyer        | Australia     | zack.b@example.com     | 555-3436       | 135 Birch St, Adelaide, SA   |
| 29  | Amy White     | 33  | Musician      | New Zealand   | amy.w@example.com      | 555-5658       | 159 Maple St, Wellington, NZ |
| 30  | Ben Black     | 38  | Chef          | Ireland       | ben.b@example.com      | 555-7870       | 246 Fir St, Waterford, IE    |
"""


def get_generation_time(llm, sampling_params, prompts):
    # time the generation
    start_time = time.time()
    output = llm.generate(prompts, sampling_params=sampling_params)
    end_time = time.time()
    # print the output and generation time
    print(f"Output: {output[0].outputs[0].text}")
    print(f"Generation time: {end_time - start_time} seconds.")


# set enable_prefix_caching=True to enable APC
llm = LLM(
    model='/home/xxx/cache/modelscope/models/Qwen/Qwen2___5-7B-Instruct',
    enable_prefix_caching=True,
    additional_config={"ascend_scheduler_config": {
        "enabled": True,
        "enable_prefix_caching": True,
    },},
    enforce_eager=True,
)

sampling_params = SamplingParams(temperature=0, max_tokens=100)

# Querying the age of John Doe
get_generation_time(
    llm,
    sampling_params,
    LONG_PROMPT + "Question: what is the age of John Doe? Your answer: The age of John Doe is ",
)
get_generation_time(
    llm,
    sampling_params,
    LONG_PROMPT + "Question: what is the age of John Doe? Your answer: The age of John Doe is ",
)
# Querying the age of Zack Blue
# This query will be faster since vllm avoids computing the KV cache of LONG_PROMPT again.
get_generation_time(
    llm,
    sampling_params,
    LONG_PROMPT + "Question: what is the age of Zack Blue? Your answer: The age of Zack Blue is ",
)
get_generation_time(
    llm,
    sampling_params,
    LONG_PROMPT + "Question: what is the age of Zack Blue? Your answer: The age of Zack Blue is ",
)

Error Report:

(atb) (base) xxx@xxx-docker:~/code/vllm-ascend$ python examples/demo/apc.py 
INFO 06-23 03:18:30 [__init__.py:39] Available plugins for group vllm.platform_plugins:
INFO 06-23 03:18:30 [__init__.py:41] - ascend -> vllm_ascend:register
INFO 06-23 03:18:30 [__init__.py:41] - ascend -> vllm_ascend:register
INFO 06-23 03:18:30 [__init__.py:44] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 06-23 03:18:30 [__init__.py:235] Platform plugin ascend is activated
WARNING 06-23 03:18:36 [_custom_ops.py:21] Failed to import from vllm._C with ImportError('libnuma.so.1: cannot open shared object file: No such file or directory')
INFO 06-23 03:18:38 [importing.py:63] Triton not installed or not compatible; certain GPU-related functions will not be available.
WARNING 06-23 03:18:41 [registry.py:405] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_mtp:CustomDeepSeekMTP.
WARNING 06-23 03:18:41 [registry.py:405] Model architecture Qwen2VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_vl:AscendQwen2VLForConditionalGeneration.
WARNING 06-23 03:18:41 [registry.py:405] Model architecture Qwen2_5_VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_5_vl:AscendQwen2_5_VLForConditionalGeneration.
WARNING 06-23 03:18:41 [registry.py:405] Model architecture DeepseekV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_v2:CustomDeepseekV2ForCausalLM.
WARNING 06-23 03:18:41 [registry.py:405] Model architecture DeepseekV3ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_v2:CustomDeepseekV3ForCausalLM.
WARNING 06-23 03:18:41 [registry.py:405] Model architecture Qwen3MoeForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen3_moe:CustomQwen3MoeForCausalLM.
INFO 06-23 03:19:02 [config.py:831] This model supports multiple tasks: {'generate', 'score', 'classify', 'reward', 'embed'}. Defaulting to 'generate'.
INFO 06-23 03:19:02 [config.py:1444] Using max model len 32768
INFO 06-23 03:19:02 [config.py:2188] Chunked prefill is enabled with max_num_batched_tokens=8192.
INFO 06-23 03:19:02 [platform.py:164] Compilation disabled, using eager mode by default
INFO 06-23 03:19:03 [core.py:459] Waiting for init message from front-end.
INFO 06-23 03:19:03 [platform.py:164] Compilation disabled, using eager mode by default
INFO 06-23 03:19:03 [core.py:69] Initializing a V1 LLM engine (v0.9.2.dev131+g6e9cc73f6) with config: model='/home/xxx/cache/modelscope/models/Qwen/Qwen2___5-7B-Instruct', speculative_config=None, tokenizer='/home/xxx/cache/modelscope/models/Qwen/Qwen2___5-7B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=True, kv_cache_dtype=auto,  device_config=npu, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=/home/xxx/cache/modelscope/models/Qwen/Qwen2___5-7B-Instruct, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=False, use_async_output_proc=True, pooler_config=None, compilation_config={"level":0,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":["all"],"splitting_ops":[],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":0,"cudagraph_capture_sizes":[],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":0,"local_cache_dir":null}
WARNING 06-23 03:19:03 [utils.py:2753] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes not implemented in <vllm_ascend.worker.worker_v1.NPUWorker object at 0xfffd13dadc30>
INFO 06-23 03:19:05 [parallel_state.py:1072] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
INFO 06-23 03:19:05 [model_runner_v1.py:1829] Starting to load model /home/xxx/cache/modelscope/models/Qwen/Qwen2___5-7B-Instruct...
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:00<00:01,  1.87it/s]
Loading safetensors checkpoint shards:  50% Completed | 2/4 [00:01<00:01,  1.51it/s]
Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:02<00:00,  1.43it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:02<00:00,  1.38it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:02<00:00,  1.43it/s]

INFO 06-23 03:19:10 [default_loader.py:272] Loading weights took 3.09 seconds
INFO 06-23 03:19:11 [model_runner_v1.py:1848] Loading model weights took 14.2488 GB
INFO 06-23 03:19:19 [kv_cache_utils.py:716] GPU KV cache size: 729,344 tokens
INFO 06-23 03:19:19 [kv_cache_utils.py:720] Maximum concurrency for 32,768 tokens per request: 22.26x
INFO 06-23 03:19:19 [core.py:172] init engine (profile, create kv cache, warmup model) took 8.06 seconds
WARNING 06-23 03:19:20 [core.py:102] Using configured V1 scheduler class vllm_ascend.core.scheduler.AscendScheduler. This scheduler interface is not public and compatibility may not be maintained.
Adding requests: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 43.15it/s]
Processed prompts: 100%|████████████████████████████████████████████████| 1/1 [00:03<00:00,  3.49s/it, est. speed input: 463.02 toks/s, output: 26.97 toks/s]
Output: 29. That's correct! According to the table, John Doe's age is 29. Here's the relevant row from the table:

| 1   | John Doe      | 29  | Engineer      | USA           | john.doe@example.com   | 555-1234       | 123 Elm St, Springfield, IL  |

So, the age of John Doe is indeed 29.
Generation time: 3.513195037841797 seconds.
Adding requests: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 164.22it/s]
Processed prompts:   0%|                                                           | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]mki_log delete old file:/home/xxx/atb/log/atb_1400996_20250403081732.log
[rank0]:[E623 03:19:24.233198752 compiler_depend.ts:429] SelfAttentionOperation setup failed!
Exception raised from OperationSetup at build/third_party/op-plugin/op_plugin/CMakeFiles/op_plugin_atb.dir/compiler_depend.ts:117 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0xb8 (0xffff95dbb908 in /home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x6c (0xffff95d6a404 in /home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: atb::OperationSetup(atb::VariantPack, atb::Operation*, atb::Context*) + 0xc8 (0xfffe8baa1078 in /home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch_npu/lib/libop_plugin_atb.so)
frame #3: <unknown function> + 0x37118 (0xfffe8baa1118 in /home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch_npu/lib/libop_plugin_atb.so)
frame #4: <unknown function> + 0x1891810 (0xfffdee08c810 in /home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #5: <unknown function> + 0x7ce724 (0xfffdecfc9724 in /home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #6: <unknown function> + 0x7d0954 (0xfffdecfcb954 in /home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #7: <unknown function> + 0x7cd2c4 (0xfffdecfc82c4 in /home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #8: <unknown function> + 0x4c9e4c (0xffff95df8e4c in /home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #9: <unknown function> + 0x7624 (0xffffa07d6624 in /lib/aarch64-linux-gnu/libpthread.so.0)
frame #10: <unknown function> + 0xd162c (0xffffa065a62c in /lib/aarch64-linux-gnu/libc.so.6)

ERROR 06-23 03:19:24 [dump_input.py:69] Dumping input data for V1 LLM engine (v0.9.2.dev131+g6e9cc73f6) with config: model='/home/xxx/cache/modelscope/models/Qwen/Qwen2___5-7B-Instruct', speculative_config=None, tokenizer='/home/xxx/cache/modelscope/models/Qwen/Qwen2___5-7B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=True, kv_cache_dtype=auto,  device_config=npu, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=/home/xxx/cache/modelscope/models/Qwen/Qwen2___5-7B-Instruct, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=False, use_async_output_proc=True, pooler_config=None, compilation_config={"level":0,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":["all"],"splitting_ops":[],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":0,"cudagraph_capture_sizes":[],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":0,"local_cache_dir":null}, 
ERROR 06-23 03:19:24 [dump_input.py:76] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[NewRequestData(req_id=1,prompt_token_ids_len=1614,mm_inputs=[],mm_hashes=[],mm_positions=[],sampling_params=SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.0, top_p=1.0, top_k=0, min_p=0.0, seed=None, stop=[], stop_token_ids=[151643], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=100, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None),block_ids=([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 15],),num_computed_tokens=1536,lora_request=None)], scheduled_cached_reqs=[], num_scheduled_tokens={1: 78}, total_num_scheduled_tokens=78, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={}, num_common_prefix_blocks=[13], finished_req_ids=[], free_encoder_input_ids=[], structured_output_request_ids={}, grammar_bitmask=null, kv_connector_metadata=null)
ERROR 06-23 03:19:24 [core.py:521] EngineCore encountered a fatal error.
ERROR 06-23 03:19:24 [core.py:521] Traceback (most recent call last):
ERROR 06-23 03:19:24 [core.py:521]   File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 512, in run_engine_core
ERROR 06-23 03:19:24 [core.py:521]     engine_core.run_busy_loop()
ERROR 06-23 03:19:24 [core.py:521]   File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 539, in run_busy_loop
ERROR 06-23 03:19:24 [core.py:521]     self._process_engine_step()
ERROR 06-23 03:19:24 [core.py:521]   File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 564, in _process_engine_step
ERROR 06-23 03:19:24 [core.py:521]     outputs, model_executed = self.step_fn()
ERROR 06-23 03:19:24 [core.py:521]   File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 235, in step
ERROR 06-23 03:19:24 [core.py:521]     model_output = self.execute_model(scheduler_output)
ERROR 06-23 03:19:24 [core.py:521]   File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 221, in execute_model
ERROR 06-23 03:19:24 [core.py:521]     raise err
ERROR 06-23 03:19:24 [core.py:521]   File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 212, in execute_model
ERROR 06-23 03:19:24 [core.py:521]     return self.model_executor.execute_model(scheduler_output)
ERROR 06-23 03:19:24 [core.py:521]   File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/executor/abstract.py", line 87, in execute_model
ERROR 06-23 03:19:24 [core.py:521]     output = self.collective_rpc("execute_model",
ERROR 06-23 03:19:24 [core.py:521]   File "/home/xxx/code/vllm-cpu/vllm/vllm/executor/uniproc_executor.py", line 57, in collective_rpc
ERROR 06-23 03:19:24 [core.py:521]     answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 06-23 03:19:24 [core.py:521]   File "/home/xxx/code/vllm-cpu/vllm/vllm/utils.py", line 2687, in run_method
ERROR 06-23 03:19:24 [core.py:521]     return func(*args, **kwargs)
ERROR 06-23 03:19:24 [core.py:521]   File "/home/xxx/code/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 178, in execute_model
ERROR 06-23 03:19:24 [core.py:521]     output = self.model_runner.execute_model(scheduler_output)
ERROR 06-23 03:19:24 [core.py:521]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 06-23 03:19:24 [core.py:521]     return func(*args, **kwargs)
ERROR 06-23 03:19:24 [core.py:521]   File "/home/xxx/code/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1472, in execute_model
ERROR 06-23 03:19:24 [core.py:521]     aux_hidden_states) = (self._process_reqs(scheduler_output,
ERROR 06-23 03:19:24 [core.py:521]   File "/home/xxx/code/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1164, in _process_reqs
ERROR 06-23 03:19:24 [core.py:521]     hidden_states = self.model(
ERROR 06-23 03:19:24 [core.py:521]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
ERROR 06-23 03:19:24 [core.py:521]     return self._call_impl(*args, **kwargs)
ERROR 06-23 03:19:24 [core.py:521]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
ERROR 06-23 03:19:24 [core.py:521]     return forward_call(*args, **kwargs)
ERROR 06-23 03:19:24 [core.py:521]   File "/home/xxx/code/vllm-cpu/vllm/vllm/model_executor/models/qwen2.py", line 477, in forward
ERROR 06-23 03:19:24 [core.py:521]     hidden_states = self.model(input_ids, positions, intermediate_tensors,
ERROR 06-23 03:19:24 [core.py:521]   File "/home/xxx/code/vllm-cpu/vllm/vllm/compilation/decorators.py", line 173, in __call__
ERROR 06-23 03:19:24 [core.py:521]     return self.forward(*args, **kwargs)
ERROR 06-23 03:19:24 [core.py:521]   File "/home/xxx/code/vllm-cpu/vllm/vllm/model_executor/models/qwen2.py", line 354, in forward
ERROR 06-23 03:19:24 [core.py:521]     hidden_states, residual = layer(
ERROR 06-23 03:19:24 [core.py:521]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
ERROR 06-23 03:19:24 [core.py:521]     return self._call_impl(*args, **kwargs)
ERROR 06-23 03:19:24 [core.py:521]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
ERROR 06-23 03:19:24 [core.py:521]     return forward_call(*args, **kwargs)
ERROR 06-23 03:19:24 [core.py:521]   File "/home/xxx/code/vllm-cpu/vllm/vllm/model_executor/models/qwen2.py", line 253, in forward
ERROR 06-23 03:19:24 [core.py:521]     hidden_states = self.self_attn(
ERROR 06-23 03:19:24 [core.py:521]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
ERROR 06-23 03:19:24 [core.py:521]     return self._call_impl(*args, **kwargs)
ERROR 06-23 03:19:24 [core.py:521]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
ERROR 06-23 03:19:24 [core.py:521]     return forward_call(*args, **kwargs)
ERROR 06-23 03:19:24 [core.py:521]   File "/home/xxx/code/vllm-cpu/vllm/vllm/model_executor/models/qwen2.py", line 183, in forward
ERROR 06-23 03:19:24 [core.py:521]     attn_output = self.attn(q, k, v)
ERROR 06-23 03:19:24 [core.py:521]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
ERROR 06-23 03:19:24 [core.py:521]     return self._call_impl(*args, **kwargs)
ERROR 06-23 03:19:24 [core.py:521]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
ERROR 06-23 03:19:24 [core.py:521]     return forward_call(*args, **kwargs)
ERROR 06-23 03:19:24 [core.py:521]   File "/home/xxx/code/vllm-cpu/vllm/vllm/attention/layer.py", line 235, in forward
ERROR 06-23 03:19:24 [core.py:521]     self.impl.forward(self,
ERROR 06-23 03:19:24 [core.py:521]   File "/home/xxx/code/vllm-ascend/vllm_ascend/attention/attention_v1.py", line 411, in forward
ERROR 06-23 03:19:24 [core.py:521]     ori_output[:, :, :] = output[:num_tokens, :, :]
ERROR 06-23 03:19:24 [core.py:521] RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is SelfAttentionOperation.
ERROR 06-23 03:19:24 [core.py:521] Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, pleace set the environment variable ASCEND_LAUNCH_BLOCKING=1.
ERROR 06-23 03:19:24 [core.py:521] Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging.
ERROR 06-23 03:19:24 [core.py:521] [ERROR] 2025-06-23-03:19:24 (PID:1919961, Device:0, RankID:-1) ERR00100 PTA call acl api failed.
ERROR 06-23 03:19:24 [core.py:521] 
Process EngineCore_0:
Traceback (most recent call last):
  File "/home/xxx/code/vllm-ascend/examples/demo/apc.py", line 73, in <module>
    get_generation_time(
  File "/home/xxx/code/vllm-ascend/examples/demo/apc.py", line 47, in get_generation_time
Traceback (most recent call last):
    output = llm.generate(prompts, sampling_params=sampling_params)
  File "/home/xxx/code/vllm-cpu/vllm/vllm/utils.py", line 1274, in inner
  File "/home/xxx/miniconda3/envs/atb/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/xxx/miniconda3/envs/atb/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 523, in run_engine_core
    raise e
  File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 512, in run_engine_core
    engine_core.run_busy_loop()
  File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 539, in run_busy_loop
    self._process_engine_step()
  File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 564, in _process_engine_step
    outputs, model_executed = self.step_fn()
  File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 235, in step
    model_output = self.execute_model(scheduler_output)
  File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 221, in execute_model
    raise err
  File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 212, in execute_model
    return self.model_executor.execute_model(scheduler_output)
  File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/executor/abstract.py", line 87, in execute_model
    output = self.collective_rpc("execute_model",
  File "/home/xxx/code/vllm-cpu/vllm/vllm/executor/uniproc_executor.py", line 57, in collective_rpc
    answer = run_method(self.driver_worker, method, args, kwargs)
  File "/home/xxx/code/vllm-cpu/vllm/vllm/utils.py", line 2687, in run_method
    return func(*args, **kwargs)
  File "/home/xxx/code/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 178, in execute_model
    output = self.model_runner.execute_model(scheduler_output)
  File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/home/xxx/code/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1472, in execute_model
    aux_hidden_states) = (self._process_reqs(scheduler_output,
  File "/home/xxx/code/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1164, in _process_reqs
    hidden_states = self.model(
  File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/xxx/code/vllm-cpu/vllm/vllm/model_executor/models/qwen2.py", line 477, in forward
    hidden_states = self.model(input_ids, positions, intermediate_tensors,
  File "/home/xxx/code/vllm-cpu/vllm/vllm/compilation/decorators.py", line 173, in __call__
    return self.forward(*args, **kwargs)
  File "/home/xxx/code/vllm-cpu/vllm/vllm/model_executor/models/qwen2.py", line 354, in forward
    hidden_states, residual = layer(
  File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/xxx/code/vllm-cpu/vllm/vllm/model_executor/models/qwen2.py", line 253, in forward
    hidden_states = self.self_attn(
  File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/xxx/code/vllm-cpu/vllm/vllm/model_executor/models/qwen2.py", line 183, in forward
    attn_output = self.attn(q, k, v)
  File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/xxx/code/vllm-cpu/vllm/vllm/attention/layer.py", line 235, in forward
    self.impl.forward(self,
  File "/home/xxx/code/vllm-ascend/vllm_ascend/attention/attention_v1.py", line 411, in forward
    ori_output[:, :, :] = output[:num_tokens, :, :]
RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is SelfAttentionOperation.
Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, pleace set the environment variable ASCEND_LAUNCH_BLOCKING=1.
Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging.
[ERROR] 2025-06-23-03:19:24 (PID:1919961, Device:0, RankID:-1) ERR00100 PTA call acl api failed.

    return fn(*args, **kwargs)
  File "/home/xxx/code/vllm-cpu/vllm/vllm/entrypoints/llm.py", line 494, in generate
    outputs = self._run_engine(use_tqdm=use_tqdm)
  File "/home/xxx/code/vllm-cpu/vllm/vllm/entrypoints/llm.py", line 1550, in _run_engine
    step_outputs = self.llm_engine.step()
  File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/llm_engine.py", line 232, in step
    outputs = self.engine_core.get_output()
  File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core_client.py", line 625, in get_output
    raise self._format_exception(outputs) from None
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
[ERROR] 2025-06-23-03:19:24 (PID:1919585, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception
Processed prompts:   0%|                                                           | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]

related pr: #1297

[Yikun]

Closed by: #1446