[Bug]: Unexpected Special Tokens in prompt_logprobs Output for Llama3 Prompt

[leejamesss]

Your current environment

The output of `python collect_env.py`

PyTorch version: 2.3.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
Clang version: Could not collect
CMake version: version 3.29.0
Libc version: glibc-2.31

Python version: 3.8.19 (default, Mar 20 2024, 19:58:24) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.4.0-162-generic-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: 11.6.124
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA A40
GPU 1: NVIDIA A40
GPU 2: NVIDIA A40
GPU 3: NVIDIA A40
GPU 4: NVIDIA A40
GPU 5: NVIDIA A40
GPU 6: NVIDIA A40
GPU 7: NVIDIA A40

Nvidia driver version: 525.105.17
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 52 bits physical, 57 bits virtual
CPU(s): 128
On-line CPU(s) list: 0-127
Thread(s) per core: 2
Core(s) per socket: 32
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 106
Model name: Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz
Stepping: 6
Frequency boost: enabled
CPU MHz: 900.000
CPU max MHz: 3200.0000
CPU min MHz: 800.0000
BogoMIPS: 4000.00
Virtualization: VT-x
L1d cache: 3 MiB
L1i cache: 2 MiB
L2 cache: 80 MiB
L3 cache: 96 MiB
NUMA node0 CPU(s): 0-31,64-95
NUMA node1 CPU(s): 32-63,96-127
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Retbleed: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 invpcid_single ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local wbnoinvd dtherm ida arat pln pts avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq rdpid md_clear pconfig flush_l1d arch_capabilities

Versions of relevant libraries:
[pip3] numpy==1.24.4
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] torch==2.3.0
[pip3] torch_geometric==2.5.2
[pip3] torchaudio==2.3.0
[pip3] triton==2.3.0
[pip3] vllm-nccl-cu12==2.18.1.0.4.0
[conda] numpy 1.24.4 pypi_0 pypi
[conda] nvidia-nccl-cu12 2.20.5 pypi_0 pypi
[conda] torch 2.3.0 pypi_0 pypi
[conda] torch-geometric 2.5.2 pypi_0 pypi
[conda] torchaudio 2.3.0 pypi_0 pypi
[conda] triton 2.3.0 pypi_0 pypi
[conda] vllm-nccl-cu12 2.18.1.0.4.0 pypi_0 pypiROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.4.2
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 CPU Affinity NUMA Affinity
GPU0 X NV4 PXB PXB SYS SYS SYS SYS 0-31,64-95 0
GPU1 NV4 X PXB PXB SYS SYS SYS SYS 0-31,64-95 0
GPU2 PXB PXB X NV4 SYS SYS SYS SYS 0-31,64-95 0
GPU3 PXB PXB NV4 X SYS SYS SYS SYS 0-31,64-95 0
GPU4 SYS SYS SYS SYS X NV4 PXB PXB 32-63,96-127 1
GPU5 SYS SYS SYS SYS NV4 X PXB PXB 32-63,96-127 1
GPU6 SYS SYS SYS SYS PXB PXB X NV4 32-63,96-127 1
GPU7 SYS SYS SYS SYS PXB PXB NV4 X 32-63,96-127 1

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

🐛 Describe the bug

I've encountered an issue where the Llama3 model's special tokens are causing errors when attempting to access prompt_logprobs. The current implementation of the code is as follows:

from vllm import LLM, SamplingParams
from transformers import AutoTokenizer
llm = LLM(model='/data/pretrain/meta-llama/Meta-Llama-3-8B-Instruct',gpu_memory_utilization=0.8,tensor_parallel_size=1,swap_space=4,trust_remote_code=True)

# Get the tokenizer and format method for the current model
tokenizer = AutoTokenizer.from_pretrained('/data/pretrain/meta-llama/Meta-Llama-3-8B-Instruct')

sampling_params = SamplingParams(temperature=0,max_tokens=1,logprobs=None,prompt_logprobs=1)

llm.set_tokenizer(tokenizer)

q3_input = "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou are a helpful and meticulous assistant.\n\nYou are provided with one or more contexts that may contain evidence to help you arrive at the answer. Answer the given question directly and briefly, starting with \"Answer:\".<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nContext1: Great Lakes The Great Lakes (), also called the Laurentian Great Lakes and the Great Lakes of North America, are a series of interconnected freshwater lakes located primarily in the upper mid-east region of North America, on the Canada\u2013United States border, which connect to the Atlantic Ocean through the Saint Lawrence River. They consist of Lakes Superior, Michigan, Huron, Erie, and Ontario, although hydrologically, there are four lakes, Superior, Erie, Ontario, and Michigan-Huron. The lakes are interconnected by the Great Lakes Waterway. The Great Lakes are the largest group of freshwater lakes on Earth by total area, and second largest\nQuestion: where do the great lakes meet the ocean<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nAnswer: the Saint Lawrence River<|begin_of_text|><|start_header_id|>"

outputs = llm.generate(prompt_token_ids =[tokenizer.encode(q3_input)], sampling_params = sampling_params)

print(outputs[0].prompt_logprobs)

The inputs are formatted in llama3 prompt format.However, instead of the expected log probabilities, the output contains a multitude of unexpected special tokens, as illustrated below (truncated for brevity):

{42679: Logprob(logprob=-0.0002244459028588608, rank=1, decoded_token='<|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|> Lakes')}, {10164: Logprob(logprob=-0.11475807428359985, rank=1, decoded_token='<|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|><|start_header_id|> Water')}, 

There are a lot of unexpected special tokens here,which cause error when calculating the probs.

This issue persists even when switching to the Llama-2-13b-chat-hf model, with a similar code structure producing similarly erroneous outputs:

from vllm import LLM, SamplingParams
from transformers import AutoTokenizer
llm = LLM(model='/data/pretrain/llama/Llama-2-13b-chat-hf',gpu_memory_utilization=0.8,tensor_parallel_size=1,swap_space=4,trust_remote_code=True)

# Get the tokenizer and format method for the current model
tokenizer = AutoTokenizer.from_pretrained('/data/pretrain/llama/Llama-2-13b-chat-hf')

sampling_params = SamplingParams(temperature=0,max_tokens=1,logprobs=None,prompt_logprobs=1)

llm.set_tokenizer(tokenizer)

q3_input = "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou are a helpful and meticulous assistant.\n\nYou are provided with one or more contexts that may contain evidence to help you arrive at the answer. Answer the given question directly and briefly, starting with \"Answer:\".<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nContext1: Great Lakes The Great Lakes (), also called the Laurentian Great Lakes and the Great Lakes of North America, are a series of interconnected freshwater lakes located primarily in the upper mid-east region of North America, on the Canada\u2013United States border, which connect to the Atlantic Ocean through the Saint Lawrence River. They consist of Lakes Superior, Michigan, Huron, Erie, and Ontario, although hydrologically, there are four lakes, Superior, Erie, Ontario, and Michigan-Huron. The lakes are interconnected by the Great Lakes Waterway. The Great Lakes are the largest group of freshwater lakes on Earth by total area, and second largest\nQuestion: where do the great lakes meet the ocean<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nAnswer: the Saint Lawrence River<|begin_of_text|><|start_header_id|>"

outputs = llm.generate(prompt_token_ids =[tokenizer.encode(q3_input)], sampling_params = sampling_params)

print(outputs[0].prompt_logprobs)

Output snippet:

{6672: Logprob(logprob=-0.00134151556994766, rank=1, decoded_token='\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nheader')}, {29918: Logprob(logprob=-2.145764938177308e-06, rank=1, decoded_token='\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n_')}, {333: Logprob(logprob=-3.2186455882765586e-06, rank=1, decoded_token='\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nid')}, {29989: Logprob(logprob=-2.2053474822314456e-05, rank=1, decoded_token='\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n|')},

It appears that the prompt_logprobs is not functioning as intended with the special tokens for both Llama3 and Llama2-13b-chat-hf models. Any guidance or assistance in resolving this issue would be greatly appreciated.

Steps to Reproduce:

• Initialize the LLM model with the specified parameters.
• Set up the tokenizer and sampling parameters.
• Encode the input prompt and generate the output.
• Attempt to print prompt_logprobs.

Expected Behavior:
The prompt_logprobs should return the log probabilities for the generated tokens without any unexpected special tokens.

Actual Behavior:
The prompt_logprobs returns a log probability dictionary with numerous unexpected special tokens, causing errors in further processing.

Additional Notes:

• The issue occurs with both Llama3 and Llama2-13b-chat-hf models.
• The provided code snippets and outputs are truncated for readability but represent the actual issue.
• Thank you for your attention to this matter.

[DarkLight1337]

#4688 might solve your issue, can you try that out?

[leejamesss]

#4688 might solve your issue, can you try that out?

Thank you for your prompt attention to the issue. However, upon careful consideration, it seems that the problem we are facing is distinct from the one addressed in issue #4688. The concern raised in #4688 is related to an additional repetition of the Beginning of Sequence (BOS) token, which is particularly relevant in the context of API interactions. In contrast, our scenario involves offline inference and does not exhibit the same pattern of BOS token repetition as described in the aforementioned issue.
Therefore, the solution or workaround suggested in issue #4688 may not be directly applicable to the problem we are encountering with the prompt_logprobs output and the proliferation of unexpected special tokens.

I appreciate your effort to provide a potential fix, and I will continue to explore alternative solutions to resolve the unexpected token issue we are experiencing. Should there be any further insights or suggestions you can offer, they would be most welcome.

Thank you once again for your assistance with this matter.

[DarkLight1337]

I'm currently investigating a similar issue in #4200. It seems that there is something wrong with the detokenizing logic where new_decoded_token_text gets pre-padded with extra whitespace characters. @Yard1 @njhill do you have any idea about this?

Edit It seems that my particular issue is related to the chat template. @DreamGenX 's issue would be more relevant to this case.

vllm/vllm/transformers_utils/detokenizer.py

Line 115 in 52f8107

(new_tokens, new_decoded_token_text, prefix_offset,

[DreamGenX]

@DarkLight1337 This sounds related to #4577 -- something between 0.4.0.post1 and 0.4.1 changed the way tokenization works. I am for whatever reason getting back a sequence of tokens like <, <|, <|im_ etc. instead of the whole <|im_start|> at once.

[leejamesss]

Hello @DarkLight1337, @DreamGenX, and everyone involved in this discussion,

Thank you for your ongoing investigation into the tokenization and detokenization logic within the vLLM project. I understand that there may be related issues, such as #4200 and #4577, which are being looked into.

However, for the issue at hand, which is #4772, I would like to clarify that our primary concern is not with the detokenization process or the formatting of the output. We are not using the detokenized text and, therefore, any irregularities in the detokenization are not relevant to our use case.

Our focus is on the correctness of the prompt_logprobs output. We are encountering a significant number of unexpected special tokens in the log probability dictionary, which is causing errors in our downstream processing. The presence of these tokens is not expected and is interfering with the intended functionality of the LLM model.

To reiterate, we need assistance in ensuring that the prompt_logprobs output is accurate and free from these unexpected special tokens for both the Llama3 and Llama2-13b-chat-hf models.

If there are any updates, insights, or suggestions on how to address this specific issue with the log probabilities, we would greatly appreciate the guidance.

Thank you for your attention to this matter, and we look forward to a resolution.

Best regards,
@leejamesss