[V1][Core] using cached vocab_size for Structured Outputs #14630
Conversation
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
There was a problem hiding this comment.
Could use len(tokenizer), which is cached: https://github.com/vllm-project/vllm/blob/main/vllm/transformers_utils/tokenizer.py#L101
Or it might be better to use tokenizer.max_token_id. I have seen cases where the vocab size is actually larger than the number of tokens in the tokenizer since some were removed.
There was a problem hiding this comment.
Will need @Ubospica confirmation here.
|
Will add tests once #14625 is merged |
aarnphm
force-pushed
the
v1/molmo-aria-support-vocab
branch
2 times, most recently
from
8fa2aa6 to
a53c058
Compare
aarnphm
changed the title
|
This pull request has merge conflicts that must be resolved before it can be |
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
aarnphm
force-pushed
the
v1/molmo-aria-support-vocab
branch
from
689278a to
8872481
Compare
| lora_config=self.vllm_config.lora_config) # type: ignore[arg-type] | ||
| tokenizer_group.ping() | ||
|
|
||
| tokenizer = tokenizer_group.get_lora_tokenizer(None) |
There was a problem hiding this comment.
Not directly related to this PR but it looks like structured output currently isn't compatible with custom vocab LoRAs.
There was a problem hiding this comment.
Sg, will add to the list for compatibility
Fwiw i don't think it ever worked with LoRA
When testing with V1 structured output + Llama-3.1-8B-Instruct, the changes made in vllm-project#14630 broke for me. I get the error: ``` ERROR 03-14 15:44:42 [core.py:337] File "/home/rbryant/vllm/vllm/v1/structured_output/__init__.py", line 77, in _delayed_init ERROR 03-14 15:44:42 [core.py:337] tokenizer_info = xgr.TokenizerInfo.from_huggingface( ERROR 03-14 15:44:42 [core.py:337] File "/home/rbryant/vllm/venv/lib/python3.10/site-packages/xgrammar/tokenizer_info.py", line 184, in from_huggingface ERROR 03-14 15:44:42 [core.py:337] raise ValueError(msg) ERROR 03-14 15:44:42 [core.py:337] ValueError: Input vocab_size less than minimum viable vocab size for tokenizer <class 'vllm.transformers_utils.tokenizer.get_cached_tokenizer.<locals>.CachedTokenizer'>. ERROR 03-14 15:44:42 [core.py:337] ``` The vocab size was off by one. The max token ID is not == the vocab size, since 0 is also a token ID. It's the max token ID + 1. Signed-off-by: Russell Bryant <rbryant@redhat.com>
…ct#14630) Signed-off-by: Aaron Pham <contact@aarnphm.xyz> Signed-off-by: Richard Liu <ricliu@google.com>
…ct#14630) Signed-off-by: Aaron Pham <contact@aarnphm.xyz> Signed-off-by: Louis Ulmer <ulmerlouis@gmail.com>
…ct#14630) Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
…ct#14630) Signed-off-by: Aaron Pham <contact@aarnphm.xyz> Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
2 participants
Previously, we obtained vocab_size for xgrammar from hf_text_config directly.
However, in the recent version of xgrammar, the detected vocab_size now include special_tokens, in which raises the issue found in #14534
By calculating the vocab size, it ensures supporting custom tokenizers with the like of Olmo, etc.