[Bug]: vLLM recognize the bge-m3-korean (embedding model) max length, 512 tokens.

[bnuazz15]

Your current environment

vLLM 0.10.2

🐛 Describe the bug

bge-m3-korean config.json
{

"architectures": [
"XLMRobertaModel"
],
"attention_probs_dropout_prob": 0.1,
"bos_token_id": 0,
"classifier_dropout": null,
"eos_token_id": 2,
"hidden_act": "gelu",
"hidden_dropout_prob": 0.1,
"hidden_size": 1024,
"initializer_range": 0.02,
"intermediate_size": 4096,
"layer_norm_eps": 1e-05,
"max_position_embeddings": 8194,
"model_type": "xlm-roberta",
"num_attention_heads": 16,
"num_hidden_layers": 24,
"output_past": true,
"pad_token_id": 1,
"position_embedding_type": "absolute",
"torch_dtype": "float32",
"transformers_version": "4.42.4",
"type_vocab_size": 1,
"use_cache": true,
"vocab_size": 250002
}

vLLM recognize the bge-m3-korean (embedding model) max length, 512 tokens despite of the config.json above.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[noooop]

PTAL #19201

Please find a way to modify "model_max_length": 512 in tokenizer_config.json

https://huggingface.co/upskyy/bge-m3-korean/blob/main/tokenizer_config.json

Otherwise, other models such as BAAI/bge-reranker-base will report an error

---

By the way, I'm not sure the fix for #19201 is the most appropriate way

Welcome to discuss

[bnuazz15]

hello, @noooop

I tried to modify tokenizer_config.json and tokenizer.json, but vLLM shows the log below if I add vLLM param max_model_length=8192.
ValueError: User-specified max_model_len (8192) is greater than the derived max_model_len (max_position_embeddings=512 or model_max_length=None in model's config.json)

I modifed tokenizer.json because of truncation config!

"truncation": {
"direction": "Right",
"max_length": 512,
"strategy": "LongestFirst",
"stride": 0
},

[noooop]

modify here

https://huggingface.co/upskyy/bge-m3-korean/blob/main/sentence_bert_config.json

[bnuazz15]

@noooop

it works! thanks :)

why this file affect the max length token??
is it for bert model?

[noooop]

If you want to know all the details

vllm/vllm/config/model.py

Lines 1768 to 1917 in 9360d34

	def _get_and_verify_max_len(
	hf_config: PretrainedConfig,
	tokenizer_config: Optional[dict],
	max_model_len: Optional[int],
	disable_sliding_window: bool,
	sliding_window: Optional[int],
	spec_target_max_model_len: Optional[int] = None,
	encoder_config: Optional[Any] = None,
	) -> int:
	"""Get and verify the model's maximum length."""
	derived_max_model_len = float("inf")
	possible_keys = [
	# OPT
	"max_position_embeddings",
	# GPT-2
	"n_positions",
	# MPT
	"max_seq_len",
	# ChatGLM2
	"seq_length",
	# Command-R
	"model_max_length",
	# Whisper
	"max_target_positions",
	# Others
	"max_sequence_length",
	"max_seq_length",
	"seq_len",
	]
	# Choose the smallest "max_length" from the possible keys
	max_len_key = None
	for key in possible_keys:
	max_len = getattr(hf_config, key, None)
	if max_len is not None:
	max_len_key = key if max_len < derived_max_model_len \
	else max_len_key
	derived_max_model_len = min(derived_max_model_len, max_len)
	# For Command-R / Cohere, Cohere2 / Aya Vision models
	if tmp_max_len := getattr(hf_config, "model_max_length", None):
	max_len_key = "model_max_length"
	derived_max_model_len = tmp_max_len
	# If sliding window is manually disabled, max_length should be less
	# than the sliding window length in the model config.
	if (disable_sliding_window and sliding_window is not None
	and sliding_window < derived_max_model_len):
	max_len_key = "sliding_window"
	derived_max_model_len = sliding_window
	# Consider model_max_length in tokenizer_config
	if tokenizer_config:
	tokenizer_model_max_length = tokenizer_config.get(
	"model_max_length", derived_max_model_len)
	derived_max_model_len = min(derived_max_model_len,
	tokenizer_model_max_length)
	# If none of the keys were found in the config, use a default and
	# log a warning.
	if derived_max_model_len == float("inf"):
	if max_model_len is not None:
	# If max_model_len is specified, we use it.
	return max_model_len
	if spec_target_max_model_len is not None:
	# If this is a speculative draft model, we use the max model len
	# from the target model.
	return spec_target_max_model_len
	default_max_len = 2048
	logger.warning(
	"The model's config.json does not contain any of the following "
	"keys to determine the original maximum length of the model: "
	"%s. Assuming the model's maximum length is %d.", possible_keys,
	default_max_len)
	derived_max_model_len = default_max_len
	rope_scaling = getattr(hf_config, "rope_scaling", None)
	# NOTE(woosuk): Gemma3's max_model_len (128K) is already scaled by RoPE
	# scaling, so we skip applying the scaling factor again.
	if rope_scaling is not None and "gemma3" not in hf_config.model_type:
	# No need to consider "type" key because of patch_rope_scaling when
	# loading HF config
	rope_type = rope_scaling["rope_type"]
	if rope_type not in ("su", "longrope", "llama3"):
	if disable_sliding_window:
	# TODO(robertgshaw): Find a model that supports rope_scaling
	# with sliding window to see if this case should be allowed.
	raise NotImplementedError(
	"Disabling sliding window is not supported for models "
	"with rope_scaling. Please raise an issue so we can "
	"investigate.")
	# NOTE: rope_type == "default" does not define factor
	# https://github.com/huggingface/transformers/blob/v4.45.2/src/transformers/modeling_rope_utils.py
	scaling_factor = rope_scaling.get("factor", 1.0)
	if rope_type == "yarn":
	derived_max_model_len = rope_scaling[
	"original_max_position_embeddings"]
	derived_max_model_len *= scaling_factor
	if encoder_config and "max_seq_length" in encoder_config:
	derived_max_model_len = encoder_config["max_seq_length"]
	# If the user specified a max length, make sure it is smaller than the
	# derived length from the HF model config.
	if max_model_len is None:
	max_model_len = int(derived_max_model_len)
	if current_platform.is_tpu():
	logger.warning(
	"--max-model-len is not specified, "
	"it's currently using model's default length %s, "
	"which might be too large."
	"Please input with --max-model-len based on your "
	"request input length and output length, to avoid "
	"unnecessary degradation.", max_model_len)
	elif max_model_len > derived_max_model_len:
	# Some models might have a separate key for specifying model_max_length
	# that will be bigger than derived_max_model_len. We compare user input
	# with model_max_length and allow this override when it's smaller.
	model_max_length = getattr(hf_config, "model_max_length", None)
	if model_max_length is not None and max_model_len <= model_max_length:
	if disable_sliding_window:
	# TODO(robertgshaw): Find a model that has model_max_length
	# with sliding window to see if this case should be allowed.
	raise NotImplementedError(
	"Disabling sliding window is not supported for models "
	"model_max_length in the config. Please raise an issue "
	"so we can investigate.")
	else:
	msg = (
	f"User-specified max_model_len ({max_model_len}) is greater "
	f"than the derived max_model_len ({max_len_key}="
	f"{derived_max_model_len} or model_max_length="
	f"{model_max_length} in model's config.json).")
	warning = (
	"VLLM_ALLOW_LONG_MAX_MODEL_LEN must be used with extreme "
	"caution. If the model uses relative position encoding (RoPE), "
	"positions exceeding derived_max_model_len lead to nan. If the "
	"model uses absolute position encoding, positions exceeding "
	"derived_max_model_len will cause a CUDA array out-of-bounds "
	"error.")
	if envs.VLLM_ALLOW_LONG_MAX_MODEL_LEN:
	logger.warning_once("%s %s", msg, warning)
	else:
	raise ValueError(
	f"{msg} To allow overriding this maximum, set "
	f"the env var VLLM_ALLOW_LONG_MAX_MODEL_LEN=1. {warning}")
	return int(max_model_len)

and here

vllm/vllm/config/model.py

Lines 1497 to 1516 in 9360d34

	def get_and_verify_max_len(self, max_model_len: int):
	# Consider max_model_len in tokenizer_config only when
	# pooling models use absolute position_embedding.
	tokenizer_config = None
	if (self.runner_type == "pooling" and getattr(
	self.hf_config, "position_embedding_type", "") == "absolute"):
	tokenizer_config = try_get_tokenizer_config(
	self.tokenizer,
	trust_remote_code=self.trust_remote_code,
	revision=self.tokenizer_revision)
	max_model_len = _get_and_verify_max_len(
	hf_config=self.hf_text_config,
	tokenizer_config=tokenizer_config,
	max_model_len=max_model_len,
	disable_sliding_window=self.disable_sliding_window,
	sliding_window=self.get_sliding_window(),
	spec_target_max_model_len=self.spec_target_max_model_len,
	encoder_config=self.encoder_config)
	logger.info("Using max model len %s", max_model_len)
	return max_model_len

[noooop]

Every branch means there's a poor user who had a problem.

[bnuazz15]

@noooop

you were the one of contributor. thanks

PTAL #19201