[Bug]: Falcon fails if trust_remote_code=True

[robertgshaw2-redhat]

Your current environment

v0.4.3

🐛 Describe the bug

from vllm import LLM
model = LLM("tiiuae/falcon-7b", trust_remote_code=True)

--- Logging error ---
Traceback (most recent call last):
  File "/home/rshaw/.pyenv/versions/3.10.14/lib/python3.10/logging/__init__.py", line 1100, in emit
    msg = self.format(record)
  File "/home/rshaw/.pyenv/versions/3.10.14/lib/python3.10/logging/__init__.py", line 943, in format
    return fmt.format(record)
  File "/home/rshaw/.pyenv/versions/vllm-upstream-pip/lib/python3.10/site-packages/vllm/logging/formatter.py", line 11, in format
    msg = logging.Formatter.format(self, record)
  File "/home/rshaw/.pyenv/versions/3.10.14/lib/python3.10/logging/__init__.py", line 678, in format
    record.message = record.getMessage()
  File "/home/rshaw/.pyenv/versions/3.10.14/lib/python3.10/logging/__init__.py", line 368, in getMessage
    msg = msg % self.args
TypeError: %d format: a real number is required, not list
Call stack:
  File "<stdin>", line 1, in <module>
  File "/home/rshaw/.pyenv/versions/vllm-upstream-pip/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 144, in __init__
    self.llm_engine = LLMEngine.from_engine_args(
  File "/home/rshaw/.pyenv/versions/vllm-upstream-pip/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 335, in from_engine_args
    engine_config = engine_args.create_engine_config()
  File "/home/rshaw/.pyenv/versions/vllm-upstream-pip/lib/python3.10/site-packages/vllm/engine/arg_utils.py", line 559, in create_engine_config
    model_config = ModelConfig(
  File "/home/rshaw/.pyenv/versions/vllm-upstream-pip/lib/python3.10/site-packages/vllm/config.py", line 133, in __init__
    self.max_model_len = _get_and_verify_max_len(
  File "/home/rshaw/.pyenv/versions/vllm-upstream-pip/lib/python3.10/site-packages/vllm/config.py", line 1208, in _get_and_verify_max_len
    logger.warning(
Message: "The model's config.json does not contain any of the following keys to determine the original maximum length of the model: %d. Assuming the model's maximum length is %d."
Arguments: (['max_position_embeddings', 'n_positions', 'max_seq_len', 'seq_length', 'model_max_length', 'max_sequence_length', 'max_seq_length', 'seq_len'], 2048)
INFO 06-09 12:53:58 llm_engine.py:161] Initializing an LLM engine (v0.4.3) with config: model='tiiuae/falcon-7b', speculative_config=None, tokenizer='tiiuae/falcon-7b', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=2048, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=tiiuae/falcon-7b)
[rank0]: Traceback (most recent call last):
[rank0]:   File "<stdin>", line 1, in <module>
[rank0]:   File "/home/rshaw/.pyenv/versions/vllm-upstream-pip/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 144, in __init__
[rank0]:     self.llm_engine = LLMEngine.from_engine_args(
[rank0]:   File "/home/rshaw/.pyenv/versions/vllm-upstream-pip/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 359, in from_engine_args
[rank0]:     engine = cls(
[rank0]:   File "/home/rshaw/.pyenv/versions/vllm-upstream-pip/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 222, in __init__
[rank0]:     self.model_executor = executor_class(
[rank0]:   File "/home/rshaw/.pyenv/versions/vllm-upstream-pip/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 41, in __init__
[rank0]:     self._init_executor()
[rank0]:   File "/home/rshaw/.pyenv/versions/vllm-upstream-pip/lib/python3.10/site-packages/vllm/executor/gpu_executor.py", line 24, in _init_executor
[rank0]:     self.driver_worker.load_model()
[rank0]:   File "/home/rshaw/.pyenv/versions/vllm-upstream-pip/lib/python3.10/site-packages/vllm/worker/worker.py", line 121, in load_model
[rank0]:     self.model_runner.load_model()
[rank0]:   File "/home/rshaw/.pyenv/versions/vllm-upstream-pip/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 134, in load_model
[rank0]:     self.model = get_model(
[rank0]:   File "/home/rshaw/.pyenv/versions/vllm-upstream-pip/lib/python3.10/site-packages/vllm/model_executor/model_loader/__init__.py", line 21, in get_model
[rank0]:     return loader.load_model(model_config=model_config,
[rank0]:   File "/home/rshaw/.pyenv/versions/vllm-upstream-pip/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 240, in load_model
[rank0]:     model = _initialize_model(model_config, self.load_config,
[rank0]:   File "/home/rshaw/.pyenv/versions/vllm-upstream-pip/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 91, in _initialize_model
[rank0]:     return model_class(config=model_config.hf_config,
[rank0]:   File "/home/rshaw/.pyenv/versions/vllm-upstream-pip/lib/python3.10/site-packages/vllm/model_executor/models/falcon.py", line 389, in __init__
[rank0]:     self.transformer = FalconModel(config, cache_config, quant_config)
[rank0]:   File "/home/rshaw/.pyenv/versions/vllm-upstream-pip/lib/python3.10/site-packages/vllm/model_executor/models/falcon.py", line 350, in __init__
[rank0]:     self.h = nn.ModuleList([
[rank0]:   File "/home/rshaw/.pyenv/versions/vllm-upstream-pip/lib/python3.10/site-packages/vllm/model_executor/models/falcon.py", line 351, in <listcomp>
[rank0]:     FalconDecoderLayer(config, cache_config, quant_config)
[rank0]:   File "/home/rshaw/.pyenv/versions/vllm-upstream-pip/lib/python3.10/site-packages/vllm/model_executor/models/falcon.py", line 249, in __init__
[rank0]:     if (config.num_ln_in_parallel_attn is None
[rank0]:   File "/home/rshaw/.pyenv/versions/vllm-upstream-pip/lib/python3.10/site-packages/transformers/configuration_utils.py", line 264, in __getattribute__
[rank0]:     return super().__getattribute__(key)
[rank0]: AttributeError: 'FalconConfig' object has no attribute 'num_ln_in_parallel_attn'

[allen-li1231]

Confirmed the same issue on my end. Actually it doesn't matter whether trust_remote_code is True or False (I have tried both), what matters is that this parameter is set. Once I remove this then all worked well to me.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!