[New Model]: Support BAAI/bge-reranker-v2-gemma model

[NaiveYan]

The model to consider.

https://huggingface.co/BAAI/bge-reranker-v2-gemma

The closest model vllm already supports.

BAAI/bge-reranker-v2-m3
google/gemma-2b

What's your difficulty of supporting the model you want?

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

I think this model is already supported? The model architecture is GemmaForCausalLM which is a supported model in vLLM. You just have to set --task classify to use it as a reranker.

[NaiveYan]

I think this model is already supported? The model architecture is GemmaForCausalLM which is a supported model in vLLM. You just have to set --task classify to use it as a reranker.

@DarkLight1337 According to the documentation, the --task score parameter should be used for reranker models. I have tried using --task score, and the logs are as below:

logs

INFO:     10.12.21.85:59088 - "POST /v1/rerank HTTP/1.1" 500 Internal Server Error
ERROR 06-16 03:13:21 [engine.py:160] AttributeError("'GemmaForCausalLM' object has no attribute 'pooler'")
ERROR 06-16 03:13:21 [engine.py:160] Traceback (most recent call last):
ERROR 06-16 03:13:21 [engine.py:160]   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 158, in start
ERROR 06-16 03:13:21 [engine.py:160]     self.run_engine_loop()
ERROR 06-16 03:13:21 [engine.py:160]   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 221, in run_engine_loop
ERROR 06-16 03:13:21 [engine.py:160]     request_outputs = self.engine_step()
ERROR 06-16 03:13:21 [engine.py:160]                       ^^^^^^^^^^^^^^^^^^
ERROR 06-16 03:13:21 [engine.py:160]   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 247, in engine_step
ERROR 06-16 03:13:21 [engine.py:160]     raise e
ERROR 06-16 03:13:21 [engine.py:160]   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 230, in engine_step
ERROR 06-16 03:13:21 [engine.py:160]     return self.engine.step()
ERROR 06-16 03:13:21 [engine.py:160]            ^^^^^^^^^^^^^^^^^^
ERROR 06-16 03:13:21 [engine.py:160]   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/llm_engine.py", line 1412, in step
ERROR 06-16 03:13:21 [engine.py:160]     outputs = self.model_executor.execute_model(
ERROR 06-16 03:13:21 [engine.py:160]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-16 03:13:21 [engine.py:160]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 299, in execute_model
ERROR 06-16 03:13:21 [engine.py:160]     driver_outputs = self._driver_execute_model(execute_model_req)
ERROR 06-16 03:13:21 [engine.py:160]                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-16 03:13:21 [engine.py:160]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/mp_distributed_executor.py", line 144, in _driver_execute_model
ERROR 06-16 03:13:21 [engine.py:160]     return self.driver_worker.execute_model(execute_model_req)
ERROR 06-16 03:13:21 [engine.py:160]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-16 03:13:21 [engine.py:160]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 420, in execute_model
ERROR 06-16 03:13:21 [engine.py:160]     output = self.model_runner.execute_model(
ERROR 06-16 03:13:21 [engine.py:160]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-16 03:13:21 [engine.py:160]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 06-16 03:13:21 [engine.py:160]     return func(*args, **kwargs)
ERROR 06-16 03:13:21 [engine.py:160]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 06-16 03:13:21 [engine.py:160]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/pooling_model_runner.py", line 148, in execute_model
ERROR 06-16 03:13:21 [engine.py:160]     self.model.pooler(hidden_states=hidden_or_intermediate_states,
ERROR 06-16 03:13:21 [engine.py:160]     ^^^^^^^^^^^^^^^^^
ERROR 06-16 03:13:21 [engine.py:160]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1928, in __getattr__
ERROR 06-16 03:13:21 [engine.py:160]     raise AttributeError(
ERROR 06-16 03:13:21 [engine.py:160] AttributeError: 'GemmaForCausalLM' object has no attribute 'pooler'
INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [1]
Process SpawnProcess-1:
Traceback (most recent call last):
  File "/usr/lib/python3.12/multiprocessing/process.py", line 317, in _bootstrap
    util._exit_function()
  File "/usr/lib/python3.12/multiprocessing/util.py", line 360, in _exit_function
    p.join()
  File "/usr/lib/python3.12/multiprocessing/process.py", line 149, in join
    res = self._popen.wait(timeout)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/multiprocessing/popen_fork.py", line 43, in wait
    return self.poll(os.WNOHANG if timeout == 0.0 else 0)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/multiprocessing/popen_fork.py", line 27, in poll
    pid, sts = os.waitpid(self.pid, flag)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 426, in signal_handler
    raise KeyboardInterrupt("MQLLMEngine terminated")
KeyboardInterrupt: MQLLMEngine terminated
ERROR 06-16 03:13:21 [multiproc_worker_utils.py:120] Worker VllmWorkerProcess pid 220 died, exit code: -15
INFO 06-16 03:13:21 [multiproc_worker_utils.py:124] Killing local vLLM worker processes
[rank0]:[W616 03:13:21.508798815 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
/usr/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/usr/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

[DarkLight1337]

cc @noooop can you help with this?

[noooop]

I tried before, the main issue was

1. preparing input_ids(some kind of format_instruction)
2. get logits from "yes" token
3. support cutoff_layers

It is very difficult to fully support the features mentioned above. We need some compromise.
Below, we explain why each feature is difficult to support.

First, BAAI/bge-reranker-v2-gemma uses a method called relevance generation, which is slightly different from other rerankers that use SequenceClassification. However, they can be converted during inference. Please refer to the implementation of qwen3 reranker.

• for 'preparing input_ids(some kind of format_instruction)'
  Offline usage is very easy; you can define a format_instruction function and call it. However, calling this format_instruction function on the server online (OpenAI-Compatible Server) is extremely difficult, so users still need to manually format when calling the API.
  Perhaps further support for templates by Sentence Transformers can simplify this.

• for 'get logits from "yes" token'
  You can convert it to GemmaForSequenceClassification offline, or add an interface "classifier_from_token": 'Yes' for online conversion when loading. It is relatively easy to implement.

• for 'support cutoff_layers'
  Currently, there is no way to process requests with different cutoff layers in the same batch.
  So there are three methods below.

1. When starting, only support one specified cutoff_layers, or only convert specific cutoff_layers when offline converting to GemmaForSequenceClassification.
2. Each batch processes only one request.
3. Add a new scheduler to schedule requests with the same cutoff_layers into the same batch.

it is obvious that the first method is the simplest and most efficient.

[noooop]

If #19675 is merged, vllm can support GemmaForSequenceClassification;

There is a simplest way to support BAAI/bge-reranker-v2-gemma

you need to convert BAAI/bge-reranker-v2-gemma into the GemmaForSequenceClassification format offline, and manually format_instruction. （It's really a bit troublesome.）

1. preparing input_ids(some kind of format_instruction) <- manually format instruction
2. get logits from "yes" token <- offline manual converting to GemmaForSequenceClassification
3. support cutoff_layers <- offline manual converting to GemmaForSequenceClassification

Refer to https://huggingface.co/Qwen/Qwen3-Reranker-0.6B/discussions/3 (This uses "yes" and "no" tokens, but the principle is the same.)

@NaiveYan do you okay with this

[NaiveYan]

If #19675 is merged, vllm can support GemmaForSequenceClassification;如果 #19675 合并，vllm 可以支持 GemmaForSequenceClassification；

There is a simplest way to support BAAI/bge-reranker-v2-gemma有一个最简单的方法来支持 BAAI/bge-reranker-v2-gemma

you need to convert BAAI/bge-reranker-v2-gemma into the GemmaForSequenceClassification format offline, and manually format_instruction. （It's really a bit troublesome.）需要先把 BAAI/bge-reranker-v2-gemma 离线转换成 GemmaForSequenceClassification 格式，再手动 format_instruction。（确实有点麻烦）

1. preparing input_ids(some kind of format_instruction) <- manually format instruction准备输入 ID（某种格式指令）<- 手动格式化指令

2. get logits from "yes" token <- offline manual converting to GemmaForSequenceClassification从“是”标记获取 logits <- 离线手动转换为 GemmaForSequenceClassification

3. support cutoff_layers <- offline manual converting to GemmaForSequenceClassification支持 cutoff_layers <- 离线手动转换为 GemmaForSequenceClassification

Refer to https://huggingface.co/Qwen/Qwen3-Reranker-0.6B/discussions/3 (This uses "yes" and "no" tokens, but the principle is the same.)请参阅 https://huggingface.co/Qwen/Qwen3-Reranker-0.6B/discussions/3 （这使用“是”和“否”标记，但原理是一样的。）

@NaiveYan do you okay with this你同意吗

@noooop Understood! Looking forward to supporting more new LLM-based reranker models in the future.

[noooop]

do you okay with this你同意吗

这个翻译应该是: 这个简单的方案能满足你的使用场景吗，以及你能把它折腾出来吗

[noooop]

@NaiveYan

PTAL #19675

我添加了转换脚本 converting2seq_cls_models.py 以及转换命令和 requests demo + formating query & document。