[Feature]: support messages input for classify api skywork-reward model

[Snowdar]

🚀 The feature, motivation and pitch

In the current inference of reward models, classification models are often used, such as the skywork reward model. There is an issue where the concepts and actual functions of several existing interfaces do not correspond:

1. The reward interface does not return the score of the BT model (pooling method and softmax also need to be considered).
2. The classification model is restricted to using only the classify interface, but this interface does not support message input.
3. If classify can only use list[str], there seems to be no apply_chat_template interface (it is not desirable to place too many elements on the client side).

Thanks.

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[noooop]

mark

---

Which API do you prefer to use, classify API, pooling API, or redesign a API for the reward model?

I personally think that /pooling needs to be refactored to better support the reward models

PTAL #21720 #20538

Welcome to share your usage scenarios and encountered problems

[noooop]

cc @xsank

Welcome to the discussion

[Snowdar]

mark

Which API do you prefer to use, classify API, pooling API, or redesign a API for the reward model?

I personally think that /pooling needs to be refactored to better support the reward models

PTAL #21720 #20538

Welcome to share your usage scenarios and encountered problems

I think we can define some classic paradigms based on the input and output of the model. For example:

1. The input can support str or messages.
2. Regarding the output, many models share a common behavior, which is obtaining the last layer's hidden feature.

• If the goal is to directly obtain the last hidden feature, it would be better to design an additional interface, such as /last_hidden_feature, which can be shared by many models.
  Going further, we need to consider the aggregation of the last hidden feature (token embedding), that is, Pooling is required, such as [CLS], LAST, MEAN. It seems this is already available.
• If the goal is to obtain embeddings (for retrieval), it is more appropriate to use the embedding interface, i.e., embed.
• If the goal is to obtain the similarity score of pairs, where the input is a group of samples, the server performs pooling to obtain embeddings and then directly calculates the score. It seems this is already available as /score, but the name might be misunderstood as the interface for reward models.
• If the goal is to obtain classification categories and confidence levels, for a classify model, it can be /classify.
• If the goal is to obtain scores, there are two types: token-level scores and sentence-level scores. This depends on whether the number of output nodes in the last Linear layer of the model is 1, the original number of tokens, or other numbers. This can use /reward, but the problem is that many BT Reward Models are actually implemented through classification models, especially Multi-label BT models (such as Decision Trees), which also need to support the /reward interface.

Currently, the biggest issue is that some support combinations have not been covered yet, which is a bit behind the development.

[noooop]

According to vllm's taxonomy

• the reward model means using all pooling (and step pooling, let's temporarily not discuss it. ), meaning there is an output for each token.

• the classify model means using [CLS], LAST, MEAN pooling, which means there is only one output for a sequence

[noooop]

try /pooling + Input like Chat API

https://github.com/vllm-project/vllm/blob/main/examples/online_serving/openai_pooling_client.py

Is this similar to your usage scenario?

---

Please use internlm/internlm2-1_8b-reward

I think jason9693/Qwen2.5-1.5B-apeach will report an error when executing this script

I'll fix it in the next PR

[xsank]

According to vllm's taxonomy根据 vllm 的分类法

• the reward model means using all pooling (and step pooling, let's temporarily not discuss it. ), meaning there is an output for each token.奖励模型意味着使用所有池化（以及阶梯池化，我们暂时不讨论它），这意味着每个代币都有一个输出。
• the classify model means using [CLS], LAST, MEAN pooling, which means there is only one output for a sequence分类模型意味着使用 [CLS]， LAST， MEAN 池化，这意味着一个序列只有一个输出

this could easily solved by overwrite pooler config.