[Frontend] Add chunked processing to handle long inputs in embedding models #22280
Conversation
…scripts, incorporating chunk processing capabilities to handle exceptionally long inputs. The README documentation has been revised to provide comprehensive instructions on usage methods and configuration options. Signed-off-by: x22x22 <wadeking@qq.com>
x22x22
requested review from
WoosukKwon,
aarnphm,
hmellor,
houseroad,
mgoin,
robertgshaw2-redhat,
simon-mo,
tlrmchlsmth and
youkaichao
as code owners
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
mergify
bot
added
documentation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces chunked processing for long text embeddings, allowing vLLM to handle inputs that exceed the model's maximum context length. The changes include new configuration options in PoolerConfig, core logic for chunking and aggregation in serving_embedding.py, and several robustness improvements in serving_engine.py. Additionally, new examples are provided to demonstrate and test the new feature.
My review found one potential high-severity issue in vllm/config.py where logic for selecting the correct transformers backend for pooling models seems to have been accidentally removed. This could lead to errors when using pooling models with the transformers backend. The rest of the implementation for chunked processing appears solid and well-designed.
Signed-off-by: x22x22 <wadeking@qq.com>
…iguration options to facilitate long-text input support. Signed-off-by: x22x22 <wadeking@qq.com>
… Extensive Texts Signed-off-by: x22x22 <wadeking@qq.com>
- Restore vllm/transformers_utils/processor.py to main branch - Restore vllm/inputs/registry.py to main branch - Ensure all file metadata matches main branch exactly Signed-off-by: x22x22 <wadeking@qq.com>
…edding generation. Signed-off-by: x22x22 <wadeking@qq.com>
…edding generation. Signed-off-by: x22x22 <wadeking@qq.com>
…edding generation. Signed-off-by: x22x22 <wadeking@qq.com>
Signed-off-by: x22x22 <wadeking@qq.com>
Signed-off-by: x22x22 <wadeking@qq.com>
Signed-off-by: x22x22 <wadeking@qq.com>
Signed-off-by: x22x22 <wadeking@qq.com>
Signed-off-by: x22x22 <wadeking@qq.com>
|
@maxdebayser @DarkLight1337 @hmellor |
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
|
Let's merge this |
DarkLight1337
added
the
ready
python -m mypy vllm/entrypoints/openai/serving_embedding.py --python-version 3.11
vllm/entrypoints/openai/serving_embedding.py:160: error: Incompatible return value type (got "PoolerConfig | bool | None", expected "bool") [return-value]
Found 1 error in 1 file (checked 1 source file)After merging the main, new errors occurred. Let me handle it. |
Signed-off-by: x22x22 <wadeking@qq.com>
…models (vllm-project#22280) Signed-off-by: x22x22 <wadeking@qq.com> Signed-off-by: Kdump <rootshellexp@gmail.com> Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Co-authored-by: Maximilien de Bayser <maxdebayser@gmail.com> Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: Diego-Castan <diego.castan@ibm.com>
…models (vllm-project#22280) Signed-off-by: x22x22 <wadeking@qq.com> Signed-off-by: Kdump <rootshellexp@gmail.com> Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Co-authored-by: Maximilien de Bayser <maxdebayser@gmail.com> Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
…models (vllm-project#22280) Signed-off-by: x22x22 <wadeking@qq.com> Signed-off-by: Kdump <rootshellexp@gmail.com> Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Co-authored-by: Maximilien de Bayser <maxdebayser@gmail.com> Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
…models (vllm-project#22280) Signed-off-by: x22x22 <wadeking@qq.com> Signed-off-by: Kdump <rootshellexp@gmail.com> Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Co-authored-by: Maximilien de Bayser <maxdebayser@gmail.com> Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
…models (vllm-project#22280) Signed-off-by: x22x22 <wadeking@qq.com> Signed-off-by: Kdump <rootshellexp@gmail.com> Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Co-authored-by: Maximilien de Bayser <maxdebayser@gmail.com> Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: Xiao Yu <xiao.yu@amd.com>
…models (vllm-project#22280) Signed-off-by: x22x22 <wadeking@qq.com> Signed-off-by: Kdump <rootshellexp@gmail.com> Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Co-authored-by: Maximilien de Bayser <maxdebayser@gmail.com> Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
6 participants
Original pr
#20837
The previous submission was not merged for too long, resulting in too many conflicts, so I'm resubmitting it.
@noooop
@hmellor
@maxdebayser