Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[INFER][LLM] Add the AutoPredictor for inference #9445

Merged
merged 9 commits into from
Dec 3, 2024

Conversation

zeroRains
Copy link
Contributor

@zeroRains zeroRains commented Nov 17, 2024

PR types

New features

PR changes

Others

Description

整合现有全部Predictor的选择条件,创建AutoPredicotr,可以处理static graph/dynamic graph,inference_mode/not inference_mode,block_attn/not block_attn的情况。

调用方式

predictor = AutoPredictor.create_predictor(predictor_args, config, model_args, tokenizer, model=model)

AutoPredictor的执行流程图如下:

Predictor流程图

当前LLM最简单的调用流程如下图:

8CD0A83C3720B34F3C11FC75FE048AFB

Copy link

paddle-bot bot commented Nov 17, 2024

Thanks for your contribution!

Copy link

codecov bot commented Nov 26, 2024

Codecov Report

Attention: Patch coverage is 0% with 3 lines in your changes missing coverage. Please review.

Project coverage is 53.08%. Comparing base (4b02477) to head (42b3e6a).
Report is 20 commits behind head on develop.

Files with missing lines Patch % Lines
paddlenlp/transformers/auto/modeling.py 0.00% 3 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #9445      +/-   ##
===========================================
+ Coverage    52.93%   53.08%   +0.14%     
===========================================
  Files          688      699      +11     
  Lines       109379   110476    +1097     
===========================================
+ Hits         57899    58641     +742     
- Misses       51480    51835     +355     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@yuanlehome yuanlehome self-assigned this Nov 27, 2024
@yuanlehome yuanlehome self-requested a review November 27, 2024 03:27
@ZHUI ZHUI merged commit 2c1387f into PaddlePaddle:develop Dec 3, 2024
9 of 12 checks passed
@zeroRains zeroRains deleted the auto branch December 3, 2024 03:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants