Skip to content

Using EvalScope evaluation #611

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Apr 22, 2025
Merged

Using EvalScope evaluation #611

merged 2 commits into from
Apr 22, 2025

Conversation

@RongRongStudio
Copy link
Contributor

@RongRongStudio RongRongStudio commented Apr 22, 2025 •
edited by Yikun
Loading

What this PR does / why we need it?

Using EvalScope to hava a evaluation (include eval and test):

Does this PR introduce any user-facing change?

No

How was this patch tested?

Test locally

@github-actions github-actions bot added the documentation Improvements or additions to documentation label Apr 22, 2025
@Yikun
Copy link
Collaborator

Yikun commented Apr 22, 2025

@RongRongStudio Much thanks for your contributions, please use git commit -s --amend to address DCO error.

image
You can preview via: https://vllm-ascend--611.org.readthedocs.build/en/611/developer_guide/evaluation/using_evalscope.html

wangxiyuan
wangxiyuan reviewed Apr 22, 2025
View reviewed changes
Copy link
Collaborator

@wangxiyuan wangxiyuan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your contribution.

docs/source/developer_guide/evaluation/using_evalscope.md Outdated
Then, you can use the following command to evaluate the model API service:

```
evalscope eval \
Copy link
Collaborator

@wangxiyuan wangxiyuan Apr 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's the diffrenece with evalscope perf, since you only add --api-url, --api-key here, but for evalscope perf it's --url instead. Can they be the same?

Copy link
Collaborator

@Yikun Yikun Apr 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://evalscope.readthedocs.io/en/latest/get_started/basic_usage.html#model-api-service-evaluation

https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/quick_start.html#basic-usage

Seems there are some diff in upstream so I think it's ok in here

@RongRongStudio @Yikun
Create using_evalscope.md
4659162 
Signed-off-by: RongRongStudio <82669040+RongRongStudio@users.noreply.github.com>
Yikun
Yikun approved these changes Apr 22, 2025
View reviewed changes
Copy link
Collaborator

@Yikun Yikun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for your contributions

@Yikun
Update docs
153ff65 
Signed-off-by: Yikun Jiang <yikunkero@gmail.com>
@Yikun
Copy link
Collaborator

Yikun commented Apr 22, 2025

I move eval to top and simplify the doc and test locally.

@Yikun Yikun merged commit 848e041 into vllm-project:main Apr 22, 2025
3 checks passed
@Yikun Yikun mentioned this pull request Apr 26, 2025
Closed
ttanzhiqiang pushed a commit to ttanzhiqiang/vllm-ascend that referenced this pull request Apr 27, 2025
@RongRongStudio @Yikun
Using EvalScope evaluation (vllm-project#611)
030054c 
### What this PR does / why we need it?
Using EvalScope to hava a evaluation (include eval and test):
-
https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/quick_start.html#basic-usage
-
https://evalscope.readthedocs.io/en/latest/get_started/basic_usage.html#model-api-service-evaluation

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Test locally

---------

Signed-off-by: RongRongStudio <82669040+RongRongStudio@users.noreply.github.com>
Signed-off-by: Yikun Jiang <yikunkero@gmail.com>
Co-authored-by: Yikun Jiang <yikunkero@gmail.com>
Angazenn pushed a commit to Angazenn/vllm-ascend that referenced this pull request Oct 21, 2025
@RongRongStudio @Yikun @Angazenn
Using EvalScope evaluation (vllm-project#611)
ccf08bc 
### What this PR does / why we need it?
Using EvalScope to hava a evaluation (include eval and test):
-
https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/quick_start.html#basic-usage
-
https://evalscope.readthedocs.io/en/latest/get_started/basic_usage.html#model-api-service-evaluation

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Test locally

---------

Signed-off-by: RongRongStudio <82669040+RongRongStudio@users.noreply.github.com>
Signed-off-by: Yikun Jiang <yikunkero@gmail.com>
Co-authored-by: Yikun Jiang <yikunkero@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Yikun Yikun Yikun approved these changes

@wangxiyuan wangxiyuan wangxiyuan left review comments

Assignees

No one assigned

Labels

documentation Improvements or additions to documentation

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@RongRongStudio @Yikun @wangxiyuan