-
Notifications
You must be signed in to change notification settings - Fork 544
Fix accuracy test config --config-list-file #2163
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Signed-off-by: Icey <1790571317@qq.com>
|
👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:
If CI fails, you can run linting and testing checks locally according Contributing and Testing. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2163 +/- ##
==========================================
- Coverage 76.89% 76.67% -0.22%
==========================================
Files 107 107
Lines 11883 11968 +85
==========================================
+ Hits 9137 9177 +40
- Misses 2746 2791 +45
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
| if single_config: | ||
| metafunc.parametrize("config_filename", | ||
| [Path(single_config).resolve()]) | ||
| if metafunc.config.getoption("--config-list-file"): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why reorder this?
the original thoughts is that --config has higer priority than --config-list-file
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Because the --config option has default settings, if --config is checked first, the logic for --config-list-file will definitely not be executed.
| parser.addoption( | ||
| "--report-dir", | ||
| action="store", | ||
| default="./benchmarks/accuracy", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it's better to keep current path, ./, otherwise the ./benchmarks/accuracy is required.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The code has ensured the creation of the ./benchmarks/accuracy directory: https://github.com/vllm-project/vllm-ascend/blob/main/tests/e2e/singlecard/models/test_lm_eval_correctness.py#L88
What this PR does / why we need it?
This PR fix accuracy test related to #2073, users can now perform accuracy tests on multiple models simultaneously and generate different report files by running:
Does this PR introduce any user-facing change?
no
How was this patch tested?