Skip to content

Load test setup from config files, fix correctness, enable plugin in V1 #25

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 10 commits into from
Jun 27, 2025
Merged

Load test setup from config files, fix correctness, enable plugin in V1 #25

merged 10 commits into from
Jun 27, 2025

Conversation

@bringlein
Copy link
Collaborator

@bringlein bringlein commented Jun 26, 2025

No description provided.

bringlein added 5 commits June 25, 2025 07:35
@bringlein
adding batch composition variable
cdc3e59 
Signed-off-by: Burkhard Ringlein <ngl@zurich.ibm.com>
@bringlein
implementing loading envs from file
f7a7ef9 
Signed-off-by: Burkhard Ringlein <ngl@zurich.ibm.com>
@bringlein
cleanup, support debug flags
548c6ef 
Signed-off-by: Burkhard Ringlein <ngl@zurich.ibm.com>
@bringlein
fix flags & correctness
90eaa88 
Signed-off-by: Burkhard Ringlein <ngl@zurich.ibm.com>
@bringlein
fix callers on AMD
c67baa6 
Signed-off-by: Burkhard Ringlein <ngl@zurich.ibm.com>
@bringlein
Copy link
Collaborator Author

bringlein commented Jun 26, 2025 

python3 /scripts/benchmark.py prefix -x /scripts/setups/prefix_correctnes.conf
python3 /scripts/benchmark.py prefix -x /scripts/setups/prefix_correctnes_rocm.conf

passes on H100/MI300

bringlein added 4 commits June 26, 2025 07:46
@bringlein
updating backend from upstream
5f287aa 
Signed-off-by: Burkhard Ringlein <ngl@zurich.ibm.com>
@bringlein
getting the plugin to work
dd3bb3b 
Signed-off-by: Burkhard Ringlein <ngl@zurich.ibm.com>
@bringlein
switch to triton-dejavu jitcache
e10bc56 
Signed-off-by: Burkhard Ringlein <ngl@zurich.ibm.com>
@bringlein
formatting, spelling...
aed0779 
Signed-off-by: Burkhard Ringlein <ngl@zurich.ibm.com>
@bringlein
Copy link
Collaborator Author

bringlein commented Jun 26, 2025

inside the docker container, just

vllm serve meta-llama/Llama-3.1-8B-Instruct

will use vLLM V1 with the vllm-triton-backend plugin and jitcache etc. from triton-dejavu

@bringlein bringlein changed the title Load test setup from config files, fix correctness Load test setup from config files, fix correctness, enable plugin in V1 Jun 26, 2025
@bringlein bringlein requested review from jvlunteren and tdoublep June 26, 2025 13:27
@bringlein
throw error if plugin is used in V0
85b0b94 
Signed-off-by: Burkhard Ringlein <ngl@zurich.ibm.com>
jvlunteren
jvlunteren approved these changes Jun 27, 2025
View reviewed changes
Copy link
Collaborator

@jvlunteren jvlunteren left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@bringlein bringlein merged commit 3422877 into main Jun 27, 2025
1 check passed
@bringlein bringlein deleted the ngl_updates_06-2025_2 branch June 27, 2025 09:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jvlunteren jvlunteren jvlunteren approved these changes

@tdoublep tdoublep Awaiting requested review from tdoublep

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@bringlein @jvlunteren