-
-
Notifications
You must be signed in to change notification settings - Fork 11.3k
[Misc] cache result of disable_inplace #26666
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Signed-off-by: Bill Nell <bnell@redhat.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request aims to optimize the disable_inplace function by caching its result, which is a good performance improvement as the function's return value is constant during program execution. However, the implementation uses functools.cache, a feature introduced in Python 3.9. Since vLLM supports Python 3.8, this will lead to a runtime error. I've suggested a fix to use the compatible @functools.lru_cache(maxsize=None) decorator to ensure compatibility with Python 3.8.
0d8b746 to
403bd63
Compare
Signed-off-by: Bill Nell <bnell@redhat.com>
Signed-off-by: Bill Nell <bnell@redhat.com> Signed-off-by: 1994 <1994@users.noreply.github.com>
Signed-off-by: Bill Nell <bnell@redhat.com> Signed-off-by: Dhruvil Bhatt <bhattdbh@amazon.com>
Signed-off-by: Bill Nell <bnell@redhat.com> Signed-off-by: bbartels <benjamin@bartels.dev>
Signed-off-by: Bill Nell <bnell@redhat.com>
Signed-off-by: Bill Nell <bnell@redhat.com>
Signed-off-by: Bill Nell <bnell@redhat.com>
Signed-off-by: Bill Nell <bnell@redhat.com> Signed-off-by: xuebwang-amd <xuebwang@amd.com>
Signed-off-by: Bill Nell <bnell@redhat.com> Signed-off-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
Signed-off-by: Bill Nell <bnell@redhat.com> Signed-off-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
Signed-off-by: Bill Nell <bnell@redhat.com>
Purpose
Cache result of
disable_inplacecc @youkaichao , @mgoin , @ProExpertProg
Test Plan
Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.