Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ENH: Safe iterate stream of ggml model #449

Merged
merged 21 commits into from
Sep 25, 2023
Merged

ENH: Safe iterate stream of ggml model #449

merged 21 commits into from
Sep 25, 2023

Conversation

codingl2k1
Copy link
Contributor

@codingl2k1 codingl2k1 commented Sep 14, 2023 •
edited
Loading

  • Use FastAPI stream.
  • Fix parallel ggml stream undefined behaviour.

Fix: #448

@XprobeBot XprobeBot added the enhancement New feature or request label Sep 14, 2023
@XprobeBot XprobeBot added this to the v0.4.2 milestone Sep 14, 2023
@codingl2k1 codingl2k1 changed the title ENH: Safe iterate stream of ggml generator ENH: Safe iterate stream of ggml model Sep 14, 2023
@codingl2k1 codingl2k1 marked this pull request as ready for review September 15, 2023 02:33
UranusSeven
UranusSeven reviewed Sep 15, 2023
View reviewed changes
xinference/core/restful_api.py Show resolved Hide resolved
xinference/core/restful_api.py Outdated Show resolved Hide resolved
@XprobeBot XprobeBot modified the milestones: v0.4.2, v0.4.3, v0.4.4 Sep 15, 2023
UranusSeven
UranusSeven reviewed Sep 18, 2023
View reviewed changes
xinference/client.py Show resolved Hide resolved
xinference/core/model.py Outdated Show resolved Hide resolved
xinference/core/restful_api.py Outdated Show resolved Hide resolved
codingl2k1 and others added 3 commits September 18, 2023 17:27
@codingl2k1 @UranusSeven
Update xinference/core/model.py
9cb46dd 
Co-authored-by: Uranus <109661872+UranusSeven@users.noreply.github.com>
@codingl2k1 @UranusSeven
Update xinference/core/restful_api.py
11ba496 
Co-authored-by: Uranus <109661872+UranusSeven@users.noreply.github.com>
@codingl2k1
Fix
74efae4
@XprobeBot XprobeBot modified the milestones: v0.4.4, v0.4.5 Sep 19, 2023
@aresnow1
Copy link
Contributor

If three requests come in simultaneously, will the last two requests fail?

@XprobeBot XprobeBot modified the milestones: v0.5.0, v0.5.1 Sep 23, 2023
aresnow1
aresnow1 approved these changes Sep 25, 2023
View reviewed changes
Copy link
Contributor

@aresnow1 aresnow1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@aresnow1 aresnow1 merged commit 5f2078b into xorbitsai:main Sep 25, 2023
@ChengjieLi28 ChengjieLi28 mentioned this pull request Oct 10, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@UranusSeven UranusSeven UranusSeven left review comments

@aresnow1 aresnow1 aresnow1 approved these changes

Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
v0.5.1
Development

Successfully merging this pull request may close these issues.

BUG: Incorrect traceback if raises an exception when stream iteration
4 participants
@codingl2k1 @aresnow1 @UranusSeven @XprobeBot