Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/evict req on client disconnect streaming case #223

Draft
wants to merge 32 commits into
base: main
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
d613565
chore: Add SimpleDelayedStreamAPI for delayed streaming of output
bhimrazy Aug 26, 2024
371cf56
add test_stream_client_disconnection
bhimrazy Aug 26, 2024
9e7f841
add request_evicted_status param to run_streaming_loop
bhimrazy Aug 26, 2024
7ce49ac
update test_stream_client_disconnection
bhimrazy Aug 26, 2024
56c8587
adds functionality to evict the request if disconnected before comple…
bhimrazy Aug 26, 2024
f5961c4
Merge branch 'main' into feat/evict-req-on-client-disconnect-streamin…
bhimrazy Aug 26, 2024
9330997
update exception
aniketmaurya Aug 26, 2024
1f0bfe5
fix test
aniketmaurya Aug 26, 2024
d41db3c
Update src/litserve/server.py
aniketmaurya Aug 26, 2024
4344720
Merge branch 'main' into feat/evict-req-on-client-disconnect-streamin…
aniketmaurya Aug 26, 2024
f177fcb
Merge branch 'main' into feat/evict-req-on-client-disconnect-streamin…
bhimrazy Aug 27, 2024
4e5045a
Merge branch 'main' into feat/evict-req-on-client-disconnect-streamin…
bhimrazy Aug 28, 2024
1d4677c
Merge branch 'main' into feat/evict-req-on-client-disconnect-streamin…
bhimrazy Aug 29, 2024
ca6fbc2
reverted changes to new updates
bhimrazy Aug 31, 2024
e61cdab
update
bhimrazy Aug 31, 2024
6c2e0c6
update
bhimrazy Aug 31, 2024
6668cc8
Merge branch 'main' into feat/evict-req-on-client-disconnect-streamin…
bhimrazy Aug 31, 2024
3448ef3
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 31, 2024
2cfd68e
chore: Add test for streaming client disconnection
bhimrazy Aug 31, 2024
c95ee45
handle client disconnection streaming nonbatched case
bhimrazy Aug 31, 2024
bac5534
chore: Optimize streaming loop performance by checking for client dis…
bhimrazy Aug 31, 2024
f08ed4b
chore: Update streaming loop to include request eviction status
bhimrazy Aug 31, 2024
e060e39
Merge branch 'main' into feat/evict-req-on-client-disconnect-streamin…
bhimrazy Sep 21, 2024
2b11fc7
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 21, 2024
5323c51
Refactor inference_worker function to remove optional parameters and …
bhimrazy Sep 21, 2024
4368b57
update
bhimrazy Sep 21, 2024
611a751
update
bhimrazy Sep 21, 2024
5cc0f77
add missing param
bhimrazy Sep 21, 2024
8d4a05d
add missing param
bhimrazy Sep 21, 2024
bd68b6c
add missing param for run streaming loop
bhimrazy Sep 21, 2024
56f1076
test by removing the check interval
bhimrazy Sep 21, 2024
49bed55
so there is performance drop with this check,
bhimrazy Sep 21, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
pre-commit-ci[bot] committed Aug 31, 2024
commit 3448ef3aeb8c08a6a9ea34566e441c83b9ddaec0
2 changes: 1 addition & 1 deletion src/litserve/server.py
Original file line number Diff line number Diff line change
@@ -471,4 +471,4 @@ def setup_auth(self):
return self.lit_api.authorize
if LIT_SERVER_API_KEY:
return api_key_auth
return no_auth
return no_auth