Handle streaming loop if client has disconnected

[aniketmaurya]

🚀 Feature

If the client is disconnected while streaming the response we still complete the loop. We can stop the loop on BrokenPipeError.

x = lit_api.decode_request(x_enc)
y_gen = lit_api.predict(x)
y_enc_gen = lit_api.encode_response(y_gen)
for y_enc in y_enc_gen:
    ####### Detect disconnected client on BrokenPipeError  ############
    with contextlib.suppress(BrokenPipeError):
        pipe_s.send(y_enc)

Motivation

Pitch

Alternatives

Additional context

cc: @lantiga

[aniketmaurya]

@lantiga we should stop processing request if the client has disconnected and allow other requests in the queue to process. wdyt?

[andyland]

@aniketmaurya Non-streaming requests should terminate the work if the client disconnects as well: https://www.loom.com/share/d6d97f0f34fd4084a72ff99471581cd7

[aniketmaurya]

@andyland in the case when the model worker has picked your request from queue and has started processing then we can't cancel it out unless we do some process communication which might add some bottlenecks but it's definitely worth looking into. Thanks for raising it.

[aniketmaurya]

the internal architecture has changed and we will solve it in a different way. This issue is now to be tracked in #165