TorchServe grpc server side streaming

[lxning]

🚀 The feature

TorchServe grpc server side streaming supports

• backend worker continuously send the intermediate prediction response to frontend
• frontend grpc endpoint continuously send the intermediate prediction response from backend to client.

Motivation, pitch

Usually the predication latency is high (eg. 5sec) for a large model inference. Some models are able to generate intermediate prediction results (eg. generator AI). This feature will send the intermediate prediction results to user once the results are ready. The user will gradually get the entire response. For example, User may get first intermediate response within 1 sec, and gradually get the entire result until 5sec. This feature is to improve user prediction experience.

Alternatives

No response

Additional context

No response