http stream response via HTTP 1.1 chunked encoding

[lxning]

🚀 The feature

TorchServe supports streaming response for both HTTP and GRPC endpoint.

• GRPC server side streaming see TorchServe grpc server side streaming #2180
• HTTP chunked encoding will be implemented in this ticket

Motivation, pitch

Usually the predication latency is high (eg. 5sec) for a large model inference. Some models are able to generate intermediate prediction results (eg. generator AI). This feature will send the intermediate prediction results to user once the results are ready. The user will gradually get the entire response. For example, User may get first intermediate response within 1 sec, and gradually get the entire result until 5sec. This feature is to improve user prediction experience.

Alternatives

No response

Additional context

No response