Support streaming responses from LLM

[talrejanikhil]

There are some LLMs like Open AI where the response can be streamed, so that the user does not have to wait for the full response to be available.

https://platform.openai.com/docs/api-reference/streaming

Is it possible to add this option to Vanna?

[mike-niemand]

It's an interesting one. There is an impression that AI is talking to you which is why you get a streamed response but a db response can be instantaneous and do you want your user to wait as it streams the top 3 sales etc

[talrejanikhil]

Actually streaming makes the user wait less. They will see the SQL appearing immediately (token by token) rather waiting for the whole SQL to be generated by the LLM

[mike-niemand]

I've managed to do this with a streamed response.