Question: How do I search asynchronously using llm response streaming?

[bldev2473]

Hi there.

I'm trying to export the final response of a RAG search to SSE using content streaming from an LLM provider.

How can I integrate the generator response of generate_content_stream() func?
The search() function of GraphRAG is synchronous and the invoke() call of the llm model is synchronous.

Do I need to customise GraphRAG?

[stellasia]

Hi @bldev2473 ,

I'm afraid streaming response is not yet possible with this package. The LLMInterface which is responsible for calling the LLM is not managing streaming at the moment.

If this is important to you and you feel like you want to contribute this feature, we'd be happy to support you in the implementation.

[bldev2473]

Thank you for your response. I'll use the custom implementation for now and ask for more later.