Langchain integration

[XBeg9]

I have tried to use ray-llm, but it doesn't work out of the box with what vllm suppose to provide. Is there any changes that ray-llm provides to the API endpoint while inference the model?

Here is comment on issue to vllm repo that was solved long time ago: vllm-project/vllm#323 (comment)

[akshay-anyscale]

you can use the ChatAnyscale module https://api.python.langchain.com/en/latest/chat_models/langchain.chat_models.anyscale.ChatAnyscale.html with the api endpoint set to your endpoint