OpenAI API support - Langchain passes prompt as a list instead of str

[yanivshimoni89]

System Info

Ubuntu 22.04
A100
TGI docker version 1.4.3

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

Using langchain and create QA Retrieval Chain.
The chain passes prompt as a list instead of str and the api should accept it.

Expected behavior

As mentioned in vllm-project/vllm#186

[drbh]

Hi @yanivshimoni89 it's unclear if this is the intended functionality of the original openai chat api.

Looking into the vllm issue linked above it looks like theres only been a temporary fix where the first value of the list is treated as the prompt. (issue above refers to this PR vllm-project/vllm#323 (comment))

This fix is small and can be added to TGI but a longer term solution (update langchain to send a string) would be best

[yanivshimoni89]

Hi, @drbh I comprehend your viewpoint, and I concur. However, this functionality is currently not incorporated in TGI, which is a feature of the OpenAI API. As such, I believe it should be addressed in TGI. Regardless, I would appreciate being informed about any progress on this matter.