[Inference] Support for Messages API OpenAI API specs #574
Description
Considering that TGI now supports the Messages API compatible with OpenAI API specs, it would be great to have native support in the Inference package.
curl localhost:3000/v1/chat/completions \
-X POST \
-d '{
"model": "tgi",
"messages": [
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": "What is deep learning?"
}
],
"stream": true,
"max_tokens": 20
}' \
-H 'Content-Type: application/json'https://huggingface.co/docs/text-generation-inference/messages_api
I tried this , but model is not send and raises a backend error
const mistral = new HfInferenceEndpoint(
"https://api-inference.huggingface.co/models/mistralai/Mixtral-8x7B-Instruct-v0.1/v1/chat/completions",
HF_TOKEN,
);
const stream = mistral.streamingRequest({
model: "tgi",
messages: [{ role: "user", content: "What's the meaning ..." }],
parameters: {
messages: messages,
temperature: 0.8,
max_new_tokens: 1000,
repetition_penalty: 1.2,
seed: Math.floor(Math.random() * Number.MAX_SAFE_INTEGER),
},
});