Added Support for the new /fim/completions route on Mistral AI API

[keshavkrishna]

Support new /fim/completions route on Mistral AI API:

Description:

• /fim/completions endpoint work in the same way as the /chat/completions route. It handles extra parameters such as suffix, prompt also.
• Implemented fimCompletionsHandler to handle requests to the /fim/completions endpoint.
• Updated the API configuration to include the new fimComplete endpoint.
• implemented response and stream chunk transformations for the fimComplete function in Mistral AI.
• Modified index.ts to register the new route v1//fim/completions with request validation.

Motivation:

• To add support in portkey gateway for the new /fim/completions route on Mistral AI API

Related Issues:

• Support for the new /fim/completions route on Mistral API #381

[narengogi]

Thank you for the contribution @keshavkrishna will review this soon

[narengogi]

Looks good to me, will have to checkout in local and test the changes.

@VisargD do you think we should refactor the handlers to be more generic?
I think this is okay because we will not be seeing new types of inference endpoints that often, but new handlers feel redundant

[VisargD]

Hey @keshavkrishna! Will you be able to take a look at the comments? And please update your branch with the latest main branch changes

[keshavkrishna]

Yes I will take care of these comments by this weekend. Is that timeline okay?

[VisargD]

Hey. Is it not possible to reuse the existing /chat/completions route? We can passthrough all the extra key as it is

[keshavkrishna]

Requests to the existing /chat/completions route are redirected to the provider's base URL with the endpoint chat/completions. In this scenario, we need to direct requests to the provider's base URL with the endpoint fim/completions.

[narengogi]

Approving only need to make one last change as mentioned in prev comment

[VisargD]

We can accept a new header x-portkey-mistral-fim-completion: true and route it conditionally to the the mistral fim completions route. Adding a new router for provider specific routes is not the most feasible solution here because it will increase the number of routers in future.

EDIT: Adding extra details

• In this function, gateway already passes providerOptions. Using it, you can check for a new param called providerOptions.mistralFimCompletion. If its true, you can change the endpoint to the mistral fim endpoint.
  

  gateway/src/providers/mistral-ai/api.ts

  Line 8 in 9724792

  getEndpoint: ({ fn }) => {

• In the chatComplete config, you can add the extra fields which are required (like suffix, prompt, etc.). The response structure seems to be OpenAI compliant. So that should not be a problem

[keshavkrishna]

Sure, I'll implement this

[keshavkrishna]

@VisargD

If I am not getting mistralFimCompletion in providerOptions. following is provider option i am getting

{
  provider: 'mistral-ai',
  apiKey: '  SOME_API_KEY_HERE',
  overrideParams: {},
  retry: {},
  cache: {}
} 

so i have passed requestHeaders in getEndpioint, and i am checking by
if (requestHeaders['x-portkey-mistral-fim-completion'] === 'true')

Is that okay??

[VisargD]

@VisargD

If I am not getting mistralFimCompletion in providerOptions. following is provider option i am getting

{
  provider: 'mistral-ai',
  apiKey: '  SOME_API_KEY_HERE',
  overrideParams: {},
  retry: {},
  cache: {}
} 

so i have passed requestHeaders in getEndpioint, and i am checking by if (requestHeaders['x-portkey-mistral-fim-completion'] === 'true')

Is that okay??

Hey! specific headers checks might not be required. Here is what you can do:

• In this function, add one more condition for mistral-ai provider and add the mapping for mistralFimCompletion in it. Check the existing conditions for other providers for reference. Once this is done, you will always get the mistralFimCompletion param in providerOptions.
  

  gateway/src/handlers/handlerUtils.ts

  Line 969 in 3bb3a80

  export function constructConfigFromRequestHeaders(

Please let me know if you want an example

[narengogi]

Is this working? I've noticed you've not made any changes in providers/mistral-ai/index.ts for request and response transformers

[keshavkrishna]

Yes it is. We are using chatComplete transforms as per visarg's comment.

[VisargD]

@keshavkrishna - Can you please run npm run format and also update with main branch? I can merge it now

[keshavkrishna]

@VisargD as requested i have updated my branch and also formatted it