Please support reranker API

[thiner]

Is your feature request related to a problem? Please describe.
Nowadays, embedding + reranker is the SOTA solution to improve the accuracy of RAG system. We've already have the embedding API support in LocalAI, it would be a big step forward if we can support reranker API.

Describe the solution you'd like
There are many reranker models out there, some famous names: bce-reranker-base_v1, CohereRerank, bge-reranker-v2-m3. I think the Jina reranker API would be a good format to implement. https://jina.ai/reranker/#apiform

Describe alternatives you've considered
n/a

Additional context
The benchmark regarding embedding+reranker for RAG:

[thiner]

I spent some time trying to figure out how to implement. Below are my workouts:

1. Use reranker model is as easy as using embedding model. Below is the example of bce-reranker-base-v1.

from sentence_transformers import CrossEncoder

# init reranker model
model = CrossEncoder('maidalun1020/bce-reranker-base_v1', max_length=512)

# calculate scores of sentence pairs
scores = model.predict(sentence_pairs)

2. backend/python/sentencetransformers/sentencetransoformers.py supports embedding already, maybe add one more Rerank method with above code is enough. The major work would be refactor protobuf and API component.

[mudler]

that's definetly a good addition - adding to our roadmap. Thanks also for pointing out the steps

[localai-bot]

Thank you for your feature request, Thiner! We appreciate the details and the code example you provided to help illustrate the implementation. Adding reranker API support to LocalAI would indeed be a valuable improvement, especially considering the current state-of-the-art solutions for RAG systems.

To proceed with this feature request, we will evaluate the feasibility of incorporating a reranker API implementation similar to the example you provided, using a model like bce-reranker-base_v1 or other popular options. We will also research the best practices for integrating this functionality into LocalAI's existing architecture.

We will update the roadmap to include this feature request. Once we have completed our internal discussions and evaluations, we will provide an estimate of when this feature can be implemented, along with any additional details regarding the implementation.

Feel free to reach out if you have any further questions or concerns in the meantime. Thanks again for your suggestion and for helping us improve LocalAI!

[shuther]

as a possible more configurable approach we may benefit from the project https://github.com/AnswerDotAI/rerankers

[mudler]

having a quick look at this - let's see if can get something working before the weekend

[mudler]

ok that was easy enough - going to open up a PR soon

[mudler]

@thiner @BradKML will be part of the next release - if you want to test out before that, PR is at #2121