[Question]: Does librechat support aphrodite-engine?

[olidietzel]

What is your question?

Does librechat support the aphrodite-engines openai api implementation?

Would be nice to have a professional looking (and working) UI for a local LLM setup based on aphrodite.

More Details

pip install aphrodite-engine

python -m aphrodite.endpoints.openai.api_server --model PygmalionAI/pygmalion-2-7b

This will create a OpenAI-compatible API server that can be accessed at port 2242 of the localhost. You can plug in the API into a UI that supports Kobold, such as SillyTavern.

What is the main subject of your question?

No response

Screenshots

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

If it’s OpenAI compatible it will likely work. I’ve never heard of anyone using this but I can try it when I get the chance. You can also try setting it up: https://docs.librechat.ai/install/configuration/custom_config.html#docker-setup

[olidietzel]

We had a good start with ollama, but now we will buy bigger boxes and need something more enterprise like.

The USP of aphrodite is said to be speed and multithreading.
https://github.com/PygmalionAI/aphrodite-engine#performance

We will give
https://docs.librechat.ai/install/configuration/custom_config.html#docker-setup
a try! Thx a lot!

[o0LINNY0o]

did it work ??? im looking at trying to get Koboldcpp going , pretty new at this and having issues following the instructions