-
-
Notifications
You must be signed in to change notification settings - Fork 107
2. Services
Various services that are integrated with Harbor. The link in the service name will lead you to a dedicated page in Harbor's wiki with details on getting started with the service.
This section covers services that can provide you with an interface for interacting with the language models.
-
AnythingLLM
Frontend
,Partial Support
The all-in-one Desktop & Docker AI application with built-in RAG, AI agents, and more. -
BionicGPT
Frontend
on-premise LLM web UI with support for OpenAI-compatible backends -
Chat Nio
Frontend
Comprehensive LLM web interface with built-in marketplace -
ComfyUI
Frontend
,Workflows
The most powerful and modular diffusion model GUI, api and backend with a graph/nodes interface. -
Hollama
Frontend
A minimal web-UI for talking to Ollama servers. -
HuggingFace ChatUI
Frontend
A chat interface using open source models, eg OpenAssistant or Llama. It is a SvelteKit app and it powers the HuggingChat app on hf.co/chat. -
KoboldCpp
Satellite
,Frontend
,Backend
KoboldCpp is an easy-to-use AI text-generation software for GGML and GGUF models. -
LibreChat
Frontend
Open-source ChatGPT UI alternative supporting multiple AI providers (Anthropic, AWS, OpenAI, Azure, Groq, Mistral, Google) with features like model switching, message search, and multi-user support. Includes integration with DALL-E-3 and various APIs. -
Lobe Chat
Frontend
An open-source, modern-design AI chat framework. Supports Multi AI Providers( OpenAI / Claude 3 / Gemini / Ollama / Azure / DeepSeek), Knowledge Base (file upload / knowledge management / RAG ), Multi-Modals (Vision/TTS) and plugin system. -
Mikupad
Frontend
LLM Frontend in a single HMTL file -
mistral.rs
Frontend
Blazingly fast LLM inference. -
ol1
Frontend
A simple Gradio app implementing an o1-like chain of reasoning with Ollama. -
Omnichain
Frontend
Visual programming for AI language models -
Open WebUI
Frontend
widely adopted and feature rich web interface for interacting with LLMs. Supports OpenAI-compatible and Ollama backends, multi-users, multi-model chats, custom prompts, TTS, Web RAG, RAG, and much much more. -
oterm
CLI
,Frontend
The text-based terminal client for Ollama. -
Parllama
Frontend
TUI for Ollama -
RAGLite
Satellite
,Frontend
Python toolkit for Retrieval-Augmented Generation (RAG)
This section covers services that provide the LLM inference capabilities.
-
AirLLM
Backend
70B inference with single 4GB GPU (very slow, though) -
Aphrodite
Backend
Large-scale LLM inference engine -
faster-whisper-server
Backend
,Audio
,Partial Support
Legacy version of Speaches, use that instead. -
KoboldCpp
Satellite
,Frontend
,Backend
KoboldCpp is an easy-to-use AI text-generation software for GGML and GGUF models. -
KTransformers
Backend
A Flexible Framework for Experiencing Cutting-edge LLM Inference Optimizations -
llama.cpp
Backend
LLM inference in C/C++ -
lmdeploy
Backend
,Partial Support
-
Nexa SDK
Backend
,Partial Support
Nexa SDK is a comprehensive toolkit for supporting ONNX and GGML models. -
Ollama
Backend
Get up and running with Llama 3.2, Mistral, Gemma 3, and other large language models. -
openedai-speech
Backend
,Audio
An OpenAI API compatible text to speech server using Coqui AI's xtts_v2 and/or piper tts as the backend. -
Parler
Backend
,Audio
Inference and training library for high-quality TTS models. -
SGLang
Backend
SGLang is a fast serving framework for large language models and vision language models. -
Speaches
Backend
,Audio
an OpenAI API-compatible speech server (formerlyfaster-whisper-server
), both TTS and STT -
TabbyAPI
Backend
An OAI compatible exllamav2 API that's both lightweight and fast -
Text Generation Inference
Backend
Inference engine from HuggingFace. -
vLLM
Backend
A high-throughput and memory-efficient inference and serving engine for LLMs
Additional services that can be integrated with various Frontends and Backends to enable more features.
-
aichat
Satellite
,CLI
All-in-one LLM CLI tool featuring Shell Assistant, Chat-REPL, RAG, AI tools & agents. -
Aider
Satellite
,CLI
Aider is AI pair programming in your terminal. -
autogpt
Satellite
,Partial Support
Create, deploy, and manage continuous AI agents that automate complex workflows. -
Bolt.new
Satellite
Prompt, run, edit, and deploy full-stack web applications. -
cloudflared
Satellite
,API
,CLI
A helper service allowing to expose Harbor services over the internet. -
cmdh
Satellite
,CLI
Create Linux commands from natural language, in the shell. -
Dify
Satellite
,Workflows
An open-source LLM app development platform. -
Fabric
Satellite
,CLI
LLM-driven processing of the text data in the terminal. -
Flowise
Satellite
,Workflows
Drag & drop UI to build your customized LLM flow. -
gptme
Satellite
,CLI
A simple CLI tool to interact with LLMs. -
Harbor Bench
Satellite
,CLI
,Built-in
,Eval
Harbor's own tool to evaluate LLMs and inference backends against custom tasks. -
Harbor Boost
Satellite
,API
,Built-in
Connects to downstream LLM API and serves a wrapper with custom workflow. For example, it can be used to add a CoT (Chain of Thought) to an existing LLM API, and much more. Scriptable with Python. -
JupyterLab
Satellite
Helper service to author/run Jupyter notebooks in Python with access to Harbor services. -
K6
Satellite
,CLI
A modern load testing tool, using Go and JavaScript - https://k6.io -
KoboldCpp
Satellite
,Frontend
,Backend
KoboldCpp is an easy-to-use AI text-generation software for GGML and GGUF models. -
LangFlow
Satellite
,Workflows
A low-code app builder for RAG and multi-agent AI applications. -
LangFuse
Satellite
,API
Open source LLM engineering platform: LLM Observability, metrics, evals, prompt management, playground, datasets. -
Latent Scope
Satellite
A new kind of workflow + tool for visualizing and exploring datasets through the lens of latent spaces. -
LiteLLM
Satellite
,API
LLM proxy that can aggregate multiple inference APIs together into a single endpoint. -
LitLytics
Satellite
,Partial Support
,Workflows
Simple analytics platform that leverages LLMs to automate data analysis. -
lm-evaluation-harness
Satellite
,CLI
,Eval
A de-facto standard framework for the few-shot evaluation of language models. -
Morphic
Satellite
An AI-powered search engine with a generative UI, similar to Perplexity and Perplexica. -
n8n
Satellite
,Workflows
Fair-code workflow automation platform with native AI capabilities. -
OmniParser
Satellite
A simple screen parsing tool towards pure vision based GUI agent. -
Open Interpreter
Satellite
,CLI
A natural language interface for computers. -
Open WebUI Pipelines
Satellite
,API
,Workflows
UI-Agnostic OpenAI API Plugin Framework. -
OpenHands
Satellite
,Partial Support
A platform for software development agents powered by AI. -
OptiLLM
Satellite
,API
Optimising LLM proxy that implements many advanced workflows to boost the performance of the LLMs. -
Perplexica
Satellite
An AI-powered search engine. It is an Open source alternative to Perplexity AI. -
Plandex
Satellite
,CLI
AI driven development in your terminal. -
Promptfoo
Satellite
,CLI
Test your prompts, agents, and RAGs. A developer-friendly local tool for testing LLM applications. -
Qdrant
Satellite
Qdrant - High-performance, massive-scale Vector Database and Vector Search Engine. -
RAGLite
Satellite
,Frontend
Python toolkit for Retrieval-Augmented Generation (RAG) -
Repopack
Satellite
,CLI
A powerful tool that packs your entire repository into a single, AI-friendly file. -
SearXNG
Satellite
A privacy-respecting, hackable metasearch engine. Highly configurable and can be used for Web RAG use-cases. -
SQL Chat
Satellite
Chat-based SQL client, which uses natural language to communicate with the database. -
TextGrad
Satellite
Automatic "Differentiation" via Text - using large language models to backpropagate textual gradients. -
Traefik
Satellite
,API
A modern HTTP reverse proxy and load balancer that makes deploying microservices easy. -
txtai RAG
Satellite
RAG WebUI built with txtai. -
Webtop
Satellite
Linux in a web browser supporting popular desktop environments.