2. Services

Various services that are integrated with Harbor. The link in the service name will lead you to a dedicated page in Harbor's wiki with details on getting started with the service.

Frontends

This section covers services that can provide you with an interface for interacting with the language models.

• AnythingLLM Frontend, Partial Support
  The all-in-one Desktop & Docker AI application with built-in RAG, AI agents, and more.

• BionicGPT Frontend
  on-premise LLM web UI with support for OpenAI-compatible backends

• Chat Nio Frontend
  Comprehensive LLM web interface with built-in marketplace

• ComfyUI Frontend, Workflows
  The most powerful and modular diffusion model GUI, api and backend with a graph/nodes interface.

• Hollama Frontend
  A minimal web-UI for talking to Ollama servers.

• HuggingFace ChatUI Frontend
  A chat interface using open source models, eg OpenAssistant or Llama. It is a SvelteKit app and it powers the HuggingChat app on hf.co/chat.

• KoboldCpp Satellite, Frontend, Backend
  KoboldCpp is an easy-to-use AI text-generation software for GGML and GGUF models.

• LibreChat Frontend
  Open-source ChatGPT UI alternative supporting multiple AI providers (Anthropic, AWS, OpenAI, Azure, Groq, Mistral, Google) with features like model switching, message search, and multi-user support. Includes integration with DALL-E-3 and various APIs.

• Lobe Chat Frontend
  An open-source, modern-design AI chat framework. Supports Multi AI Providers( OpenAI / Claude 3 / Gemini / Ollama / Azure / DeepSeek), Knowledge Base (file upload / knowledge management / RAG ), Multi-Modals (Vision/TTS) and plugin system.

• Mikupad Frontend
  LLM Frontend in a single HMTL file

• mistral.rs Frontend
  Blazingly fast LLM inference.

• ol1 Frontend
  A simple Gradio app implementing an o1-like chain of reasoning with Ollama.

• Omnichain Frontend
  Visual programming for AI language models

• Open WebUI Frontend
  widely adopted and feature rich web interface for interacting with LLMs. Supports OpenAI-compatible and Ollama backends, multi-users, multi-model chats, custom prompts, TTS, Web RAG, RAG, and much much more.

• oterm CLI, Frontend
  The text-based terminal client for Ollama.

• Parllama Frontend
  TUI for Ollama

• RAGLite Satellite, Frontend
  Python toolkit for Retrieval-Augmented Generation (RAG)

Backends

This section covers services that provide the LLM inference capabilities.

• AirLLM Backend
  70B inference with single 4GB GPU (very slow, though)

• Aphrodite Backend
  Large-scale LLM inference engine

• faster-whisper-server Backend, Audio, Partial Support
  Legacy version of Speaches, use that instead.

• KoboldCpp Satellite, Frontend, Backend
  KoboldCpp is an easy-to-use AI text-generation software for GGML and GGUF models.

• KTransformers Backend
  A Flexible Framework for Experiencing Cutting-edge LLM Inference Optimizations

• llama.cpp Backend
  LLM inference in C/C++

• lmdeploy Backend, Partial Support
  

• Nexa SDK Backend, Partial Support
  Nexa SDK is a comprehensive toolkit for supporting ONNX and GGML models.

• Ollama Backend
  Get up and running with Llama 3.2, Mistral, Gemma 3, and other large language models.

• openedai-speech Backend, Audio
  An OpenAI API compatible text to speech server using Coqui AI's xtts_v2 and/or piper tts as the backend.

• Parler Backend, Audio
  Inference and training library for high-quality TTS models.

• SGLang Backend
  SGLang is a fast serving framework for large language models and vision language models.

• Speaches Backend, Audio
  an OpenAI API-compatible speech server (formerly faster-whisper-server), both TTS and STT

• TabbyAPI Backend
  An OAI compatible exllamav2 API that's both lightweight and fast

• Text Generation Inference Backend
  Inference engine from HuggingFace.

• vLLM Backend
  A high-throughput and memory-efficient inference and serving engine for LLMs

Satellites

Additional services that can be integrated with various Frontends and Backends to enable more features.

• aichat Satellite, CLI
  All-in-one LLM CLI tool featuring Shell Assistant, Chat-REPL, RAG, AI tools & agents.

• Aider Satellite, CLI
  Aider is AI pair programming in your terminal.

• autogpt Satellite, Partial Support
  Create, deploy, and manage continuous AI agents that automate complex workflows.

• Bolt.new Satellite
  Prompt, run, edit, and deploy full-stack web applications.

• cloudflared Satellite, API, CLI
  A helper service allowing to expose Harbor services over the internet.

• cmdh Satellite, CLI
  Create Linux commands from natural language, in the shell.

• Dify Satellite, Workflows
  An open-source LLM app development platform.

• Fabric Satellite, CLI
  LLM-driven processing of the text data in the terminal.

• Flowise Satellite, Workflows
  Drag & drop UI to build your customized LLM flow.

• gptme Satellite, CLI
  A simple CLI tool to interact with LLMs.

• Harbor Bench Satellite, CLI, Built-in, Eval
  Harbor's own tool to evaluate LLMs and inference backends against custom tasks.

• Harbor Boost Satellite, API, Built-in
  Connects to downstream LLM API and serves a wrapper with custom workflow. For example, it can be used to add a CoT (Chain of Thought) to an existing LLM API, and much more. Scriptable with Python.

• JupyterLab Satellite
  Helper service to author/run Jupyter notebooks in Python with access to Harbor services.

• K6 Satellite, CLI
  A modern load testing tool, using Go and JavaScript - https://k6.io

• KoboldCpp Satellite, Frontend, Backend
  KoboldCpp is an easy-to-use AI text-generation software for GGML and GGUF models.

• LangFlow Satellite, Workflows
  A low-code app builder for RAG and multi-agent AI applications.

• LangFuse Satellite, API
  Open source LLM engineering platform: LLM Observability, metrics, evals, prompt management, playground, datasets.

• Latent Scope Satellite
  A new kind of workflow + tool for visualizing and exploring datasets through the lens of latent spaces.

• LiteLLM Satellite, API
  LLM proxy that can aggregate multiple inference APIs together into a single endpoint.

• LitLytics Satellite, Partial Support, Workflows
  Simple analytics platform that leverages LLMs to automate data analysis.

• lm-evaluation-harness Satellite, CLI, Eval
  A de-facto standard framework for the few-shot evaluation of language models.

• Morphic Satellite
  An AI-powered search engine with a generative UI, similar to Perplexity and Perplexica.

• n8n Satellite, Workflows
  Fair-code workflow automation platform with native AI capabilities.

• OmniParser Satellite
  A simple screen parsing tool towards pure vision based GUI agent.

• Open Interpreter Satellite, CLI
  A natural language interface for computers.

• Open WebUI Pipelines Satellite, API, Workflows
  UI-Agnostic OpenAI API Plugin Framework.

• OpenHands Satellite, Partial Support
  A platform for software development agents powered by AI.

• OptiLLM Satellite, API
  Optimising LLM proxy that implements many advanced workflows to boost the performance of the LLMs.

• Perplexica Satellite
  An AI-powered search engine. It is an Open source alternative to Perplexity AI.

• Plandex Satellite, CLI
  AI driven development in your terminal.

• Promptfoo Satellite, CLI
  Test your prompts, agents, and RAGs. A developer-friendly local tool for testing LLM applications.

• Qdrant Satellite
  Qdrant - High-performance, massive-scale Vector Database and Vector Search Engine.

• RAGLite Satellite, Frontend
  Python toolkit for Retrieval-Augmented Generation (RAG)

• Repopack Satellite, CLI
  A powerful tool that packs your entire repository into a single, AI-friendly file.

• SearXNG Satellite
  A privacy-respecting, hackable metasearch engine. Highly configurable and can be used for Web RAG use-cases.

• SQL Chat Satellite
  Chat-based SQL client, which uses natural language to communicate with the database.

• TextGrad Satellite
  Automatic "Differentiation" via Text - using large language models to backpropagate textual gradients.

• Traefik Satellite, API
  A modern HTTP reverse proxy and load balancer that makes deploying microservices easy.

• txtai RAG Satellite
  RAG WebUI built with txtai.

• Webtop Satellite
  Linux in a web browser supporting popular desktop environments.