Skip to content

2. Services

av edited this page Mar 15, 2025 · 48 revisions

Various services that are integrated with Harbor. The link in the service name will lead you to a dedicated page in Harbor's wiki with details on getting started with the service.

Frontends

This section covers services that can provide you with an interface for interacting with the language models.

  • AnythingLLM Frontend, Partial Support
    The all-in-one Desktop & Docker AI application with built-in RAG, AI agents, and more.

  • BionicGPT Frontend
    on-premise LLM web UI with support for OpenAI-compatible backends

  • Chat Nio Frontend
    Comprehensive LLM web interface with built-in marketplace

  • ComfyUI Frontend, Workflows
    The most powerful and modular diffusion model GUI, api and backend with a graph/nodes interface.

  • Hollama Frontend
    A minimal web-UI for talking to Ollama servers.

  • HuggingFace ChatUI Frontend
    A chat interface using open source models, eg OpenAssistant or Llama. It is a SvelteKit app and it powers the HuggingChat app on hf.co/chat.

  • KoboldCpp Satellite, Frontend, Backend
    KoboldCpp is an easy-to-use AI text-generation software for GGML and GGUF models.

  • LibreChat Frontend
    Open-source ChatGPT UI alternative supporting multiple AI providers (Anthropic, AWS, OpenAI, Azure, Groq, Mistral, Google) with features like model switching, message search, and multi-user support. Includes integration with DALL-E-3 and various APIs.

  • Lobe Chat Frontend
    An open-source, modern-design AI chat framework. Supports Multi AI Providers( OpenAI / Claude 3 / Gemini / Ollama / Azure / DeepSeek), Knowledge Base (file upload / knowledge management / RAG ), Multi-Modals (Vision/TTS) and plugin system.

  • Mikupad Frontend
    LLM Frontend in a single HMTL file

  • mistral.rs Frontend
    Blazingly fast LLM inference.

  • ol1 Frontend
    A simple Gradio app implementing an o1-like chain of reasoning with Ollama.

  • Omnichain Frontend
    Visual programming for AI language models

  • Open WebUI Frontend
    widely adopted and feature rich web interface for interacting with LLMs. Supports OpenAI-compatible and Ollama backends, multi-users, multi-model chats, custom prompts, TTS, Web RAG, RAG, and much much more.

  • oterm CLI, Frontend
    The text-based terminal client for Ollama.

  • Parllama Frontend
    TUI for Ollama

  • RAGLite Satellite, Frontend
    Python toolkit for Retrieval-Augmented Generation (RAG)

Backends

This section covers services that provide the LLM inference capabilities.

  • AirLLM Backend
    70B inference with single 4GB GPU (very slow, though)

  • Aphrodite Backend
    Large-scale LLM inference engine

  • faster-whisper-server Backend, Audio, Partial Support
    Legacy version of Speaches, use that instead.

  • KoboldCpp Satellite, Frontend, Backend
    KoboldCpp is an easy-to-use AI text-generation software for GGML and GGUF models.

  • KTransformers Backend
    A Flexible Framework for Experiencing Cutting-edge LLM Inference Optimizations

  • llama.cpp Backend
    LLM inference in C/C++

  • lmdeploy Backend, Partial Support

  • Nexa SDK Backend, Partial Support
    Nexa SDK is a comprehensive toolkit for supporting ONNX and GGML models.

  • Ollama Backend
    Get up and running with Llama 3.2, Mistral, Gemma 3, and other large language models.

  • openedai-speech Backend, Audio
    An OpenAI API compatible text to speech server using Coqui AI's xtts_v2 and/or piper tts as the backend.

  • Parler Backend, Audio
    Inference and training library for high-quality TTS models.

  • SGLang Backend
    SGLang is a fast serving framework for large language models and vision language models.

  • Speaches Backend, Audio
    an OpenAI API-compatible speech server (formerly faster-whisper-server), both TTS and STT

  • TabbyAPI Backend
    An OAI compatible exllamav2 API that's both lightweight and fast

  • Text Generation Inference Backend
    Inference engine from HuggingFace.

  • vLLM Backend
    A high-throughput and memory-efficient inference and serving engine for LLMs

Satellites

Additional services that can be integrated with various Frontends and Backends to enable more features.

  • aichat Satellite, CLI
    All-in-one LLM CLI tool featuring Shell Assistant, Chat-REPL, RAG, AI tools & agents.

  • Aider Satellite, CLI
    Aider is AI pair programming in your terminal.

  • autogpt Satellite, Partial Support
    Create, deploy, and manage continuous AI agents that automate complex workflows.

  • Bolt.new Satellite
    Prompt, run, edit, and deploy full-stack web applications.

  • cloudflared Satellite, API, CLI
    A helper service allowing to expose Harbor services over the internet.

  • cmdh Satellite, CLI
    Create Linux commands from natural language, in the shell.

  • Dify Satellite, Workflows
    An open-source LLM app development platform.

  • Fabric Satellite, CLI
    LLM-driven processing of the text data in the terminal.

  • Flowise Satellite, Workflows
    Drag & drop UI to build your customized LLM flow.

  • gptme Satellite, CLI
    A simple CLI tool to interact with LLMs.

  • Harbor Bench Satellite, CLI, Built-in, Eval
    Harbor's own tool to evaluate LLMs and inference backends against custom tasks.

  • Harbor Boost Satellite, API, Built-in
    Connects to downstream LLM API and serves a wrapper with custom workflow. For example, it can be used to add a CoT (Chain of Thought) to an existing LLM API, and much more. Scriptable with Python.

  • JupyterLab Satellite
    Helper service to author/run Jupyter notebooks in Python with access to Harbor services.

  • K6 Satellite, CLI
    A modern load testing tool, using Go and JavaScript - https://k6.io

  • KoboldCpp Satellite, Frontend, Backend
    KoboldCpp is an easy-to-use AI text-generation software for GGML and GGUF models.

  • LangFlow Satellite, Workflows
    A low-code app builder for RAG and multi-agent AI applications.

  • LangFuse Satellite, API
    Open source LLM engineering platform: LLM Observability, metrics, evals, prompt management, playground, datasets.

  • Latent Scope Satellite
    A new kind of workflow + tool for visualizing and exploring datasets through the lens of latent spaces.

  • LiteLLM Satellite, API
    LLM proxy that can aggregate multiple inference APIs together into a single endpoint.

  • LitLytics Satellite, Partial Support, Workflows
    Simple analytics platform that leverages LLMs to automate data analysis.

  • lm-evaluation-harness Satellite, CLI, Eval
    A de-facto standard framework for the few-shot evaluation of language models.

  • Morphic Satellite
    An AI-powered search engine with a generative UI, similar to Perplexity and Perplexica.

  • n8n Satellite, Workflows
    Fair-code workflow automation platform with native AI capabilities.

  • OmniParser Satellite
    A simple screen parsing tool towards pure vision based GUI agent.

  • Open Interpreter Satellite, CLI
    A natural language interface for computers.

  • Open WebUI Pipelines Satellite, API, Workflows
    UI-Agnostic OpenAI API Plugin Framework.

  • OpenHands Satellite, Partial Support
    A platform for software development agents powered by AI.

  • OptiLLM Satellite, API
    Optimising LLM proxy that implements many advanced workflows to boost the performance of the LLMs.

  • Perplexica Satellite
    An AI-powered search engine. It is an Open source alternative to Perplexity AI.

  • Plandex Satellite, CLI
    AI driven development in your terminal.

  • Promptfoo Satellite, CLI
    Test your prompts, agents, and RAGs. A developer-friendly local tool for testing LLM applications.

  • Qdrant Satellite
    Qdrant - High-performance, massive-scale Vector Database and Vector Search Engine.

  • RAGLite Satellite, Frontend
    Python toolkit for Retrieval-Augmented Generation (RAG)

  • Repopack Satellite, CLI
    A powerful tool that packs your entire repository into a single, AI-friendly file.

  • SearXNG Satellite
    A privacy-respecting, hackable metasearch engine. Highly configurable and can be used for Web RAG use-cases.

  • SQL Chat Satellite
    Chat-based SQL client, which uses natural language to communicate with the database.

  • TextGrad Satellite
    Automatic "Differentiation" via Text - using large language models to backpropagate textual gradients.

  • Traefik Satellite, API
    A modern HTTP reverse proxy and load balancer that makes deploying microservices easy.

  • txtai RAG Satellite
    RAG WebUI built with txtai.

  • Webtop Satellite
    Linux in a web browser supporting popular desktop environments.

Clone this wiki locally