diff --git a/docs/docs/assets/logo-with-bg.png b/docs/docs/assets/logo-with-bg.png new file mode 100644 index 00000000..b3efbc6b Binary files /dev/null and b/docs/docs/assets/logo-with-bg.png differ diff --git a/docs/docs/docker-compose-setup.md b/docs/docs/docker-compose-setup.md index df96f131..d6119a34 100644 --- a/docs/docs/docker-compose-setup.md +++ b/docs/docs/docker-compose-setup.md @@ -1,142 +1,234 @@ # Docker Compose Setup Guide -Get Exosphere running locally with Docker Compose in under 2 minutes. This guide provides everything you need to run the complete Exosphere stack locally for development and testing. +Get Exosphere running locally with Docker Compose in under 2 minutes. + +## Setup + +This guide provides everything you need to run the complete Exosphere stack locally for development and testing. + +Exosphere uses MongoDB as the database to manage states. You can either have mongodb setup locally or use a cloud hosted instance (recommended) + +=== "Exosphere Local Setup with Cloud MongoDB(Rec)" + + 1) Download Docker Compose Files + + First, download the Docker Compose files from the GitHub repository: + + === "curl" + ```bash + # Download docker-compose file for cloud MongoDB (recommended) + curl -O https://raw.githubusercontent.com/exospherehost/exospherehost/main/docker-compose/docker-compose.yml + ``` + === "wget" + ```bash + # Download docker-compose file for cloud MongoDB (recommended) + wget https://raw.githubusercontent.com/exospherehost/exospherehost/main/docker-compose/docker-compose.yml + ``` + + 2) Set up a cloud MongoDB + Set up a mongodb cluster in a provider of your choice MongoDB Atlas, AWS DocumentDB. Get the mongo db url. + + 3) Set up the secrets in env + === ".env file" + Create a `.env` file with your MongoDB connection: + ```bash + MONGO_URI=mongodb+srv://username:password@your-cluster.mongodb.net/ + ``` + === "Environment Variables" + On your terminal: + ```bash + export MONGO_URI=mongodb+srv://username:password@your-cluster.mongodb.net/ + ``` + + 4) Download and start the services: + ```bash + docker compose -f docker-compose.yml up -d + ``` + + 5) Done! This will start: + + - Exosphere State Manager : [http://localhost:8000](http://localhost:8000) + - Exosphere Dashboard: [http://localhost:3000](http://localhost:3000) + +=== "Exosphere Local Setup with Local MongoDB" + + For quick local testing only: + + 1) Download Docker Compose Files + + First, download the Docker Compose files from the GitHub repository: + + === "curl" + ```bash + # Download docker-compose file with local MongoDB included + curl -O https://raw.githubusercontent.com/exospherehost/exospherehost/main/docker-compose/docker-compose-with-mongodb.yml + ``` + === "wget" + ```bash + # Download docker-compose file with local MongoDB included + wget https://raw.githubusercontent.com/exospherehost/exospherehost/main/docker-compose/docker-compose-with-mongodb.yml + ``` + + 2) Download and start the services: + + ```bash + curl -O https://raw.githubusercontent.com/exospherehost/exospherehost/main/docker-compose/docker-compose-with-mongodb.yml && docker compose -f docker-compose-with-mongodb.yml up -d + ``` + + 3) Done! This will start: + + - MongoDB database: [http://localhost:27017](http://localhost:27017) + - Exosphere Dashboard: [http://localhost:3000](http://localhost:3000) + - Exosphere State Manager: [http://localhost:8000](http://localhost:8000) -## Download Docker Compose Files -First, download the Docker Compose files from the GitHub repository: +## Beta Version -### Option 1: Download Both Files +To run the latest beta version of Exosphere with the newest features, replace container tags with `beta-latest`: -```bash -# Download docker-compose file for cloud MongoDB (recommended) -curl -O https://raw.githubusercontent.com/exospherehost/exospherehost/main/docker-compose/docker-compose.yml +=== ".env File" -# Download docker-compose file with local MongoDB included -curl -O https://raw.githubusercontent.com/exospherehost/exospherehost/main/docker-compose/docker-compose-with-mongodb.yml -``` + ```bash + EXOSPHERE_TAG=beta-latest + ``` +=== "Environment Variables" -### Option 2: Using wget + ```bash + export EXOSPHERE_TAG=beta-latest + ``` -```bash -# Download docker-compose file for cloud MongoDB (recommended) -wget https://raw.githubusercontent.com/exospherehost/exospherehost/main/docker-compose/docker-compose.yml +Get the docker image running: -# Download docker-compose file with local MongoDB included -wget https://raw.githubusercontent.com/exospherehost/exospherehost/main/docker-compose/docker-compose-with-mongodb.yml -``` +=== "Cloud Mongodb" -## Quick Start + ```bash + docker compose -f docker-compose.yml up -d + ``` +=== "Local Mongodb" -We **recommend using a cloud MongoDB managed service** (MongoDB Atlas, AWS DocumentDB, etc.) for better performance, reliability, and security. + ```bash + docker compose -f docker-compose-with-mongodb.yml up -d + ``` -### Option 1: With Cloud MongoDB (Recommended) +## Access Your Services -1. Set up a cloud MongoDB instance (MongoDB Atlas, AWS DocumentDB, etc.) +After running the Docker Compose command: -2. Create a `.env` file with your MongoDB connection: - ```bash - MONGO_URI=mongodb+srv://username:password@your-cluster.mongodb.net/ - ``` +- **Exosphere Dashboard**: `http://localhost:3000` +- **State Manager API**: `http://localhost:8000` +- **MongoDB** (if using with-mongodb): `mongodb://localhost:27017` (not HTTP - use MongoDB clients like MongoDB Compass or mongosh) +- **API Documentation**: `http://localhost:8000/docs` -3. Download and start the services: - ```bash - curl -O https://raw.githubusercontent.com/exospherehost/exospherehost/main/docker-compose/docker-compose.yml && docker compose -f docker-compose.yml up -d - ``` +## Development Commands -This will start: -- Exosphere State Manager (port 8000) -- Exosphere Dashboard (port 3000) +=== "Cloud Mongodb" + ```bash + # Start services in background + docker compose -f docker-compose.yml up -d -### Option 2: With Local MongoDB (Development Only) + # View logs + docker compose -f docker-compose.yml logs -f -For quick local testing only: + # Stop services + docker compose -f docker-compose.yml down -```bash -curl -O https://raw.githubusercontent.com/exospherehost/exospherehost/main/docker-compose/docker-compose-with-mongodb.yml && docker compose -f docker-compose-with-mongodb.yml up -d -``` + # Stop services and remove volumes + docker compose -f docker-compose.yml down -v -This will start: -- MongoDB database (port 27017) -- Exosphere State Manager (port 8000) -- Exosphere Dashboard (port 3000) + # Pull latest images + docker compose -f docker-compose.yml pull -## Beta Version + # Restart a specific service + docker compose -f docker-compose.yml restart exosphere-state-manager + ``` +=== "Local Mongodb" + ```bash + # Start services in background + docker compose -f docker-compose-with-mongodb.yml up -d -To run the latest beta version of Exosphere with the newest features, replace container tags with `beta-latest`: + # View logs + docker compose -f docker-compose-with-mongodb.yml logs -f -### Quick Beta Setup with Cloud MongoDB + # Stop services + docker compose -f docker-compose-with-mongodb.yml down -**Option 1: Environment Variable Approach (Recommended)** -```bash -# Download compose file -curl -O https://raw.githubusercontent.com/exospherehost/exospherehost/main/docker-compose/docker-compose.yml -# Set image tag via environment variable -export EXOSPHERE_TAG=beta-latest -# Set your MONGO_URI in .env file, then: -docker compose -f docker-compose.yml up -d -``` + # Stop services and remove volumes (⚠️ This will delete your data) + docker compose -f docker-compose-with-mongodb.yml down -v -**Option 2: In-place File Editing** -```bash -# Download and modify for beta with cloud MongoDB -curl -O https://raw.githubusercontent.com/exospherehost/exospherehost/main/docker-compose/docker-compose.yml + # Pull latest images + docker compose -f docker-compose-with-mongodb.yml pull -# Portable approach (works on all platforms): -perl -pi -e 's|(ghcr\.io/exospherehost/[^:]+):latest|\1:beta-latest|g' docker-compose.yml + # Restart a specific service + docker compose -f docker-compose-with-mongodb.yml restart exosphere-state-manager + ``` -# Platform-specific alternatives: -# Linux/GNU sed: -# sed -i 's|(ghcr\.io/exospherehost/[^:]+):latest|\1:beta-latest|g' docker-compose.yml -# macOS/BSD sed: -# sed -i '' 's|(ghcr\.io/exospherehost/[^:]+):latest|\1:beta-latest|g' docker-compose.yml -# Set your MONGO_URI in .env file, then: -docker compose -f docker-compose.yml up -d -``` +### Check Service Health -### Quick Beta Setup with Local MongoDB +=== "Cloud Mongodb" + ```bash + # Check if all containers are running + docker compose -f docker-compose.yml ps -**Option 1: Environment Variable Approach (Recommended)** -```bash -# Download compose file -curl -O https://raw.githubusercontent.com/exospherehost/exospherehost/main/docker-compose/docker-compose-with-mongodb.yml -# Set image tag via environment variable -export EXOSPHERE_TAG=beta-latest -docker compose -f docker-compose-with-mongodb.yml up -d -``` + # Check state manager health + curl http://localhost:8000/health -**Option 2: In-place File Editing** -```bash -# Download and modify for beta with local MongoDB -curl -O https://raw.githubusercontent.com/exospherehost/exospherehost/main/docker-compose/docker-compose-with-mongodb.yml + # View container logs + docker compose -f docker-compose.yml logs exosphere-state-manager + ``` -# Portable approach (works on all platforms): -perl -pi -e 's|(ghcr\.io/exospherehost/[^:]+):latest|\1:beta-latest|g' docker-compose-with-mongodb.yml -# Platform-specific alternatives: -# Linux/GNU sed: -# sed -i 's|(ghcr\.io/exospherehost/[^:]+):latest|\1:beta-latest|g' docker-compose-with-mongodb.yml -# macOS/BSD sed: -# sed -i '' 's|(ghcr\.io/exospherehost/[^:]+):latest|\1:beta-latest|g' docker-compose-with-mongodb.yml +=== "Local Mongodb" + ```bash + # Check if all containers are running + docker compose -f docker-compose-with-mongodb.yml ps -docker compose -f docker-compose-with-mongodb.yml up -d -``` + # Check state manager health + curl http://localhost:8000/health -### Manual Beta Setup (Alternative) + # View container logs + docker compose -f docker-compose-with-mongodb.yml logs exosphere-state-manager + ``` + +### Testing Your Setup + +You can validate your docker-compose configuration before starting services: + +=== "Cloud Mongodb" + ```bash + # Test configuration syntax + docker compose -f docker-compose.yml config + + # Pull all required images + docker compose -f docker-compose.yml pull + + # Start with health monitoring (--wait waits for all health checks) + docker compose -f docker-compose.yml up -d --wait + ``` + + The `--wait` flag ensures all services pass their health checks before returning. The startup sequence will be: + + 1. State Manager starts and passes health check (~10-30 seconds) + 2. Dashboard starts and passes health check (~10-30 seconds) + +=== "Local Mongodb" + ```bash + # Test configuration syntax + docker compose -f docker-compose-with-mongodb.yml config + + # Pull all required images + docker compose -f docker-compose-with-mongodb.yml pull + + # Start with health monitoring (--wait waits for all health checks) + docker compose -f docker-compose-with-mongodb.yml up -d --wait + ``` + + The `--wait` flag ensures all services pass their health checks before returning. The startup sequence will be: + + 1. MongoDB starts and passes health check (~10-30 seconds) + 2. State Manager starts and passes health check (~10-30 seconds) + 3. Dashboard starts and passes health check (~10-30 seconds) -1. Download the docker-compose file -2. Edit the file and change: - ```yaml - # Change from: - image: ghcr.io/exospherehost/exosphere-state-manager:latest - image: ghcr.io/exospherehost/exosphere-dashboard:latest - - # To: - image: ghcr.io/exospherehost/exosphere-state-manager:beta-latest - image: ghcr.io/exospherehost/exosphere-dashboard:beta-latest - ``` -3. Start the services with `docker compose up -d` ## Environment Variables @@ -153,13 +245,13 @@ docker compose -f docker-compose-with-mongodb.yml up -d ### Dashboard Environment Variables -#### 🔒 **Server-Side Variables (REQUIRED - NOT exposed to browser)** +#### **Server-Side Variables (REQUIRED - NOT exposed to browser)** | Variable | Description | Default Value | |----------|-------------|---------------| | `EXOSPHERE_STATE_MANAGER_URI` | State manager API URI | `http://exosphere-state-manager:8000` | | `EXOSPHERE_API_KEY` | **REQUIRED**: Secure API key for state manager access | `exosphere@123` | -#### 🌐 **Client-Side Variables (Optional - exposed to browser)** +#### **Client-Side Variables (Optional - exposed to browser)** | Variable | Description | Default Value | |----------|-------------|---------------| | `NEXT_PUBLIC_DEFAULT_NAMESPACE` | Default namespace for workflows | `default` | @@ -174,40 +266,6 @@ docker compose -f docker-compose-with-mongodb.yml up -d > > **🔐 Authentication**: When the dashboard sends API requests to the state-manager, the `EXOSPHERE_API_KEY` value is checked for equality with the `STATE_MANAGER_SECRET` value in the state-manager container. -## 🔒 **Security Architecture** - -### **Server-Side Rendering (SSR) Implementation** - -The Exosphere Dashboard has been refactored to use Next.js API routes for enhanced security: - -- **API Key Protection**: All sensitive credentials are stored server-side -- **Secure Communication**: Client never directly communicates with state-manager -- **Environment Isolation**: Sensitive config separated from public code -- **Production Ready**: Enterprise-grade security for production deployments - -### **API Route Structure** - -``` -/api/runs → Secure runs fetching with pagination -/api/graph-structure → Protected graph visualization data -/api/namespace-overview → Secure namespace summary -/api/graph-template → Protected template management -``` - -### **Security Benefits** - -1. **No API Key Exposure**: Credentials never visible in browser -2. **Server-Side Validation**: All requests validated before reaching state-manager -3. **Environment Security**: Sensitive variables isolated from client bundle -4. **Audit Trail**: All API calls logged server-side for monitoring - -### **Docker Security Features** - -- **Environment Variable Isolation**: Server-side environment variables are set in containers and available to server processes, but are not exposed to the browser/client bundle -- **Network Security**: Services communicate over isolated Docker networks -- **Health Checks**: Built-in health monitoring for all services -- **Resource Limits**: Configurable resource constraints for production use - ### MongoDB Local Setup Variables (for docker-compose-with-mongodb.yml only) | Variable | Description | Default Value | @@ -216,10 +274,6 @@ The Exosphere Dashboard has been refactored to use Next.js API routes for enhanc | `MONGO_INITDB_ROOT_PASSWORD` | MongoDB root password | `password` | | `MONGO_INITDB_DATABASE` | Initial MongoDB database | `exosphere` | -> **⚠️ Development Only — Do Not Use in Production** -> -> The MongoDB credentials above (`MONGO_INITDB_ROOT_USERNAME`, `MONGO_INITDB_ROOT_PASSWORD`, and `MONGO_INITDB_DATABASE`) are intended **only for local development**. These default values must **never be used in production environments**. For production deployments, use environment variable overrides, secure secrets management systems, or generate strong, unique credentials. Always rotate these values before deploying to any non-development environment. - ### SDK Environment Variables To use the Exosphere Python SDK with your running instance, set these environment variables in your development environment: @@ -229,19 +283,6 @@ To use the Exosphere Python SDK with your running instance, set these environmen | `EXOSPHERE_STATE_MANAGER_URI` | URL where the state manager is running | `http://localhost:8000` | | `EXOSPHERE_API_KEY` | API key for authentication (same as STATE_MANAGER_SECRET) | `exosphere@123` | -> **🔐 Authentication**: When making API requests to the state-manager, the `EXOSPHERE_API_KEY` value is checked for equality with the `STATE_MANAGER_SECRET` value in the state-manager container. - -**Example SDK setup**: -```bash -# Set environment variables for SDK -export EXOSPHERE_STATE_MANAGER_URI=http://localhost:8000 -export EXOSPHERE_API_KEY=exosphere@123 - -# Or add to your .env file for your Python project -echo "EXOSPHERE_STATE_MANAGER_URI=http://localhost:8000" >> .env -echo "EXOSPHERE_API_KEY=exosphere@123" >> .env -``` - ## Custom Configuration ### Using Your Own Environment Variables @@ -279,27 +320,6 @@ docker compose -f docker-compose-with-mongodb.yml up -d **Note**: The docker-compose files now automatically use `.env` files in the same directory and provide sensible defaults for all optional variables. -### Legacy Docker Compose v1 Compatibility - -If you have the legacy `docker-compose` (v1) binary instead of the newer `docker compose` (v2) plugin, you can use the hyphenated command format: - -```bash -# Legacy v1 commands (replace "docker compose" with "docker-compose"): -docker-compose -f docker-compose.yml up -d -docker-compose -f docker-compose-with-mongodb.yml up -d -docker-compose -f docker-compose-with-mongodb.yml logs -f -docker-compose -f docker-compose-with-mongodb.yml down -``` - -**Creating an alias** (optional): To use the same commands as shown in this guide, create an alias: -```bash -# Add to your shell profile (~/.bashrc, ~/.zshrc, etc.): -alias 'docker compose'='docker-compose' - -# Or install the v2 plugin: -# https://docs.docker.com/compose/install/ -``` - ### Generating a New Encryption Key To generate a secure encryption key for `SECRETS_ENCRYPTION_KEY`: @@ -312,71 +332,79 @@ python -c "import base64; import os; print(base64.urlsafe_b64encode(os.urandom(3 openssl rand -base64 32 ``` -## Access Your Services +## Security Architecture -After running the Docker Compose command: - -- **Exosphere Dashboard**: `http://localhost:3000` -- **State Manager API**: `http://localhost:8000` -- **MongoDB** (if using with-mongodb): `mongodb://localhost:27017` (not HTTP - use MongoDB clients like MongoDB Compass or mongosh) -- **API Documentation**: `http://localhost:8000/docs` +### Server-Side Rendering (SSR) Implementation -## Development Commands +The Exosphere Dashboard has been refactored to use Next.js API routes for enhanced security: -```bash -# Start services in background -docker compose -f docker-compose-with-mongodb.yml up -d +- **API Key Protection**: All sensitive credentials are stored server-side +- **Secure Communication**: Client never directly communicates with state-manager +- **Environment Isolation**: Sensitive config separated from public code +- **Production Ready**: Enterprise-grade security for production deployments -# View logs -docker compose -f docker-compose-with-mongodb.yml logs -f +### API Route Structure -# Stop services -docker compose -f docker-compose-with-mongodb.yml down +``` +/api/runs → Secure runs fetching with pagination +/api/graph-structure → Protected graph visualization data +/api/namespace-overview → Secure namespace summary +/api/graph-template → Protected template management +``` -# Stop services and remove volumes (⚠️ This will delete your data) -docker compose -f docker-compose-with-mongodb.yml down -v +### Security Benefits -# Pull latest images -docker compose -f docker-compose-with-mongodb.yml pull +1. **No API Key Exposure**: Credentials never visible in browser +2. **Server-Side Validation**: All requests validated before reaching state-manager +3. **Environment Security**: Sensitive variables isolated from client bundle +4. **Audit Trail**: All API calls logged server-side for monitoring -# Restart a specific service -docker compose -f docker-compose-with-mongodb.yml restart exosphere-state-manager -``` +### Docker Security Features -## Troubleshooting +- **Environment Variable Isolation**: Server-side environment variables are set in containers and available to server processes, but are not exposed to the browser/client bundle +- **Network Security**: Services communicate over isolated Docker networks +- **Health Checks**: Built-in health monitoring for all services +- **Resource Limits**: Configurable resource constraints for production use -### Check Service Health +> **🔐 Authentication**: When making API requests to the state-manager, the `EXOSPHERE_API_KEY` value is checked for equality with the `STATE_MANAGER_SECRET` value in the state-manager container. +**Example SDK setup**: ```bash -# Check if all containers are running -docker compose -f docker-compose-with-mongodb.yml ps - -# Check state manager health -curl http://localhost:8000/health +# Set environment variables for SDK +export EXOSPHERE_STATE_MANAGER_URI=http://localhost:8000 +export EXOSPHERE_API_KEY=exosphere@123 -# View container logs -docker compose -f docker-compose-with-mongodb.yml logs exosphere-state-manager +# Or add to your .env file for your Python project +echo "EXOSPHERE_STATE_MANAGER_URI=http://localhost:8000" >> .env +echo "EXOSPHERE_API_KEY=exosphere@123" >> .env ``` -### Testing Your Setup +> **⚠️ Development Only — Do Not Use in Production** +> +> The MongoDB credentials above (`MONGO_INITDB_ROOT_USERNAME`, `MONGO_INITDB_ROOT_PASSWORD`, and `MONGO_INITDB_DATABASE`) are intended **only for local development**. These default values must **never be used in production environments**. For production deployments, use environment variable overrides, secure secrets management systems, or generate strong, unique credentials. Always rotate these values before deploying to any non-development environment. -You can validate your docker-compose configuration before starting services: +## Legacy Docker Compose v1 Compatibility + +If you have the legacy `docker-compose` (v1) binary instead of the newer `docker compose` (v2) plugin, you can use the hyphenated command format: ```bash -# Test configuration syntax -docker compose -f docker-compose-with-mongodb.yml config +# Legacy v1 commands (replace "docker compose" with "docker-compose"): +docker-compose -f docker-compose.yml up -d +docker-compose -f docker-compose-with-mongodb.yml up -d +docker-compose -f docker-compose-with-mongodb.yml logs -f +docker-compose -f docker-compose-with-mongodb.yml down +``` -# Pull all required images -docker compose -f docker-compose-with-mongodb.yml pull +**Creating an alias** (optional): To use the same commands as shown in this guide, create an alias: +```bash +# Add to your shell profile (~/.bashrc, ~/.zshrc, etc.): +alias 'docker compose'='docker-compose' -# Start with health monitoring (--wait waits for all health checks) -docker compose -f docker-compose-with-mongodb.yml up -d --wait +# Or install the v2 plugin: +# https://docs.docker.com/compose/install/ ``` -The `--wait` flag ensures all services pass their health checks before returning. The startup sequence will be: -1. MongoDB starts and passes health check (~10-30 seconds) -2. State Manager starts and passes health check (~10-30 seconds) -3. Dashboard starts and passes health check (~10-30 seconds) +## Troubleshooting ### Common Issues diff --git a/docs/docs/exosphere/architecture.md b/docs/docs/exosphere/architecture.md index e05610d5..51b7838a 100644 --- a/docs/docs/exosphere/architecture.md +++ b/docs/docs/exosphere/architecture.md @@ -2,6 +2,8 @@ This document provides a comprehensive overview of Exosphere's top-level architecture, focusing on state execution, fanout mechanisms, and the unites keyword for stage unification. +> **📚 Getting Started**: For a complete local setup guide covering both the state manager and dashboard, see our [Local Setup Guide](./local-setup.md). + ## Overview Exosphere is built around a **state-based execution model** where workflows are composed of discrete states that can be executed independently. The architecture consists of several key components: diff --git a/docs/docs/exosphere/create-runtime.md b/docs/docs/exosphere/create-runtime.md index 7d83482a..a34fddcc 100644 --- a/docs/docs/exosphere/create-runtime.md +++ b/docs/docs/exosphere/create-runtime.md @@ -2,6 +2,8 @@ The `Runtime` class is the core component that manages the execution environment for your Exosphere nodes. It handles node registration, state polling, execution, and communication with the state manager. +> **📚 Getting Started**: For a complete local setup guide covering both the state manager and dashboard, see our [Local Setup Guide](./local-setup.md). + ## Runtime Setup Before creating a runtime, you need to set up the state manager and configure your environment variables. diff --git a/docs/docs/exosphere/dashboard.md b/docs/docs/exosphere/dashboard.md index 8b64d91a..4ea2897e 100644 --- a/docs/docs/exosphere/dashboard.md +++ b/docs/docs/exosphere/dashboard.md @@ -2,6 +2,8 @@ The Exosphere dashboard provides a comprehensive web interface for monitoring, debugging, and managing your workflows. This guide shows you how to set up and use the dashboard effectively. +> **📚 Getting Started**: For a complete local setup guide covering both the state manager and dashboard, see our [Local Setup Guide](./local-setup.md). + ## Dashboard Overview The Exosphere dashboard is a modern web application that connects to your state manager backend through secure server-side routes and provides: diff --git a/docs/docs/exosphere/local-setup.md b/docs/docs/exosphere/local-setup.md new file mode 100644 index 00000000..0f135b53 --- /dev/null +++ b/docs/docs/exosphere/local-setup.md @@ -0,0 +1,90 @@ +# Local Setup Guide + +This guide walks you through setting up Exosphere locally for development and testing. You'll learn how to run both the state manager and dashboard components, either using Docker Compose for quick setup or running them individually for more control. + +## Overview + +Exosphere consists of two main components that work together: + +1. **State Manager** - Backend service that handles workflow execution and state management +2. **Dashboard** - Web interface for monitoring and managing workflows + +You can set these up using: + +- **Docker Compose** (recommended for quick start) - runs both services together +- **Individual Setup** - run each service separately for development + +## Prerequisites + +Before you begin, ensure you have: + +- [Docker](https://docs.docker.com/get-docker/) installed (for Docker Compose approach) +- [Python 3.12+](https://www.python.org/downloads/) (for individual setup) +- [Node.js 18+](https://nodejs.org/) (for dashboard development) +- [MongoDB](https://www.mongodb.com/try/download/community) instance (cloud or local) + +## Setup + +=== "Docker Compose Setup (Recommended)" + + The fastest way to get Exosphere running locally is using Docker Compose. This approach handles all the configuration and networking automatically. + + Follow the guide: [Get Exosphere running locally with Docker](../docker-compose-setup.md) + +=== "Individual Component Setup" + + If you prefer more control over each component or want to develop locally, you can set up the state manager and dashboard separately. + + To get Exosphere state manager running, follow [State Manager Setup](./state-manager-setup.md) + + For complete monitoring dashboard setup details, see our [Dashboard Guide](./dashboard.md). + + +## Testing Your Setup + +Once both services are running, test the complete setup: + +1. **Verify state manager**: + ```bash + curl http://localhost:8000/health + ``` + +2. **Verify dashboard**: + ```bash + curl http://localhost:3000 + ``` + +3. **Check API documentation**: + - Open http://localhost:8000/docs in your browser + - Test the `/health` endpoint + +## Next Steps + +With your local Exosphere instance running, you're ready to: + +1. **[Register your first node](./register-node.md)** - Create custom processing logic +2. **[Create and run workflows](./create-graph.md)** - Build your first workflow +3. **[Monitor execution](./dashboard.md)** - Use the dashboard to track progress + +## Troubleshooting + +### Common Issues + +**Port conflicts**: If ports 8000 or 3000 are already in use, modify the port mappings in your Docker commands or configuration files. + +**MongoDB connection**: Ensure your MongoDB instance is accessible and the connection string is correct. + +**Authentication errors**: Verify that `EXOSPHERE_API_KEY` matches `STATE_MANAGER_SECRET`. + +**Dashboard can't connect**: Check that the state manager is running and accessible at the configured URI. + +### Getting Help + +- Check the [Docker Compose Setup Guide](../docker-compose-setup.md) for detailed configuration +- Review the [State Manager Setup Guide](./state-manager-setup.md) for backend troubleshooting +- Consult the [Dashboard Guide](./dashboard.md) for frontend issues +- Join our [Discord community](https://discord.com/invite/zT92CAgvkj) for support + +## Production Considerations + +For production deployment options, see our [hosted platform](https://exosphere.host) or contact our team for enterprise solutions. \ No newline at end of file diff --git a/docs/docs/exosphere/register-node.md b/docs/docs/exosphere/register-node.md index 00a178ee..37a70e65 100644 --- a/docs/docs/exosphere/register-node.md +++ b/docs/docs/exosphere/register-node.md @@ -2,6 +2,8 @@ Nodes are the building blocks of Exosphere workflows. Each node defines a specific piece of processing logic with typed inputs, outputs, and secrets. This guide shows you how to create and register custom nodes. +> **📚 Getting Started**: For a complete local setup guide covering both the state manager and dashboard, see our [Local Setup Guide](./local-setup.md). + ## Node Structure Every node inherits from `BaseNode` and defines three key components: @@ -28,6 +30,32 @@ class MyNode(BaseNode): pass ``` + +### Node Lifecycle Architecture + +```mermaid +stateDiagram-v2 + [*] --> Registered + Registered --> Ready + Ready --> Executing + Executing --> Completed + Executing --> Failed + Failed --> Ready + Completed --> Ready + + state Executing { + [*] --> ValidateInputs + ValidateInputs --> ProcessData + ProcessData --> ValidateOutputs + ValidateOutputs --> [*] + } + + note right of Failed + Automatic retry with + exponential backoff + end note +``` + ### Inputs Define the data your node expects to receive: diff --git a/docs/docs/exosphere/state-manager-setup.md b/docs/docs/exosphere/state-manager-setup.md index 9965afb1..44be3c63 100644 --- a/docs/docs/exosphere/state-manager-setup.md +++ b/docs/docs/exosphere/state-manager-setup.md @@ -2,6 +2,8 @@ This guide provides step-by-step instructions for setting up the Exosphere state manager in both local and production environments. +> **📚 Getting Started**: For a complete local setup guide covering both the state manager and dashboard, see our [Local Setup Guide](./local-setup.md). + ## Overview The Exosphere state manager is the core backend service that handles workflow execution, state management, and coordination between nodes. It provides a REST API for managing graph templates, registered nodes, and workflow execution states. diff --git a/docs/docs/exosphere/trigger-graph.md b/docs/docs/exosphere/trigger-graph.md index 4ff2cb83..4444658f 100644 --- a/docs/docs/exosphere/trigger-graph.md +++ b/docs/docs/exosphere/trigger-graph.md @@ -2,6 +2,8 @@ Once you have created a graph template, you can trigger its execution by sending trigger states to the state manager. This guide shows you how to trigger graphs and monitor their execution. +> **📚 Getting Started**: For a complete local setup guide covering both the state manager and dashboard, see our [Local Setup Guide](./local-setup.md). + The recommended way to trigger graphs is using the Exosphere Python SDK, which provides a clean interface to the State Manager API. === "Exosphere Python SDK" diff --git a/docs/docs/getting-started.md b/docs/docs/getting-started.md index be1d5691..e441604e 100644 --- a/docs/docs/getting-started.md +++ b/docs/docs/getting-started.md @@ -159,14 +159,6 @@ If you're working in a Jupyter notebook or Python REPL, consider these alternati ).start() # Blocks and runs forever ``` -## Next Steps - -Now that you have the basics, explore: - -- **[Register Node](./exosphere/register-node.md)** - Understand how to create and register custom nodes -- **[Create Runtime](./exosphere/create-runtime.md)** - Learn how to set up and configure your runtime -- **[Create Graph](./exosphere/create-graph.md)** - Build workflows by connecting nodes together -- **[Trigger Graph](./exosphere/trigger-graph.md)** - Execute your workflows and monitor their progress ## Key Features @@ -178,16 +170,41 @@ Now that you have the basics, explore: - **Error Handling**: Built-in retry mechanisms and error recovery - **Scalability**: Designed for high-volume batch processing and workflows -## Architecture +## Next Steps -``` -+-------------------+ +--------------------+ +-------------------+ -| Your Nodes | | Runtime | | State Manager | -| |<-->| |<-->| | -| • Inputs | | • Registration | | • Orchestration | -| • Outputs | | • Execution | | • State Mgmt | -| • Secrets | | • Error Handling | | • Dashboard | -+-------------------+ +--------------------+ +-------------------+ +Now that you have the basics, explore: + +- **[Local Setup](./exosphere/local-setup.md)** - Set up Exosphere locally for development and testing +- **[Register Node](./exosphere/register-node.md)** - Understand how to create and register custom nodes +- **[Create Runtime](./exosphere/create-runtime.md)** - Learn how to set up and configure your runtime +- **[Create Graph](./exosphere/create-graph.md)** - Build workflows by connecting nodes together +- **[Trigger Graph](./exosphere/trigger-graph.md)** - Execute your workflows and monitor their progress + + +### Data Flow Architecture + +```mermaid +sequenceDiagram + participant Client as Client Application + participant Runtime as Runtime + participant Node as Node Executor + participant StateMgr as State Manager + participant MongoDB as State Store + + Client->>Runtime: Initialize with nodes + Runtime->>StateMgr: Register runtime & nodes + StateMgr->>MongoDB: Store registration info + + loop Workflow Execution + StateMgr->>Runtime: Trigger node execution + Runtime->>Node: Execute node logic + Node->>Runtime: Return outputs + Runtime->>StateMgr: Update execution state + StateMgr->>MongoDB: Persist state changes + StateMgr->>Runtime: Trigger next node (if any) + end + + StateMgr->>Client: Return final results ``` ## Data Model (v1) diff --git a/docs/docs/index.md b/docs/docs/index.md index 78b1162c..41db50c8 100644 --- a/docs/docs/index.md +++ b/docs/docs/index.md @@ -1,14 +1,46 @@ -# Exosphere Docs + +
-Exosphere is an open-source infrastructure layer to run distributed AI workflows and agents with Python based on a node-based architecture. +# Exosphere: Distributed AI Workflow Infrastructure ---- +**Exosphere** is an open-source, Kubernetes-native infrastructure platform designed to run distributed AI workflows and autonomous agents at scale. Built with Python and based on a flexible node-based architecture, Exosphere enables developers to create, deploy, and manage robust AI workflows that can handle large-scale data processing and long-running operations. + +## What Exosphere Can Do + +Exosphere provides a powerful foundation for building and orchestrating AI applications with these key capabilities: + +### **Scalable AI Workflows** +- **Infinite Parallel Agents**: Run multiple AI agents simultaneously across distributed infrastructure +- **Dynamic State Management**: Create and manage state at runtime with persistent storage +- **Fault Tolerance**: Built-in failure handling and recovery mechanisms for production reliability -**Documentation**: [https://docs.exosphere.host](https://docs.exosphere.host) +### **Developer Experience** +- **Plug-and-Play Nodes**: Create reusable, atomic workflow components that can be mixed and matched +- **Python-First**: Native Python support with Pydantic models for type-safe inputs/outputs -**Source Code**: [https://github.com/exospherehost/exospherehost](https://github.com/exospherehost/exospherehost) +### **Production Infrastructure** +- **Kubernetes Native**: Deploy seamlessly on Kubernetes clusters for enterprise-grade scalability +- **State Persistence**: Maintain workflow state across restarts and failures +- **Interactive Dashboard**: Visual workflow management, monitoring, and debugging tools + +### **AI Agent Capabilities** +- **Autonomous Execution**: Build agents that can make decisions and execute complex workflows +- **Data Processing**: Handle large datasets with distributed processing capabilities +- **API Integration**: Connect to external services and APIs through configurable nodes + +Whether you're building data pipelines, AI agents, or complex workflow orchestrations, Exosphere provides the infrastructure backbone to make your AI applications production-ready and scalable. + +--- **Watch the Step by Step Demo**: @@ -31,7 +63,7 @@ uv add exospherehost ## Example -### Create it +### Create Create a file `main.py` with: @@ -65,114 +97,7 @@ Runtime( !!! info "Interactive Environments" If you're using Jupyter notebooks or Python REPLs, `Runtime.start()` will block your session. See the [Getting Started guide](./getting-started.md#important-blocking-behavior) for non-blocking alternatives. -### Run it - -Run the server with: - -```bash -uv run main.py -``` - -### Check it - -Your runtime is now running and ready to process workflows! - -### Interactive Dashboard - -Now go to your Exosphere dashboard to: - -* View your registered nodes -* Create and manage graph templates -* Trigger workflows -* Monitor execution states -* Debug and troubleshoot - -Ref: [Dashboard Guide](./exosphere/dashboard.md) - -## Example graph - -Now modify the file `main.py` to add more complex processing: - -```python -from exospherehost import Runtime, BaseNode -from pydantic import BaseModel -import json - -class DataProcessorNode(BaseNode): - class Inputs(BaseModel): - data: str - operation: str - - class Outputs(BaseModel): - result: str - status: str - - class Secrets(BaseModel): - api_key: str - - async def execute(self) -> Outputs: - # Parse the input data - try: - data = json.loads(self.inputs.data) - except: - return self.Outputs( - result="", - status="error: invalid json" - ) - - # Process based on operation - if self.inputs.operation == "transform": - result = {"transformed": data, "processed": True} - else: - result = {"original": data, "processed": False} - - return self.Outputs( - result=json.dumps(result), - status="success" - ) - -# Initialize the runtime -Runtime( - namespace="MyProject", - name="DataProcessor", - nodes=[DataProcessorNode] -).start() -``` - -The runtime will automatically reload and register the updated node. - -### Create it - -Create a file `main.py` with: - -```python -from exospherehost import Runtime, BaseNode -from pydantic import BaseModel - -class HelloWorldNode(BaseNode): - class Inputs(BaseModel): - name: str - - class Outputs(BaseModel): - message: str - - class Secrets(BaseModel): - pass - - async def execute(self) -> Outputs: - return self.Outputs( - message=f"Hello, {self.inputs.name}!" - ) - -# Initialize the runtime -Runtime( - namespace="MyProject", - name="HelloWorld", - nodes=[HelloWorldNode] -).start() -``` - -### Run it +### Run Run the server with: @@ -180,21 +105,16 @@ Run the server with: uv run main.py ``` -### Check it +### Check Your runtime is now running and ready to process workflows! -### Interactive Dashboard -Now go to your Exosphere dashboard to: +### Local Development -* View your registered nodes -* Create and manage graph templates -* Trigger workflows -* Monitor execution states -* Debug and troubleshoot +Get the Exosphere State Manager and Dashboard ready to start building workflows locally. -Ref: [Dashboard Guide](./exosphere/dashboard.md) +Ref: [Local Setup](./exosphere/local-setup.md) ## Example graph diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 0ba81881..055e5a97 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -108,6 +108,7 @@ plugins: Introduction: - index.md - getting-started.md + - exosphere/local-setup.md - docker-compose-setup.md - exosphere/state-manager-setup.md - exosphere/register-node.md @@ -126,7 +127,11 @@ markdown_extensions: pygments_lang_class: true - pymdownx.inlinehilite - pymdownx.snippets - - pymdownx.superfences + - pymdownx.superfences: + custom_fences: + - name: mermaid + class: mermaid + format: !!python/name:pymdownx.superfences.fence_code_format - admonition - pymdownx.details - pymdownx.tabbed: @@ -138,6 +143,7 @@ extra_css: nav: - Introduction: index.md - Getting Started: getting-started.md + - Local Setup: exosphere/local-setup.md - Docker Compose: docker-compose-setup.md - State Manager Setup: exosphere/state-manager-setup.md - Register Node: exosphere/register-node.md diff --git a/docs/uv.lock b/docs/uv.lock index f6a8c8f3..54ebd16b 100644 --- a/docs/uv.lock +++ b/docs/uv.lock @@ -111,6 +111,7 @@ dependencies = [ { name = "mkdocs-git-revision-date-localized-plugin" }, { name = "mkdocs-llmstxt" }, { name = "mkdocs-material" }, + { name = "mkdocs-mermaid2-plugin" }, ] [package.metadata] @@ -119,6 +120,16 @@ requires-dist = [ { name = "mkdocs-git-revision-date-localized-plugin", specifier = ">=1.4.7" }, { name = "mkdocs-llmstxt", specifier = ">=0.2.0" }, { name = "mkdocs-material", specifier = ">=9.6.14" }, + { name = "mkdocs-mermaid2-plugin", specifier = ">=1.0.0" }, +] + +[[package]] +name = "editorconfig" +version = "0.17.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/88/3a/a61d9a1f319a186b05d14df17daea42fcddea63c213bcd61a929fb3a6796/editorconfig-0.17.1.tar.gz", hash = "sha256:23c08b00e8e08cc3adcddb825251c497478df1dada6aefeb01e626ad37303745", size = 14695, upload-time = "2025-06-09T08:21:37.097Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/96/fd/a40c621ff207f3ce8e484aa0fc8ba4eb6e3ecf52e15b42ba764b457a9550/editorconfig-0.17.1-py3-none-any.whl", hash = "sha256:1eda9c2c0db8c16dbd50111b710572a5e6de934e39772de1959d41f64fc17c82", size = 16360, upload-time = "2025-06-09T08:21:35.654Z" }, ] [[package]] @@ -178,6 +189,19 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/62/a1/3d680cbfd5f4b8f15abc1d571870c5fc3e594bb582bc3b64ea099db13e56/jinja2-3.1.6-py3-none-any.whl", hash = "sha256:85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67", size = 134899, upload-time = "2025-03-05T20:05:00.369Z" }, ] +[[package]] +name = "jsbeautifier" +version = "1.15.4" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "editorconfig" }, + { name = "six" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/ea/98/d6cadf4d5a1c03b2136837a435682418c29fdeb66be137128544cecc5b7a/jsbeautifier-1.15.4.tar.gz", hash = "sha256:5bb18d9efb9331d825735fbc5360ee8f1aac5e52780042803943aa7f854f7592", size = 75257, upload-time = "2025-02-27T17:53:53.252Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/2d/14/1c65fccf8413d5f5c6e8425f84675169654395098000d8bddc4e9d3390e1/jsbeautifier-1.15.4-py3-none-any.whl", hash = "sha256:72f65de312a3f10900d7685557f84cb61a9733c50dcc27271a39f5b0051bf528", size = 94707, upload-time = "2025-02-27T17:53:46.152Z" }, +] + [[package]] name = "markdown" version = "3.8.2" @@ -405,6 +429,23 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/5b/54/662a4743aa81d9582ee9339d4ffa3c8fd40a4965e033d77b9da9774d3960/mkdocs_material_extensions-1.3.1-py3-none-any.whl", hash = "sha256:adff8b62700b25cb77b53358dad940f3ef973dd6db797907c49e3c2ef3ab4e31", size = 8728, upload-time = "2023-11-22T19:09:43.465Z" }, ] +[[package]] +name = "mkdocs-mermaid2-plugin" +version = "1.2.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "beautifulsoup4" }, + { name = "jsbeautifier" }, + { name = "mkdocs" }, + { name = "pymdown-extensions" }, + { name = "requests" }, + { name = "setuptools" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/3c/d4/efbabe9d04252b3007bc79b0d6db2206b40b74e20619cbed23c1e1d03b2a/mkdocs_mermaid2_plugin-1.2.2.tar.gz", hash = "sha256:20a44440d32cf5fd1811b3e261662adb3e1b98be272e6f6fb9a476f3e28fd507", size = 16209, upload-time = "2025-08-27T23:51:51.078Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/8e/d5/15f6eeeb755e57a501fad6dcfb3fe406dea5f6a6347a77c3be114294f7bb/mkdocs_mermaid2_plugin-1.2.2-py3-none-any.whl", hash = "sha256:a003dddd6346ecc0ad530f48f577fe6f8b21ea23fbee09eabf0172bbc1f23df8", size = 17300, upload-time = "2025-08-27T23:51:49.988Z" }, +] + [[package]] name = "packaging" version = "25.0" @@ -537,6 +578,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/7c/e4/56027c4a6b4ae70ca9de302488c5ca95ad4a39e190093d6c1a8ace08341b/requests-2.32.4-py3-none-any.whl", hash = "sha256:27babd3cda2a6d50b30443204ee89830707d396671944c998b5975b031ac2b2c", size = 64847, upload-time = "2025-06-09T16:43:05.728Z" }, ] +[[package]] +name = "setuptools" +version = "80.9.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/18/5d/3bf57dcd21979b887f014ea83c24ae194cfcd12b9e0fda66b957c69d1fca/setuptools-80.9.0.tar.gz", hash = "sha256:f36b47402ecde768dbfafc46e8e4207b4360c654f1f3bb84475f0a28628fb19c", size = 1319958, upload-time = "2025-05-27T00:56:51.443Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a3/dc/17031897dae0efacfea57dfd3a82fdd2a2aeb58e0ff71b77b87e44edc772/setuptools-80.9.0-py3-none-any.whl", hash = "sha256:062d34222ad13e0cc312a4c02d73f059e86a4acbfbdea8f8f76b28c99f306922", size = 1201486, upload-time = "2025-05-27T00:56:49.664Z" }, +] + [[package]] name = "six" version = "1.17.0"