[Feature Request]: Granular Configuration Export & Import (via UI & API)

[crivetimihai]

🧭 Epic

Title: Comprehensive Configuration Export & Import System
Goal: Enable complete backup, restore, and migration of MCP Gateway configurations including Tools, Gateways, Virtual Servers, Prompts, Resources, and Roots through both CLI and Admin UI interfaces.
Why now: Critical for production operations, disaster recovery, environment promotion, and configuration management workflows.

Depends on #185

🧭 Type of Feature

• New functionality

---

🙋‍♂️ User Story 1: Complete CLI Export

As a: Platform engineer
I want: a CLI sub-command mcpgateway export --out config-YYYYMMDD.json
So that: I can capture all configuration entities (Tools, Gateways, Servers, Prompts, Resources, Roots) with proper authentication encryption.

✅ Acceptance Criteria

Scenario: Complete configuration export
Given a running MCP Gateway with:
  - 15 tools with various auth types (basic, bearer, api_key)
  - 3 peer gateways with federation capabilities  
  - 5 virtual servers with composed tool sets
  - 12 prompts with input/output schemas
  - 8 resources with different content types
  - 6 roots with file and HTTP URIs
When I run `mcpgateway export --out file.json`
Then the file must contain "version":"2025-03-26"
And 15 objects under $.entities.tools with encrypted auth_value fields
And 3 objects under $.entities.gateways with capabilities and health_check configs
And 5 objects under $.entities.servers with tool_ids and endpoint configurations
And 12 objects under $.entities.prompts with template and schema definitions
And 8 objects under $.entities.resources with uri and content metadata
And 6 objects under $.entities.roots with uri and name fields
And all encrypted fields use AES-256-GCM with base64 encoding
And the export includes dependency relationships between entities

---

🙋‍♂️ User Story 2: Comprehensive CLI Import

As a: SRE restoring into a fresh cluster
I want: mcpgateway import file.json --dry-run --conflict-strategy=update
So that: I can safely restore all configuration with conflict resolution and validation.

✅ Acceptance Criteria

Scenario: Complete idempotent import
Given an empty target Gateway with identical AUTH_ENCRYPTION_SECRET
When I run `mcpgateway import file.json`
Then the import should process entities in dependency order:
  1. First: Tools, Gateways, and Roots (no dependencies)
  2. Second: Servers (depend on Tools)
  3. Third: Prompts and Resources
And for each entity type it should POST to create new items
And if a name already exists it should PATCH to update
And auth_value fields should be properly decrypted and re-encrypted
And the command must finish with exit code 0
And provide detailed summary: "Imported: 15 tools, 3 gateways, 5 servers, 12 prompts, 8 resources, 6 roots"

Scenario: Dry-run validation
Given a valid export file
When I run `mcpgateway import file.json --dry-run`
Then it should validate all schemas without making changes
And report potential conflicts: "Would update 3 existing tools, create 12 new tools"
And exit with code 0 if validation passes

---

🙋‍♂️ User Story 3: Admin UI Export Interface

As a: Platform administrator
I want: export functionality integrated into the Admin UI with granular selection
So that: I can visually select specific configurations and download exports through the web interface.

✅ Acceptance Criteria

Scenario: UI-based selective export
Given the Admin UI with populated Tools, Gateways, Servers, Prompts, Resources, and Roots tabs
When I navigate to each tab and select specific items via checkboxes
And click "Export Selected" button
Then a modal should show export preview with dependency inclusion options
And clicking "Download Export" should generate an export-YYYYMMDD-HHMMSS.json file
And the export should contain only selected items plus their dependencies
And provide download progress indication for large exports

Scenario: Full configuration export via UI
Given the Admin UI dashboard
When I click "Export All Configuration" button
Then it should show confirmation dialog with entity counts
And generate complete system export including all entity types
And display export summary: "Exported 49 total items across 6 categories"

---

📦 Complete Export File Schema

{
  "version": "2025-03-26",
  "exported_at": "2025-06-26T12:34:56Z",
  "exported_by": "admin",
  "source_gateway": "https://gateway.example.com:4444",
  "encryption_method": "AES-256-GCM",
  "entities": {
    "tools": [
      {
        "name": "get_weather",
        "url": "https://api.weather.com/v1/current",
        "integration_type": "REST",
        "request_type": "GET", 
        "description": "Get current weather data",
        "headers": {"X-API-Key": "weather-key"},
        "auth_type": "bearer",
        "auth_value": "<encrypted-base64>", // AES-GCM encrypted
        "input_schema": {
          "type": "object",
          "properties": {"location": {"type": "string"}},
          "required": ["location"]
        },
        "output_schema": {"type": "object"},
        "tags": ["weather", "api"],
        "rate_limit": 100,
        "timeout": 30,
        "is_active": true,
        "created_at": "2025-01-15T10:00:00Z",
        "updated_at": "2025-01-15T10:00:00Z"
      }
    ],
    "gateways": [
      {
        "name": "production-east",
        "url": "https://prod-east.example.com:4444",
        "description": "Production East Coast Gateway",
        "auth_type": "basic",
        "auth_value": "<encrypted-base64>",
        "capabilities": {
          "tools": {"list_changed": true},
          "prompts": {"list_changed": true},
          "resources": {"subscribe": true, "list_changed": true}
        },
        "health_check": {
          "url": "https://prod-east.example.com:4444/health",
          "interval": 30,
          "timeout": 10,
          "retries": 3
        },
        "is_active": true,
        "federation_enabled": true
      }
    ],
    "servers": [
      {
        "name": "ai-tools-server",
        "description": "Composed AI tools virtual server",
        "tool_ids": ["get_weather", "summarize_text", "translate"],
        "sse_endpoint": "/servers/ai-tools-server/sse",
        "websocket_endpoint": "/servers/ai-tools-server/ws",
        "jsonrpc_endpoint": "/servers/ai-tools-server/jsonrpc",
        "capabilities": {
          "tools": {"list_changed": true},
          "prompts": {"list_changed": true}
        },
        "is_active": true
      }
    ],
    "prompts": [
      {
        "name": "summarize_document",
        "template": "Please summarize the following document:\n\n{{ document }}\n\nProvide a {{ length }} summary.",
        "description": "Document summarization with configurable length",
        "input_schema": {
          "type": "object",
          "properties": {
            "document": {"type": "string", "description": "The document to summarize"},
            "length": {"type": "string", "enum": ["short", "medium", "long"], "default": "medium"}
          },
          "required": ["document"]
        },
        "tags": ["summarization", "nlp"],
        "is_active": true
      }
    ],
    "resources": [
      {
        "name": "company-docs",
        "uri": "file:///data/docs/",
        "description": "Company documentation repository",
        "mime_type": "application/octet-stream",
        "tags": ["documentation", "internal"],
        "is_active": true,
        "last_modified": "2025-01-15T10:00:00Z"
      }
    ],
    "roots": [
      {
        "uri": "file:///data/workspace",
        "name": "Workspace"
      },
      {
        "uri": "file:///tmp",
        "name": "temp"
      },
      {
        "uri": "http://example.com/api/files",
        "name": "Remote Files API"
      }
    ]
  },
  "metadata": {
    "entity_counts": {
      "tools": 15,
      "gateways": 3, 
      "servers": 5,
      "prompts": 12,
      "resources": 8,
      "roots": 3
    },
    "dependencies": {
      "servers_to_tools": {
        "ai-tools-server": ["get_weather", "summarize_text", "translate"]
      }
    },
    "export_options": {
      "include_inactive": false,
      "include_dependencies": true,
      "selected_types": ["tools", "gateways", "servers", "prompts", "resources", "roots"]
    }
  }
}

Every block uses the same field names returned by GET /tools, GET /gateways, GET /servers, GET /prompts, GET /resources, and GET /roots, so import can stream the objects right back with POST/PATCH requests.

---

🙋‍♂️ User Story 4: Admin UI Import Wizard

As a: Operations engineer
I want: drag-and-drop import functionality in the Admin UI with conflict resolution
So that: I can safely restore configurations with visual feedback and control.

✅ Acceptance Criteria

Scenario: Import wizard with preview
Given the Admin UI is open
When I drag an export file onto the import drop zone
Then the system should parse and validate the file
And display an import preview showing:
  - Entity counts by type
  - Potential conflicts (existing names)
  - Dependencies that will be created
And provide conflict resolution options:
  - Skip conflicting items
  - Update existing items
  - Rename on import
When I select resolution strategy and click "Import"
Then show progress bar with real-time status
And display final summary with success/failure counts

Scenario: Import validation errors
Given an export file with invalid schemas
When I attempt import via UI
Then show validation errors with specific field details
And highlight problematic entities in the preview
And prevent import until issues are resolved

---

🙋‍♂️ User Story 5: Security and Encryption

As a: Security officer
I want: all sensitive authentication data encrypted in exports with key rotation support
So that: exports can be safely stored and shared without exposing credentials.

✅ Acceptance Criteria

Scenario: Secure credential handling
Given tools with various auth types (basic, bearer, api_key)
When exported from source gateway
Then all auth_value fields are AES-256-GCM encrypted with AUTH_ENCRYPTION_SECRET
And exports never contain plaintext credentials
And the encryption includes authentication tags for integrity

Scenario: Cross-environment import with key rotation
Given an export from gateway A with encryption key "key-a"
And target gateway B with encryption key "key-b"
When I run `mcpgateway import file.json --rekey-secret key-b`
Then the system should decrypt with "key-a" and re-encrypt with "key-b"
And import should succeed with all credentials functional
And audit logs should record the key rotation operation

---

🙋‍♂️ User Story 6: Partial and Filtered Export/Import

As a: Development team lead
I want: granular export/import with filtering and selection options
So that: I can promote only specific configurations between environments.

✅ Acceptance Criteria

Scenario: Filtered CLI export
When I run `mcpgateway export --type tool --tags "production,api" --out prod-tools.json`
Then the export should contain only tools with specified tags
And include their dependencies (servers that reference them)
And exclude unrelated entities

Scenario: Selective import with dependencies
Given an export containing 10 tools and 3 servers
When I run `mcpgateway import file.json --include "tool:weather_api,server:ai-server"`
Then only specified entities and their dependencies should be imported
And unspecified entities should be skipped
And provide summary: "Imported 1 tool, 1 server (2 items skipped)"

---

🙋‍♂️ User Story 7: API-Based Export/Import

As a: CI/CD pipeline developer
I want: REST endpoints for export/import operations
So that: I can automate configuration management in deployment pipelines.

✅ Acceptance Criteria

Scenario: API export endpoint
When I call `GET /admin/export?format=json&types=tools,servers,roots&include_inactive=false`
Then receive 200 response with Content-Type: application/json
And Content-Disposition header for file download
And response body matches the export schema
And includes only requested entity types

Scenario: API import endpoint
When I POST to `/admin/import` with multipart form data containing export file
Then receive 202 response for async processing
And get import job ID for status tracking
When I call `GET /admin/import/status/{job_id}`
Then receive current import progress and results
And final status shows detailed success/failure counts

---

🔐 Enhanced Encryption Notes

• auth_value is produced by encode_auth() (AES-256-GCM derived from AUTH_ENCRYPTION_SECRET) and parsed by decode_auth() during import.
• Keep the passphrase identical across environments; otherwise re-encrypt before import.
• Non-secret fields remain plaintext for diff-friendliness.

---

📐 Comprehensive Design Architecture

flowchart TD
    subgraph "CLI Interface"
        ExportCLI["mcpgateway export<br/>--type, --tags, --filter"]
        ImportCLI["mcpgateway import<br/>--dry-run, --rekey-secret"]
    end
    
    subgraph "Admin UI"
        ExportUI["Export Modal<br/>- Selective checkboxes<br/>- Dependency resolution"]
        ImportUI["Import Wizard<br/>- Drag & drop<br/>- Conflict resolution"]
    end
    
    subgraph "REST API"
        ExportAPI["/admin/export<br/>GET with filters"]
        ImportAPI["/admin/import<br/>POST multipart"]
        StatusAPI["/admin/import/status/{id}<br/>GET progress"]
    end
    
    subgraph "Core Services"
        ExportService["Export Service<br/>- Entity collection<br/>- Dependency resolution<br/>- Encryption"]
        ImportService["Import Service<br/>- Validation<br/>- Conflict resolution<br/>- Batch processing"]
        CryptoService["Crypto Service<br/>- AES-256-GCM<br/>- Key rotation"]
    end
    
    subgraph "Storage"
        DB[(Database<br/>SQLAlchemy ORM)]
        FS[("Export Files<br/>JSON")]
    end
    
    ExportCLI --> ExportService
    ExportUI --> ExportAPI
    ExportAPI --> ExportService
    ExportService --> DB
    ExportService --> CryptoService
    ExportService --> FS
    
    ImportCLI --> ImportService
    ImportUI --> ImportAPI
    ImportAPI --> ImportService
    ImportService --> StatusAPI
    FS --> ImportService
    ImportService --> CryptoService
    ImportService --> DB

Loading

Component	Change	Detail
CLI Commands	NEW	mcpgateway export/import with comprehensive filtering
Export Service	NEW	Entity collection, dependency resolution, encryption
Import Service	NEW	Validation, conflict resolution, async processing
Crypto Service	ENHANCE	Key rotation, cross-environment re-encryption
Admin UI Export	NEW	Modal with entity selection, dependency visualization
Admin UI Import	NEW	Drag-drop wizard, conflict resolution, progress tracking
REST API	NEW	/admin/export and /admin/import endpoints
Database Layer	ENHANCE	Extend existing schemas for export metadata
Validation	NEW	JSON Schema validation for import files
Audit Logging	ENHANCE	Track export/import operations with user attribution

---

🔄 Alternatives Considered

Alternative	Pros	Cons
DB dump via pg_dump	Fast, proven	Couples to SQL dialect, exposes secrets in plaintext
Helm chart values	GitOps-friendly	Loses runtime edits, can’t carry encrypted auth
Separate files per resource type	Fine-grained git history	Harder to ensure referential integrity

---

📓 Additional Context

• Encryption key configurable (AUTH_ENCRYPTION_SECRET) in .env or Helm values.
• The same JSON can be pushed to multiple environments for canary testing.
• Import respects naming uniqueness to avoid collisions.

---

🚀 Implementation Priority & Dependencies

Phase 1: Core Export/Import Foundation

• Priority: P0 (Critical for MVP)
• Export Service with basic entity collection
• Import Service with validation and upsert logic
• CLI commands: mcpgateway export/import
• Basic encryption/decryption with existing encode_auth/decode_auth

Phase 2: Advanced Features

• Priority: P1 (High)
• Filtering and selective export (--type, --tags, --filter)
• Conflict resolution strategies in import
• Dry-run functionality
• Dependency resolution and inclusion

Phase 3: UI Integration

• Priority: P1 (High)
• Admin UI export modal with entity selection
• Import wizard with drag-drop and preview
• Progress tracking and real-time feedback

Phase 4: API & Automation

• Priority: P2 (Medium)
• REST API endpoints for CI/CD integration
• Async import processing with job tracking
• Cross-environment key rotation support

---

Comprehensive Usage Examples

# === CLI Export Examples ===

# Complete system backup
mcpgateway export --out backup-$(date +%F).json

# Export only production tools
mcpgateway export --type tool --tags "production" --out prod-tools.json

# Export specific entities by name  
mcpgateway export --include "tool:weather_api,server:ai-server,root:workspace" --out subset.json

# Export only roots and resources
mcpgateway export --type root,resource --out filesystem-config.json

# Export with inactive entities included
mcpgateway export --include-inactive --out complete-backup.json

# === CLI Import Examples ===

# Dry run to preview changes
mcpgateway import backup.json --dry-run

# Import with conflict resolution
mcpgateway import backup.json --conflict-strategy update

# Cross-environment import with key rotation
mcpgateway import prod-export.json --rekey-secret $NEW_ENV_SECRET

# Selective import of specific entities
mcpgateway import backup.json --include "tool:*,server:ai-*,root:workspace"

# === API Examples ===

# Export via REST API
curl -H "Authorization: Bearer $TOKEN" \
     "https://gateway.example.com/admin/export?types=tools,servers,roots&format=json" \
     -o export.json

# Import via REST API  
curl -H "Authorization: Bearer $TOKEN" \
     -F file=@export.json \
     "https://gateway.example.com/admin/import"

# Check import status
curl -H "Authorization: Bearer $TOKEN" \
     "https://gateway.example.com/admin/import/status/job-123"

This comprehensive export/import system provides the foundation for robust configuration management, disaster recovery, and seamless environment promotion workflows in production MCP Gateway deployments.