Epic 3.6: Production Deployment Architecture #194
Description
Epic 3.6: Production Deployment Architecture
Status: ✅ COMPLETE (2025-12-27 Security Hardened)
Multi-region production architecture deployed, operational, and security hardened.
Overview
Design and implement minimal scalable global deployment architecture for M3W with multi-layer fallback for high availability.
Core Targets
| # | Target | Issue | Status |
|---|---|---|---|
| 1 | Research minimal scalable global deployment | #195 | ✅ Done |
| 2 | Deploy first Scale Unit (Japan) | #197 | ✅ Done |
| 3 | Deploy second Scale Unit (SEA) | #238 | ✅ Done |
| 4 | Gateway Architecture | #204 | ✅ Done |
| 5 | Multi-Gateway Architecture | #239 | ✅ Done |
Final Architecture (2025-12-27)
┌─────────────────────────────────────────────────┐
│ Cloudflare │
│ ┌─────────────────────────────────────────┐ │
│ │ CF Pages (m3w.example.com) │ │
│ │ - Frontend static files │ │
│ │ - CF Worker handles /api/* routing │ │
│ │ - Multi-gateway failover support │ │
│ └──────────────────┬──────────────────────┘ │
└─────────────────────┼──────────────────────────┘
│
┌────────────────┴────────────────┐
│ │
▼ ▼
┌─────────────────────────────┐ ┌─────────────────────────────┐
│ Gateway1 VM (JP) │ │ Gateway2 VM (HK) │
│ ┌───────────────────────┐ │ │ ┌───────────────────────┐ │
│ │ Docker Compose │ │ │ │ Docker Compose │ │
│ │ ├── OpenResty │ │ │ │ └── OpenResty │ │
│ │ └── Redis (primary) │ │ │ └───────────────────────┘ │
│ ├───────────────────────┤ │ │ ┌───────────────────────┐ │
│ │ WireGuard (10.10.0.1) │ │ │ │ WireGuard (10.10.0.5) │ │
│ └───────────────────────┘ │ │ └───────────────────────┘ │
└─────────────┬───────────────┘ └─────────────┬───────────────┘
│ │
└─────────────┬───────────────────┘
│
WireGuard VPN Mesh (10.10.0.0/24)
│
┌─────────────┴───────────────┐
│ │
┌─────────────▼───────────────┐ ┌────────▼────────────────────┐
│ JP K3s │ │ SEA K3s │
│ VPN: 10.10.0.2 │ │ VPN: 10.10.0.3 │
│ ┌───────────────────────┐ │ │ ┌───────────────────────┐ │
│ │ K8s Gateway (30400) │ │ │ │ K8s Gateway (30400) │ │
│ │ Backend Pod │ │ │ │ Backend Pod │ │
│ │ PostgreSQL StatefulSet│ │ │ │ PostgreSQL StatefulSet│ │
│ │ MinIO StatefulSet │ │ │ │ MinIO StatefulSet │ │
│ └───────────────────────┘ │ │ └───────────────────────┘ │
└─────────────────────────────┘ └─────────────────────────────┘
WireGuard VPN Mesh (Full Connectivity)
| Node | VPN IP | Peers |
|---|---|---|
| Gateway1 (JP) | 10.10.0.1 | gateway2, jp, sea |
| Gateway2 (HK) | 10.10.0.5 | gateway1, jp, sea |
| JP K3s | 10.10.0.2 | gateway1, gateway2 |
| SEA K3s | 10.10.0.3 | gateway1, gateway2 |
Request Flow
Browser → CF Pages → CF Worker (/api/*) → Gateway (OpenResty)
↓
WireGuard VPN
↓
K3s NodePort (30400)
↓
Backend Pod
Security Architecture (2025-12-27) 🔒
Network Security Groups
Gateway NSG - 允许端口：
| Port | Protocol | Purpose |
|---|---|---|
| 22 | TCP | SSH 管理访问 |
| 80 | TCP | HTTP (重定向到 HTTPS) |
| 443 | TCP | HTTPS 公网入口 |
| 51820 | UDP | WireGuard VPN |
K3s Scale Unit NSG - 允许端口：
| Port | Protocol | Purpose |
|---|---|---|
| 22 | TCP | SSH 管理访问 |
| 6443 | TCP | K3s API (kubectl) |
| 51820 | UDP | WireGuard VPN |
所有 Web 流量必须通过 Gateway → WireGuard VPN → K3s NodePort (30400)
SSL 证书策略
- 使用 Let's Encrypt 通配符证书
*.m3w.test3207.fun - 通过 Cloudflare DNS-01 challenge 验证
- 证书保存在 Gateway Docker volume
gateway_letsencrypt - 同一证书服务所有 K3s 子域名 (jp.m3w., sea.m3w., etc.)
DNS 路由
*.m3w.test3207.fun → Gateway Public IP
↓
Gateway OpenResty
↓ (WireGuard VPN)
K3s NodePort :30400
所有 K3s 域名都指向 Gateway，DNS 不直接暴露 K3s 公网 IP。
Key Architecture Decisions
| Decision | Choice | Reason |
|---|---|---|
| Frontend Hosting | CF Pages | Free CDN, auto-deploy from GitHub releases |
| API Routing | CF Worker | Config injection, multi-gateway failover |
| Gateway Runtime | Docker Compose on VM | Lightweight, Let's Encrypt SSL |
| VPN | WireGuard | Secure internal traffic, full mesh topology |
| K8s Distribution | k3s | Lightweight, single-node sufficient |
| Upload Limit | 100MB | Configured at all layers |
| Redis | Gateway1 VM only | Cross-region routing, all nodes connect via VPN |
| IaC | Terraform + Ansible | nodes.json as single source, one-click deploy |
| SSL | Wildcard cert | Single cert for all subdomains |
| K3s Public Ports | 80/443 CLOSED | Security: all traffic via Gateway |
Completed Sub-Issues
Phase 1: Infrastructure ✅
- Architecture design and resource planning #195 Architecture design and resource planning
- Define Scale Unit specification #196 Scale Unit k8s manifests
- Deploy first production Scale Unit #197 Deploy JP Scale Unit
Phase 2: CF Worker ✅
- CF Worker for API routing + config injection
- Upload timeout (60s for /api/upload/*)
Phase 3: Gateway Architecture ✅
- Independent Gateway K3s + Wireguard VPN #212 Gateway VM + Docker Compose + WireGuard
- Gateway K8s Manifests (OpenResty + Redis) #233 Gateway Config (OpenResty + Redis)
- 区域后端 VPN 接入 + 关闭公网 #232 K3s VPN connection + REDIS_URL
- Backend should return relative paths for media URLs (multi-gateway support) #237 Backend returns relative paths
Phase 4: Multi-Region ✅
- refactor: remove coverUrl, use songId to build URLs #238 Deploy SEA K3s Scale Unit
- docs: add release workflow guide to development-standards.instructions.md #239 Deploy Gateway2 (HK) + Multi-gateway mesh
Phase 5: Production Ready ✅
- SSL certificates (Let's Encrypt via Gateway)
- 100MB upload support (all layers configured)
- Frontend auto-deploy (CF Pages + GitHub workflow)
- Backend auto-update (GitHub workflow)
- Redis cross-region routing verified
- Multi-gateway failover ready
Phase 6: Automation ✅
- CI/CD pipeline for production deployment #198 CI/CD pipeline (GitHub Actions)
- Multi-region one-click deployment automation #201 One-click deployment (Terraform + Ansible)
-
GeoDNS configuration (future) #199 GeoDNS(not needed - CF Worker handles routing)
Phase 7: Security Hardening ✅ (2025-12-27)
- Close K3s public HTTP/HTTPS ports (80/443)
- Route all K3s DNS through Gateway
- Wildcard SSL cert
*.m3w.test3207.fun - Document security architecture in DEPLOYMENT.md
IaC Structure
m3w-k8s/
├── terraform/
│ ├── nodes.json # Single source of truth for IPs
│ ├── cloudflare.tf # DNS records (all K3s → Gateway)
│ └── modules/
│ ├── gateway/main.tf # Gateway NSG: 22, 80, 443, 51820
│ └── scale-unit/main.tf # K3s NSG: 22, 6443, 51820 (NO 80/443)
├── ansible/
│ ├── inventory/hosts.yaml # Dynamic from terraform
│ └── playbooks/
│ ├── gateway.yaml # Gateway + wildcard SSL cert
│ ├── wireguard-k3s.yaml # K3s WireGuard
│ └── templates/
├── scripts/
│ └── deploy.ps1 # Unified deployment script
├── docs/
│ └── DEPLOYMENT.md # Full deployment + security docs
└── .wireguard/ # Auto-generated public keys
Version History
| Date | Version | Changes |
|---|---|---|
| 2025-12-27 | - | Security Hardening - Close K3s 80/443, wildcard SSL, DNS via Gateway |
| 2025-12-21 | - | Epic Complete - All sub-issues closed |
| 2025-12-21 | - | Multi-gateway architecture (Gateway2 HK + SEA K3s) |
| 2025-12-21 | - | WireGuard full mesh (4 nodes interconnected) |
| 2025-12-20 | - | Initial Gateway architecture deployed |
| 2025-12-19 | - | K3s Scale Unit operational |