Add circuit breaker user guide

Author: Anya

docs/application-layer/circuit-breaker-guide.md

@@ -0,0 +1,181 @@
+---
+sidebar_position: 6
+title: Circuit Breaker User Guide
+---
+
+# Circuit Breaker User Guide
+
+## What is a Circuit Breaker?
+
+Circuit breaking is a critical pattern for building resilient microservice applications. It helps prevent cascading failures by limiting the impact of service failures, latency spikes, and other network issues.
+
+In Kmesh, circuit breakers:
+- Limit concurrent connections and requests to a service
+- Automatically detect failing services
+- Prevent overload by rejecting excess traffic
+- Allow controlled recovery after failures
+
+## When to Use Circuit Breakers
+
+Consider implementing circuit breakers when:
+- You need to protect services from being overwhelmed
+- You want to improve system resilience during partial outages
+- You need to control retry behavior and prevent cascading failures
+- You want to implement graceful degradation
+
+## Circuit Breaker Configuration
+
+Kmesh circuit breakers can be configured using DestinationRules:
+
+```yaml
+apiVersion: networking.istio.io/v1alpha3
+kind: DestinationRule
+metadata:
+  name: my-service-circuit-breaker
+spec:
+  host: my-service
+  trafficPolicy:
+    connectionPool:
+      tcp:
+        maxConnections: 100           # Maximum number of TCP connections
+        connectTimeout: 30ms          # TCP connection timeout
+      http:
+        http1MaxPendingRequests: 1    # Maximum number of pending HTTP requests
+        maxRequestsPerConnection: 1    # Maximum number of requests per connection
+        maxRetries: 3                 # Maximum number of retries
+    outlierDetection:
+      consecutive5xxErrors: 5         # Number of 5xx errors before ejection
+      interval: 10s                   # Interval for checking errors
+      baseEjectionTime: 30s           # Minimum ejection duration
+      maxEjectionPercent: 100         # Maximum percentage of instances to eject
+```
+
+## Key Configuration Parameters
+
+### Connection Pool Settings
+
+| Parameter | Description | Default |
+|-----------|-------------|---------|
+| `maxConnections` | Maximum number of TCP connections | 1024 |
+| `connectTimeout` | TCP connection timeout | 30ms |
+| `http1MaxPendingRequests` | Maximum number of pending HTTP requests | 1024 |
+| `maxRequestsPerConnection` | Maximum number of requests per connection | No limit |
+| `maxRetries` | Maximum number of retries | 3 |
+
+### Outlier Detection Settings
+
+| Parameter | Description | Default |
+|-----------|-------------|---------|
+| `consecutive5xxErrors` | Number of 5xx errors before ejection | 5 |
+| `interval` | Interval for checking errors | 10s |
+| `baseEjectionTime` | Minimum ejection duration | 30s |
+| `maxEjectionPercent` | Maximum percentage of instances to eject | 10% |
+
+## Step-by-Step Implementation Guide
+
+### 1. Identify Services Needing Protection
+
+First, identify which services in your mesh need circuit breaker protection. Good candidates include:
+- Services with limited capacity
+- Critical services that must remain available
+- Services with known stability issues
+
+### 2. Create a Circuit Breaker Configuration
+
+Create a YAML file with your circuit breaker configuration:
+
+```yaml
+# circuit-breaker.yaml
+apiVersion: networking.istio.io/v1alpha3
+kind: DestinationRule
+metadata:
+  name: my-service-circuit-breaker
+spec:
+  host: my-service
+  trafficPolicy:
+    connectionPool:
+      tcp:
+        maxConnections: 100
+      http:
+        http1MaxPendingRequests: 10
+        maxRequestsPerConnection: 10
+    outlierDetection:
+      consecutive5xxErrors: 5
+      interval: 10s
+      baseEjectionTime: 30s
+```
+
+### 3. Apply the Configuration
+
+Apply the configuration to your cluster:
+
+```bash
+kubectl apply -f circuit-breaker.yaml
+```
+
+### 4. Verify the Configuration
+
+Check that your circuit breaker has been correctly applied:
+
+```bash
+kubectl get destinationrule my-service-circuit-breaker -o yaml
+```
+
+### 5. Test the Circuit Breaker
+
+You can test your circuit breaker using a load testing tool like Fortio:
+
+```bash
+# Install Fortio if needed
+kubectl apply -f https://raw.githubusercontent.com/kmesh-net/kmesh/main/samples/fortio.yaml
+
+# Run a load test
+kubectl exec -it $(kubectl get pod -l app=fortio -o jsonpath={.items[0].metadata.name}) \
+  -c fortio -- fortio load -c 20 -qps 0 -n 50 http://my-service:8000/
+```
+
+When the circuit breaker trips, you'll see HTTP 503 errors in the Fortio output.
+
+## Monitoring Circuit Breakers
+
+### Using Prometheus and Grafana
+
+If you have Prometheus and Grafana set up with Kmesh, you can monitor circuit breaker metrics:
+
+1. Open Grafana dashboard
+2. Look for metrics with the following patterns:
+   - `istio_requests_total` (filtered by response code 503)
+   - `istio_request_duration_milliseconds` (to monitor latency)
+
+### Using kubectl
+
+You can also check circuit breaker status using kubectl:
+
+```bash
+# Get circuit breaker metrics from Istio proxy
+kubectl exec $(kubectl get pod -l app=my-client -o jsonpath={.items[0].metadata.name}) \
+  -c istio-proxy -- pilot-agent request GET stats | grep circuit_breakers
+```
+
+## Best Practices
+
+1. **Start Conservative**: Begin with higher limits and gradually tighten them
+2. **Test Thoroughly**: Test your circuit breakers under load before deploying to production
+3. **Monitor Behavior**: Keep track of circuit breaker activations to fine-tune settings
+4. **Layer Protection**: Use circuit breakers alongside retries, timeouts, and fallbacks
+5. **Document Settings**: Keep a record of circuit breaker settings for each service
+
+## Troubleshooting
+
+| Problem | Possible Cause | Solution |
+|---------|---------------|----------|
+| Too many 503 errors | Circuit breaker thresholds too low | Increase connection pool limits |
+| Circuit breaker not triggering | Thresholds too high | Lower connection pool limits |
+| Intermittent failures | Outlier detection not configured properly | Adjust consecutive error settings |
+| Slow recovery | Base ejection time too long | Reduce baseEjectionTime |
+
+## Further Reading
+
+- [Detailed Circuit Breaker Documentation](/docs/transpot-layer/circuit-breaker.md)
+- [Kmesh Performance Monitoring](/docs/performance/monitoring.md)
+- [Service Metrics with Grafana](/docs/transpot-layer/service-metrics.md)