Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Completed: Fix #61 #505

Open
wants to merge 12 commits into
base: master
Choose a base branch
from
14 changes: 14 additions & 0 deletions .web/docs/.vitepress/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,20 @@ export default defineConfig({
text: 'Rate Limiting',
link: '/guide/rate-limiting',
},
{
text: 'OpenTelemetry',
link: '/guide/otel/',
items: [
{
text: 'Grafana Cloud',
link: '/guide/otel/grafana-cloud/',
},
{
text: 'Honeycomb',
link: '/guide/otel/honeycomb/',
},
],
},
],
},
{
Expand Down
251 changes: 251 additions & 0 deletions .web/docs/developers/telemetry/telemetry.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,251 @@
# Gate Telemetry

Gate supports OpenTelemetry for metrics and distributed tracing, allowing operators to monitor their deployment's health, performance, and behavior.

## Configuration

Enable telemetry in your `config.yml`:

```yaml
telemetry:
# Metrics configuration using OpenTelemetry
metrics:
enabled: true
endpoint: "0.0.0.0:8888" # Endpoint for /metrics
anonymousMetrics: true # Send anonymous usage metrics
exporter: prometheus # Supported: prometheus, otlp
prometheus:
path: "/metrics" # Path for Prometheus scraping

# Distributed tracing configuration
tracing:
enabled: false # Disabled by default
endpoint: "localhost:4317" # OTLP collector endpoint
sampler: "parentbased_always_on"
exporter: otlp # Supported: otlp, jaeger, stdout
```

## Setup Options

### Self-Hosted Stack

1. **Prometheus + Grafana + Loki + Tempo**

```yaml
version: '3'
services:
prometheus:
image: prom/prometheus
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
ports:
- "9090:9090"

grafana:
image: grafana/grafana
ports:
- "3000:3000"
environment:
- GF_AUTH_ANONYMOUS_ENABLED=true
- GF_AUTH_ANONYMOUS_ORG_ROLE=Admin

loki:
image: grafana/loki
ports:
- "3100:3100"

tempo:
image: grafana/tempo
ports:
- "14250:14250"
```

prometheus.yml:
```yaml
scrape_configs:
- job_name: 'gate'
static_configs:
- targets: ['localhost:8888']
```

2. **Jaeger All-in-One**

```bash
docker run -d --name jaeger \
-e COLLECTOR_OTLP_ENABLED=true \
-p 16686:16686 \
-p 4317:4317 \
jaegertracing/all-in-one:latest
```

### Managed Services

1. **Grafana Cloud**
- Create account at https://grafana.com/
- Get endpoints and credentials
- Configure Gate:
```yaml
telemetry:
metrics:
enabled: true
endpoint: "prometheus-us-central1.grafana.net:9090"
exporter: otlp
tracing:
enabled: true
endpoint: "tempo-us-central1.grafana.net:4317"
exporter: otlp
```

2. **Honeycomb**
- Sign up at https://www.honeycomb.io/
- Get API key
- Configure Gate:
```yaml
telemetry:
tracing:
enabled: true
endpoint: "api.honeycomb.io:443"
exporter: otlp
```

3. **New Relic**
- Create account at https://newrelic.com/
- Get license key
- Configure Gate:
```yaml
telemetry:
metrics:
enabled: true
endpoint: "otlp.nr-data.net:4317"
exporter: otlp
tracing:
enabled: true
endpoint: "otlp.nr-data.net:4317"
exporter: otlp
```

## Sample Grafana Dashboard

```json
{
"annotations": {
"list": []
},
"editable": true,
"panels": [
{
"title": "Connected Players",
"type": "graph",
"datasource": "Prometheus",
"targets": [
{
"expr": "gate_players_current",
"legendFormat": "Players"
}
]
},
{
"title": "Server Performance",
"type": "gauge",
"datasource": "Prometheus",
"targets": [
{
"expr": "gate_performance_tps",
"legendFormat": "TPS"
}
],
"options": {
"maxValue": 20,
"minValue": 0,
"thresholds": [
{ "value": 15, "color": "red" },
{ "value": 18, "color": "yellow" },
{ "value": 19.5, "color": "green" }
]
}
},
{
"title": "Player Sessions",
"type": "heatmap",
"datasource": "Prometheus",
"targets": [
{
"expr": "rate(gate_connection_duration_bucket[5m])",
"legendFormat": "{{le}}"
}
]
},
{
"title": "Command Executions",
"type": "timeseries",
"datasource": "Prometheus",
"targets": [
{
"expr": "sum(rate(gate_command_executions_total[5m])) by (command)",
"legendFormat": "{{command}}"
}
]
},
{
"title": "Server Load",
"type": "timeseries",
"datasource": "Prometheus",
"targets": [
{
"expr": "sum(rate(gate_server_connections_total[5m])) by (server)",
"legendFormat": "{{server}}"
}
]
}
],
"rows": [
{
"panels": [
{
"title": "Player Logs",
"type": "logs",
"datasource": "Loki",
"targets": [
{
"expr": "{job=\"gate\"} |= \"player\""
}
]
}
]
},
{
"panels": [
{
"title": "Traces",
"type": "traces",
"datasource": "Tempo",
"targets": [
{
"query": "service.name=\"gate\""
}
]
}
]
}
]
}
```

## Anonymous Metrics

When enabled, Gate sends the following anonymous data:
- Random installation ID
- Gate version
- Operating system and architecture
- Number of connected players (aggregate only)
- Performance metrics (TPS, latency histograms)
- Error rates and types
- Feature usage statistics

This data helps:
- Identify performance bottlenecks
- Prioritize features and fixes
- Understand deployment patterns
- Improve stability

No personal data or player information is collected.
78 changes: 78 additions & 0 deletions .web/docs/guide/otel/grafana-cloud/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# Grafana Cloud

[Grafana Cloud](https://grafana.com/products/cloud/) is a fully managed observability platform that supports OpenTelemetry. Follow these steps to set up Gate with Grafana Cloud:

1. **Create a Grafana Cloud Account**

- Sign up at [Grafana.com](https://grafana.com/auth/sign-up/create-user)
- Navigate to your organization
- Create an Access Policy with write permissions at [Access Policies](https://grafana.com/orgs/your-org/access-policies)
- Generate and save your API token

2. **Configure Stack**

Navigate to your Grafana Cloud Stack (e.g., grafana.com/orgs/your-org/stacks/xxxxx) and:

- Click "Send Traces" in the Tempo section to get your traces endpoint
- Click "Send Metrics" in the Prometheus section to get your metrics endpoint

![Stack](./stack.png)

3. **Prepare Your Authentication**

You'll need to encode your credentials in base64 format. Use one of these methods:

- Using the command line:

```bash
echo "YOUR_INSTANCE_ID:YOUR_API_TOKEN" | base64
```

- Or visit an online base64 encoder like [base64encode.org](https://www.base64encode.org/)

4. **Configure Gate**

Export the following environment variables before starting Gate:

```bash
# For traces (Tempo)
export OTEL_EXPORTER_OTLP_ENDPOINT="https://tempo-prod-XX-prod-XX-XXXXX.grafana.net/tempo"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic YOUR_BASE64_ENCODED_CREDENTIALS"

# For metrics (Prometheus)
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_METRICS_EXPORTER="otlp"
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT="https://prometheus-prod-XX-prod-XX-XXXXX.grafana.net/api/prom/push"
export OTEL_EXPORTER_OTLP_METRICS_HEADERS="Authorization=Basic YOUR_BASE64_ENCODED_CREDENTIALS"
```

::: tip
For production deployments, consider setting these environment variables in your system configuration or container orchestration platform rather than exporting them manually.
:::

5. **Start Gate**

Once the environment variables are set, start Gate normally. It will automatically begin sending telemetry data to Grafana Cloud.

```bash
gate
```

See [Install](/guide/install/) for more information on how to start Gate.

6. **View Your Data**

Log into your Grafana Cloud account and click on the "Launch" button for Grafana:

- Navigate to the Tempo service to view your traces
- Navigate to the Prometheus service to view your metrics

![Launch](./launch.png)

- Go to the "Explore" section and select "Tempo" to in the sources

![tempo-source](./tempo-source.png)

![Trace](./trace.png)

- Or select "Prometheus" to view your metrics
Binary file added .web/docs/guide/otel/grafana-cloud/launch.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added .web/docs/guide/otel/grafana-cloud/stack.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added .web/docs/guide/otel/grafana-cloud/trace.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading