Include distributed docker-compose example

Without this change, there is not an example of how a user might proceed with
local testing and development using docker-compose.  Here we copy the local
docker-compose example and extend it to include the distributed configuration
pieces necessary to get a functional example.

CHANGELOG.md

@@ -22,6 +22,7 @@
 * [ENHANCEMENT] Add retry middleware in query-frontend. [#814](https://github.com/grafana/tempo/pull/814) (@kvrhdn)
 * [ENHANCEMENT] Add `-use-otel-tracer` to use the OpenTelemetry tracer, this will also capture traces emitted by the gcs sdk. Experimental: not all features are supported (i.e. remote sampling). [#842](https://github.com/grafana/tempo/pull/842) (@kvrhdn)
 * [ENHANCEMENT] Add `/services` endpoint. [#863](https://github.com/grafana/tempo/pull/863) (@kvrhdn)
+* [ENHANCEMENT] Include distributed docker-compose example [#859](https://github.com/grafana/tempo/pull/859) (@zalegrala)
 
 ## v1.0.1
 

example/docker-compose/distributed/docker-compose.yaml

@@ -0,0 +1,102 @@
+version: "3"
+services:
+
+  distributor:
+    image: tempo:latest
+    command: "-target=distributor -config.file=/etc/tempo.yaml"
+    volumes:
+      - ./tempo-distributed.yaml:/etc/tempo.yaml
+    ports:
+      - "3200"   # tempo
+
+  ingester-0:
+    image: tempo:latest
+    command: "-target=ingester -config.file=/etc/tempo.yaml"
+    restart: always
+    volumes:
+      - ./tempo-distributed.yaml:/etc/tempo.yaml
+    ports:
+      - "3200"   # tempo
+
+  ingester-1:
+    image: tempo:latest
+    command: "-target=ingester -config.file=/etc/tempo.yaml"
+    restart: always
+    volumes:
+      - ./tempo-distributed.yaml:/etc/tempo.yaml
+    ports:
+      - "3200"   # tempo
+
+  ingester-2:
+    image: tempo:latest
+    command: "-target=ingester -config.file=/etc/tempo.yaml"
+    restart: always
+    volumes:
+      - ./tempo-distributed.yaml:/etc/tempo.yaml
+    ports:
+      - "3200"   # tempo
+
+  query-frontend:
+    image: tempo:latest
+    command: "-target=query-frontend -config.file=/etc/tempo.yaml"
+    volumes:
+      - ./tempo-distributed.yaml:/etc/tempo.yaml
+    ports:
+      - "3200"   # tempo
+
+  querier:
+    image: tempo:latest
+    command: "-target=querier -config.file=/etc/tempo.yaml"
+    restart: always
+    volumes:
+      - ./tempo-distributed.yaml:/etc/tempo.yaml
+    ports:
+      - "3200"   # tempo
+
+  compactor:
+    image: tempo:latest
+    command: "-target=compactor -config.file=/etc/tempo.yaml"
+    restart: always
+    volumes:
+      - ./tempo-distributed.yaml:/etc/tempo.yaml
+    ports:
+      - "3200"   # tempo
+
+  minio:
+    image: minio/minio:latest
+    environment:
+      - MINIO_ACCESS_KEY=tempo
+      - MINIO_SECRET_KEY=supersecret
+    ports:
+      - "9001:9001"
+    entrypoint:
+      - sh
+      - -euc
+      - mkdir -p /data/tempo && /usr/bin/minio server /data --console-address ':9001'
+
+  synthetic-load-generator:
+    image: omnition/synthetic-load-generator:1.0.25
+    volumes:
+      - ../shared/load-generator.json:/etc/load-generator.json
+    environment:
+      - TOPOLOGY_FILE=/etc/load-generator.json
+      - JAEGER_COLLECTOR_URL=http://distributor:14268
+
+  prometheus:
+    image: prom/prometheus:latest
+    command: [ "--config.file=/etc/prometheus.yaml" ]
+    volumes:
+      - ./prometheus.yaml:/etc/prometheus.yaml
+    ports:
+      - "9090:9090"
+
+  grafana:
+    image: grafana/grafana:8.1.1
+    volumes:
+      - ./grafana-datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml
+    environment:
+      - GF_AUTH_ANONYMOUS_ENABLED=true
+      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
+      - GF_AUTH_DISABLE_LOGIN_FORM=true
+    ports:
+      - "3000:3000"

example/docker-compose/distributed/grafana-datasources.yaml

@@ -0,0 +1,23 @@
+apiVersion: 1
+
+datasources:
+- name: Prometheus
+  type: prometheus
+  access: proxy
+  orgId: 1
+  url: http://prometheus:9090
+  basicAuth: false
+  isDefault: false
+  version: 1
+  editable: false
+- name: Tempo
+  type: tempo
+  access: proxy
+  orgId: 1
+  url: http://query-frontend:3200
+  basicAuth: false
+  isDefault: true
+  version: 1
+  editable: false
+  apiVersion: 1
+  uid: tempo

example/docker-compose/distributed/prometheus.yaml

@@ -0,0 +1,17 @@
+global:
+  scrape_interval:     15s
+  evaluation_interval: 15s
+
+scrape_configs:
+  - job_name: 'prometheus'
+    static_configs:
+      - targets: [ 'localhost:9090' ]
+  - job_name: 'tempo'
+    static_configs:
+      - targets:
+        - 'distributor:3200'
+        - 'ingester-0:3200'
+        - 'ingester-1:3200'
+        - 'ingester-2:3200'
+        - 'query-frontend:3200'
+        - 'querier:3200'

example/docker-compose/distributed/readme.md

@@ -0,0 +1,66 @@
+## Local Storage
+
+In this example all data is stored locally in the `tempo-data*` directories for each of the ingester instances. Local storage is fine for experimenting with Tempo
+or when using the single binary, and each instance mounts its own data directory to keep the cluster resources separated.
+
+1. First start up the local stack.
+
+```console
+docker-compose up -d
+```
+
+At this point, the following containers should be spun up -
+
+```console
+docker-compose ps
+```
+```
+                 Name                               Command               State                         Ports                       
+------------------------------------------------------------------------------------------------------------------------------------
+distributed_compactor_1                  /tempo -target=compactor - ...   Up      0.0.0.0:49662->3200/tcp,:::49652->3200/tcp        
+distributed_distributor_1                /tempo -target=distributor ...   Up      0.0.0.0:49659->3200/tcp,:::49649->3200/tcp        
+distributed_grafana_1                    /run.sh                          Up      0.0.0.0:3000->3000/tcp,:::3000->3000/tcp          
+distributed_ingester-0_1                 /tempo -target=ingester -c ...   Up      0.0.0.0:49663->3200/tcp,:::49653->3200/tcp        
+distributed_ingester-1_1                 /tempo -target=ingester -c ...   Up      0.0.0.0:49660->3200/tcp,:::49650->3200/tcp        
+distributed_ingester-2_1                 /tempo -target=ingester -c ...   Up      0.0.0.0:49665->3200/tcp,:::49655->3200/tcp        
+distributed_minio_1                      sh -euc mkdir -p /data/tem ...   Up      9000/tcp, 0.0.0.0:9001->9001/tcp,:::9001->9001/tcp
+distributed_prometheus_1                 /bin/prometheus --config.f ...   Up      0.0.0.0:9090->9090/tcp,:::9090->9090/tcp          
+distributed_querier_1                    /tempo -target=querier -co ...   Up      0.0.0.0:49664->3200/tcp,:::49654->3200/tcp        
+distributed_query-frontend_1             /tempo -target=query-front ...   Up      0.0.0.0:49661->3200/tcp,:::49651->3200/tcp        
+distributed_synthetic-load-generator_1   ./start.sh                       Up                                                        
+```
+
+2. If you're interested you can see the wal/blocks as they are being created.
+
+```console
+ls tempo-data*/
+```
+
+3. The synthetic-load-generator is now printing out trace ids it's flushing into Tempo.  To view its logs use -
+
+```console
+docker-compose logs -f synthetic-load-generator
+```
+```
+synthetic-load-generator_1  | 20/10/24 08:27:09 INFO ScheduledTraceGenerator: Emitted traceId 57aedb829f352625 for service frontend route /product
+synthetic-load-generator_1  | 20/10/24 08:27:09 INFO ScheduledTraceGenerator: Emitted traceId 25fa96b9da24b23f for service frontend route /cart
+synthetic-load-generator_1  | 20/10/24 08:27:09 INFO ScheduledTraceGenerator: Emitted traceId 15b3ad814b77b779 for service frontend route /shipping
+synthetic-load-generator_1  | 20/10/24 08:27:09 INFO ScheduledTraceGenerator: Emitted traceId 3803db7d7d848a1a for service frontend route /checkout
+```
+
+Logs are in the form
+
+```
+Emitted traceId <traceid> for service frontend route /cart
+```
+
+Copy one of these trace ids.
+
+4. Navigate to [Grafana](http://localhost:3000/explore) and paste the trace id to request it from Tempo.
+Also notice that you can query Tempo metrics from the Prometheus data source setup in Grafana.
+
+5. To stop the setup use -
+
+```console
+docker-compose down -v
+```

example/docker-compose/distributed/tempo-distributed.yaml

@@ -0,0 +1,63 @@
+server:
+  http_listen_port: 3200
+
+distributor:
+  receivers:                           # this configuration will listen on all ports and protocols that tempo is capable of.
+    jaeger:                            # the receives all come from the OpenTelemetry collector.  more configuration information can
+      protocols:                       # be found there: https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver
+        thrift_http:                   #
+        grpc:                          # for a production deployment you should only enable the receivers you need!
+        thrift_binary:
+        thrift_compact:
+    zipkin:
+    otlp:
+      protocols:
+        http:
+        grpc:
+    opencensus:
+
+ingester:
+  trace_idle_period: 10s               # the length of time after a trace has not received spans to consider it complete and flush it
+  max_block_bytes: 1_000_000           # cut the head block when it hits this size or ...
+  max_block_duration: 5m               #   this much time passes
+  lifecycler:
+    ring:
+      replication_factor: 3
+
+memberlist:
+  abort_if_cluster_join_fails: false
+  bind_port: 7946
+  join_members:
+  - ingester-0:7946
+  - ingester-1:7946
+  - ingester-2:7946
+
+compactor:
+  compaction:
+    block_retention: 1h
+    compacted_block_retention: 10m
+
+query_frontend:
+    query_shards: 2
+
+querier:
+  frontend_worker:
+    frontend_address: query-frontend:9095
+
+storage:
+  trace:
+    backend: s3
+    s3:
+      bucket: tempo
+      endpoint: minio:9000
+      access_key: tempo
+      secret_key: supersecret
+      insecure: true
+    wal:
+      path: /tmp/tempo/wal             # where to store the the wal locally
+      encoding: none                   # wal encoding/compression.  options: none, gzip, lz4-64k, lz4-256k, lz4-1M, lz4, snappy, zstd
+    local:
+      path: /tmp/tempo/blocks
+    pool:
+      max_workers: 100                 # worker pool determines the number of parallel requests to the object store backend
+      queue_depth: 10000