Include distributed docker-compose example

Without this change, there is not an example of how a user might proceed with
local testing and development using docker-compose.  Here we copy the local
docker-compose example and extend it to include the distributed configuration
pieces necessary to get a functional example.

CHANGELOG.md

@@ -1,5 +1,6 @@
 ## main / unreleased
 
+* [ENHANCEMENT] Include distributed docker-compose example [#859](https://github.com/grafana/tempo/pull/859) (@zalegrala)
 * [BUGFIX] Allow only valid trace ID characters when decoding [#854](https://github.com/grafana/tempo/pull/854) (@zalegrala)
 * [CHANGE] Upgrade Cortex from v1.9.0 to v1.9.0-131-ga4bf10354 [#841](https://github.com/grafana/tempo/pull/841) (@aknuds1)
 * [CHANGE] Change default tempo port from 3100 to 3200 [#770](https://github.com/grafana/tempo/pull/809) (@MurzNN)

example/docker-compose/local-distributed/docker-compose.yaml

@@ -0,0 +1,107 @@
+version: "3"
+services:
+
+  distributor:
+    image: tempo:latest
+    command: "-target=distributor -config.file=/etc/tempo.yaml"
+    volumes:
+      - ./tempo-distributed.yaml:/etc/tempo.yaml
+    ports:
+      - "14268"  # jaeger ingest
+      - "3200"   # tempo
+      - "55680"  # otlp grpc
+      - "55681"  # otlp http
+
+  ingester-0:
+    image: tempo:latest
+    command: "-target=ingester -config.file=/etc/tempo.yaml"
+    restart: always
+    volumes:
+      - ./tempo-distributed.yaml:/etc/tempo.yaml
+    ports:
+      - "3200"   # tempo
+      - "7946"
+
+  ingester-1:
+    image: tempo:latest
+    command: "-target=ingester -config.file=/etc/tempo.yaml"
+    restart: always
+    volumes:
+      - ./tempo-distributed.yaml:/etc/tempo.yaml
+    ports:
+      - "3200"   # tempo
+      - "7946"
+
+  ingester-2:
+    image: tempo:latest
+    command: "-target=ingester -config.file=/etc/tempo.yaml"
+    restart: always
+    volumes:
+      - ./tempo-distributed.yaml:/etc/tempo.yaml
+    ports:
+      - "3200"   # tempo
+      - "7946"
+
+  query-frontend:
+    image: tempo:latest
+    command: "-target=query-frontend -config.file=/etc/tempo.yaml"
+    volumes:
+      - ./tempo-distributed.yaml:/etc/tempo.yaml
+    ports:
+      - "3200"   # tempo
+
+  querier:
+    image: tempo:latest
+    command: "-target=querier -config.file=/etc/tempo.yaml"
+    volumes:
+      - ./tempo-distributed.yaml:/etc/tempo.yaml
+    ports:
+      - "3200"   # tempo
+
+  compactor:
+    image: tempo:latest
+    command: "-target=compactor -config.file=/etc/tempo.yaml"
+    volumes:
+      - ./tempo-distributed.yaml:/etc/tempo.yaml
+    ports:
+      - "3200"   # tempo
+
+
+  minio:
+    image: minio/minio:latest
+    environment:
+      - MINIO_ACCESS_KEY=tempo
+      - MINIO_SECRET_KEY=supersecret
+    ports:
+      - "9000:9000"
+    entrypoint:
+      - sh
+      - -euc
+      - mkdir -p /data/tempo && /usr/bin/minio server /data
+
+  synthetic-load-generator:
+    image: omnition/synthetic-load-generator:1.0.25
+    volumes:
+      - ../shared/load-generator.json:/etc/load-generator.json
+    environment:
+      - TOPOLOGY_FILE=/etc/load-generator.json
+      - JAEGER_COLLECTOR_URL=http://distributor:14268
+
+  prometheus:
+    image: prom/prometheus:latest
+    command: [ "--config.file=/etc/prometheus.yaml" ]
+    volumes:
+      - ./prometheus.yaml:/etc/prometheus.yaml
+    ports:
+      - "9090:9090"
+
+  grafana:
+    image: grafana/grafana:8.1.1
+    volumes:
+      - ./grafana-datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml
+    environment:
+      - GF_AUTH_ANONYMOUS_ENABLED=true
+      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
+      - GF_AUTH_DISABLE_LOGIN_FORM=true
+    ports:
+      - "3000:3000"

example/docker-compose/local-distributed/grafana-datasources.yaml

@@ -0,0 +1,23 @@
+apiVersion: 1
+
+datasources:
+- name: Prometheus
+  type: prometheus
+  access: proxy
+  orgId: 1
+  url: http://prometheus:9090
+  basicAuth: false
+  isDefault: false
+  version: 1
+  editable: false
+- name: Tempo
+  type: tempo
+  access: proxy
+  orgId: 1
+  url: http://query-frontend:3200
+  basicAuth: false
+  isDefault: true
+  version: 1
+  editable: false
+  apiVersion: 1
+  uid: tempo

example/docker-compose/local-distributed/prometheus.yaml

@@ -0,0 +1,17 @@
+global:
+  scrape_interval:     15s
+  evaluation_interval: 15s
+
+scrape_configs:
+  - job_name: 'prometheus'
+    static_configs:
+      - targets: [ 'localhost:9090' ]
+  - job_name: 'tempo'
+    static_configs:
+      - targets:
+        - 'distributor:3200'
+        - 'ingester-0:3200'
+        - 'ingester-1:3200'
+        - 'ingester-2:3200'
+        - 'query-frontend:3200'
+        - 'querier:3200'

example/docker-compose/local-distributed/readme.md

@@ -0,0 +1,67 @@
+## Local Storage
+
+In this example all data is stored locally in the `tempo-data*` directories for each of the ingester instances. Local storage is fine for experimenting with Tempo
+or when using the single binary, and each instance mounts its own data directory to keep the cluster resources separated.
+
+1. First start up the local stack.
+
+```console
+docker-compose up -d
+```
+
+At this point, the following containers should be spun up -
+
+```console
+docker-compose ps
+```
+```
+                    Name                                  Command               State                                                                                                   Ports                                                                                                 
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+local-distributed_distributor_1                /tempo -target=distributor ...   Up      0.0.0.0:49583->14268/tcp,:::49583->14268/tcp, 0.0.0.0:49585->3200/tcp,:::49585->3200/tcp, 0.0.0.0:49582->55680/tcp,:::49582->55680/tcp, 0.0.0.0:49581->55681/tcp,:::49581->55681/tcp,                 
+                                                                                        0.0.0.0:49584->7946/tcp,:::49584->7946/tcp                                                                                                                                                            
+local-distributed_grafana_1                    /run.sh                          Up      0.0.0.0:3000->3000/tcp,:::3000->3000/tcp                                                                                                                                                              
+local-distributed_ingester-0_1                 /tempo -target=ingester -c ...   Up      0.0.0.0:49575->14268/tcp,:::49575->14268/tcp, 0.0.0.0:49577->3200/tcp,:::49577->3200/tcp, 0.0.0.0:49576->7946/tcp,:::49576->7946/tcp                                                                  
+local-distributed_ingester-1_1                 /tempo -target=ingester -c ...   Up      0.0.0.0:49586->14268/tcp,:::49586->14268/tcp, 0.0.0.0:49588->3200/tcp,:::49588->3200/tcp, 0.0.0.0:49587->7946/tcp,:::49587->7946/tcp                                                                  
+local-distributed_ingester-2_1                 /tempo -target=ingester -c ...   Up      0.0.0.0:49578->14268/tcp,:::49578->14268/tcp, 0.0.0.0:49580->3200/tcp,:::49580->3200/tcp, 0.0.0.0:49579->7946/tcp,:::49579->7946/tcp                                                                  
+local-distributed_prometheus_1                 /bin/prometheus --config.f ...   Up      0.0.0.0:9090->9090/tcp,:::9090->9090/tcp                                                                                                                                                              
+local-distributed_querier_1                    /tempo -target=querier -co ...   Up      0.0.0.0:49592->14268/tcp,:::49592->14268/tcp, 0.0.0.0:49596->3200/tcp,:::49596->3200/tcp, 0.0.0.0:49590->55680/tcp,:::49590->55680/tcp, 0.0.0.0:49589->55681/tcp,:::49589->55681/tcp,                 
+                                                                                        0.0.0.0:49594->7946/tcp,:::49594->7946/tcp                                                                                                                                                            
+local-distributed_query-frontend_1             /tempo -target=query-front ...   Up      0.0.0.0:49595->14268/tcp,:::49595->14268/tcp, 0.0.0.0:49598->3200/tcp,:::49598->3200/tcp, 0.0.0.0:49593->55680/tcp,:::49593->55680/tcp, 0.0.0.0:49591->55681/tcp,:::49591->55681/tcp,                 
+                                                                                        0.0.0.0:49597->7946/tcp,:::49597->7946/tcp                                                                                                                                                            
+local-distributed_synthetic-load-generator_1   ./start.sh                       Up
+```
+
+2. If you're interested you can see the wal/blocks as they are being created.
+
+```console
+ls tempo-data*/
+```
+
+3. The synthetic-load-generator is now printing out trace ids it's flushing into Tempo.  To view its logs use -
+
+```console
+docker-compose logs -f synthetic-load-generator
+```
+```
+synthetic-load-generator_1  | 20/10/24 08:27:09 INFO ScheduledTraceGenerator: Emitted traceId 57aedb829f352625 for service frontend route /product
+synthetic-load-generator_1  | 20/10/24 08:27:09 INFO ScheduledTraceGenerator: Emitted traceId 25fa96b9da24b23f for service frontend route /cart
+synthetic-load-generator_1  | 20/10/24 08:27:09 INFO ScheduledTraceGenerator: Emitted traceId 15b3ad814b77b779 for service frontend route /shipping
+synthetic-load-generator_1  | 20/10/24 08:27:09 INFO ScheduledTraceGenerator: Emitted traceId 3803db7d7d848a1a for service frontend route /checkout
+```
+
+Logs are in the form
+
+```
+Emitted traceId <traceid> for service frontend route /cart
+```
+
+Copy one of these trace ids.
+
+4. Navigate to [Grafana](http://localhost:3000/explore) and paste the trace id to request it from Tempo.
+Also notice that you can query Tempo metrics from the Prometheus data source setup in Grafana.
+
+5. To stop the setup use -
+
+```console
+docker-compose down -v
+```

example/docker-compose/local-distributed/tempo-distributed.yaml

@@ -0,0 +1,63 @@
+server:
+  http_listen_port: 3200
+
+distributor:
+  receivers:                           # this configuration will listen on all ports and protocols that tempo is capable of.
+    jaeger:                            # the receives all come from the OpenTelemetry collector.  more configuration information can
+      protocols:                       # be found there: https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver
+        thrift_http:                   #
+        grpc:                          # for a production deployment you should only enable the receivers you need!
+        thrift_binary:
+        thrift_compact:
+    zipkin:
+    otlp:
+      protocols:
+        http:
+        grpc:
+    opencensus:
+
+ingester:
+  trace_idle_period: 10s               # the length of time after a trace has not received spans to consider it complete and flush it
+  max_block_bytes: 1_000_000           # cut the head block when it hits this size or ...
+  max_block_duration: 5m               #   this much time passes
+  lifecycler:
+    ring:
+      replication_factor: 3
+
+memberlist:
+  abort_if_cluster_join_fails: false
+  bind_port: 7946
+  join_members:
+  - ingester-0:7946
+  - ingester-1:7946
+  - ingester-2:7946
+
+compactor:
+  compaction:
+    block_retention: 1h
+    compacted_block_retention: 10m
+
+query_frontend:
+    query_shards: 2
+
+querier:
+  frontend_worker:
+    frontend_address: query-frontend:9095
+
+storage:
+  trace:
+    backend: s3
+    s3:
+      bucket: tempo
+      endpoint: minio:9000
+      access_key: tempo
+      secret_key: supersecret
+      insecure: true
+    wal:
+      path: /tmp/tempo/wal             # where to store the the wal locally
+      encoding: none                   # wal encoding/compression.  options: none, gzip, lz4-64k, lz4-256k, lz4-1M, lz4, snappy, zstd
+    local:
+      path: /tmp/tempo/blocks
+    pool:
+      max_workers: 100                 # worker pool determines the number of parallel requests to the object store backend
+      queue_depth: 10000