Build and run OpenSearch exporter for OpenTelemetry collector

[MaxKsyunz]

To build OpenSearch Exporter from source and use it in local Docker instance, you will need to:

1. Clone the repo and check out this branch -- integ/opensearch-exporter,
2. Run make docker-otelcontribcol in the root directory.
   This will build the otelcontribcol binary and the Docker image defined in cmd/otelcontribcol/Dockerfile.
   The image will be called otelcontribcol:latest.

[MaxKsyunz]

Here's how to use the otelcontribcol image built above in a demo application.

Create a directory for Docker files

1. Add docker-compose.yml file, see bellow.
2. Add otel-collector-config.yaml file, see bellow.
3. Add .env file with the following content:

OTELCOL_IMG=otelcontribcol:latest
COLROOT=/Users/maxk/repos/otel/opentelemetry-collector-contrib
OPENSEARCH_VERSION=2.8.0

COLROOT is the absolute path to the open-telelemetry-collector-contrib working copy. The docker-compose.yml uses Dockerfiles for demo-client and demo-server used in this application.

This demo starts up OpenSearch server, Dashboards, Otel collector, and demo-client and demo-server services.
The demo services generate trace signals and send them to the collector. The collector sends the signals to OpenSearch and they can be looked at in Dashboards.

Note that most Observability features do not support this data either because they require a service map and/or because they expect data in certain indices.

docker-compose.yml

version: "2"
services:
  # Collector
  otel-collector:
    image: ${OTELCOL_IMG}
    restart: always
    command: ["--config=/etc/otel-collector-config.yaml", "${OTELCOL_ARGS}"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "1888:1888"   # pprof extension
      - "8888:8888"   # Prometheus metrics exposed by the collector
      - "8889:8889"   # Prometheus exporter metrics
      - "13133:13133" # health_check extension
      - "4317:4317"   # OTLP gRPC receiver
      - "55679:55679" # zpages extension
    depends_on:
      - opensearch-node

  demo-client:
    build:
      dockerfile: Dockerfile
      context: ${COLROOT}/examples/demo/client
    restart: always
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317
      - DEMO_SERVER_ENDPOINT=http://demo-server:7080/hello
    depends_on:
      - demo-server

  demo-server:
    build:
      dockerfile: Dockerfile
      context: ${COLROOT}/examples/demo/server
    restart: always
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317
    ports:
      - "7080"
    depends_on:
      - otel-collector

  opensearch-node:
    image: opensearchproject/opensearch:${OPENSEARCH_VERSION}
    container_name: opensearch-node
    environment:
    - cluster.name=opensearch-cluster
    - DISABLE_SECURITY_PLUGIN=true
    - discovery.seed_hosts=opensearch-node # Nodes to look for when discovering the cluster
    - cluster.initial_cluster_manager_nodes=opensearch-node # Nodes eligibile to serve as cluster manager
    - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
    - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    - node.name=opensearch-node
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - .:/usr/share/opensearch/parent
    ports:
      - 9201:9200
      - 9601:9600 # required for Performance Analyzer

  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:${OPENSEARCH_VERSION}
    container_name: opensearch-dashboards-demo
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
        OPENSEARCH_HOSTS: '["http://opensearch-node:9200"]'
        DISABLE_SECURITY_DASHBOARDS_PLUGIN: true

otel-collector-config.yaml

receivers:
  otlp:
    protocols:
      grpc:

exporters:
  opensearch/trace:
      endpoints: [http://opensearch-node:9200]

processors:
  batch:

extensions:
  health_check:
  pprof:
    endpoint: :1888
  zpages:
    endpoint: :55679

service:
  extensions: [pprof, zpages, health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [opensearch/trace]
  telemetry:
    logs:
      level: "debug"

[MaxKsyunz]

Here's quick video of this demo in action:

OpenSearch.Otel.Exporter.mp4