Opensearch observability otel

.env

@@ -43,7 +43,7 @@ EMAIL_SERVICE_PORT=6060
 EMAIL_SERVICE_ADDR=http://emailservice:${EMAIL_SERVICE_PORT}
 
 # Feature Flag Service
-FEATURE_FLAG_SERVICE_PORT=8081
+FEATURE_FLAG_SERVICE_PORT=8881
 FEATURE_FLAG_SERVICE_ADDR=featureflagservice:${FEATURE_FLAG_SERVICE_PORT}
 FEATURE_FLAG_SERVICE_HOST=feature-flag-service
 FEATURE_FLAG_GRPC_SERVICE_PORT=50053
@@ -53,6 +53,10 @@ FEATURE_FLAG_GRPC_SERVICE_ADDR=featureflagservice:${FEATURE_FLAG_GRPC_SERVICE_PO
 FRONTEND_PORT=8080
 FRONTEND_ADDR=frontend:${FRONTEND_PORT}
 
+# Nginx Proxy
+NGINX_PORT=90
+NGINX_ADDR=nginx:${FRONTEND_PORT}
+
 # Frontend Proxy (Envoy)
 FRONTEND_HOST=frontend
 ENVOY_PORT=8080
@@ -111,3 +115,13 @@ JAEGER_SERVICE_HOST=jaeger
 PROMETHEUS_SERVICE_PORT=9090
 PROMETHEUS_SERVICE_HOST=prometheus
 PROMETHEUS_ADDR=${PROMETHEUS_SERVICE_HOST}:${PROMETHEUS_SERVICE_PORT}
+
+# OpenSearch Node1
+OPENSEARCH1_PORT=9200
+OPENSEARCH1_HOST=opensearch-node1
+OPENSEARCH1_ADDR=${OPENSEARCH1_HOST}:${OPENSEARCH1_PORT}
+
+# OpenSearch Dashboard
+OPENSEARCH_DASHBOARD_PORT=5601
+OPENSEARCH_DASHBOARD_HOST=opensearch-dashboards
+OPENSEARCH_DASHBOARD_ADDR=${OPENSEARCH_DASHBOARD_HOST}:${OPENSEARCH_DASHBOARD_PORT}

.github/README.md

@@ -0,0 +1,57 @@
+# <img src="https://opentelemetry.io/img/logos/opentelemetry-logo-nav.png" alt="OTel logo" width="32"> + <img src="https://avatars.githubusercontent.com/u/80134844?s=240&v=4" alt="OTel logo" width="35"> OpenTelemetry Demo with OpenSearch 
+
+The following guide describes how to setup the OpenTelemetry demo with OpenSearch Observability using [Docker compose](#docker-compose) or [Kubernetes](#kubernetes).
+
+## Docker compose
+
+### Prerequisites
+
+- Docker
+- Docker Compose v2.0.0+
+- 4 GB of RAM for the application
+
+### Running this demo
+
+```bash
+git clone https://github.com/opensearch/opentelemetry-demo.git
+cd opentelemetry-demo
+docker compose up -d
+```
+
+### Services
+
+Once the images are built and containers are started you can access:
+
+- Webstore-Proxy (Via Nginx Proxy): http://localhost:90/
+- Webstore: http://localhost:8080/
+- Dashboards: http://localhost:5061/
+- Feature Flags UI: http://localhost:8080/feature/
+- Load Generator UI: http://localhost:8080/loadgen/
+
+OpenSearch has [documented](https://opensearch.org/docs/latest/observing-your-data/trace/trace-analytics-jaeger/#setting-up-opensearch-to-use-jaeger-data) the usage of the Observability plugin with jaeger as a trace signal source.
+
+The next instructions are similar and use the same docker compose file.
+1. Start the demo with the following command from the repository's root directory:
+   ```
+   docker compose up -d
+   ```
+**Note:** The docker compose `--no-build` flag is used to fetch released docker images from [ghcr](http://ghcr.io/open-telemetry/demo) instead of building from source.
+Removing the `--no-build` command line option will rebuild all images from source. It may take more than 20 minutes to build if the flag is omitted.
+
+### Explore and analyze the data With OpenSearch Observability
+Review revised OpenSearch [Observability Architecture](architecture.md)
+
+### Start learning OpenSearch Observability using our tutorial
+[Getting started Tutorial](../tutorial/README.md)
+
+#### Service map
+![Service map](https://docs.aws.amazon.com/images/opensearch-service/latest/developerguide/images/ta-dashboards-services.png)
+
+#### Traces
+![Traces](https://opensearch.org/docs/2.6/images/ta-trace.png)
+
+#### Correlation
+![Correlation](https://opensearch.org/docs/latest/images/observability-trace.png)
+
+#### Logs
+![Logs](https://opensearch.org/docs/latest/images/trace_log_correlation.gif)

.github/architecture.md

@@ -0,0 +1,104 @@
+# Opensearch OTEL Demo Architecture
+This document will review the OpenSearch architecture for the [OTEL demo](https://opentelemetry.io/docs/demo/) and will review how to use the new Observability capabilities
+implemented into OpenSearch.
+---
+This diagram provides an overview of the system components, showcasing the configuration derived from the OpenTelemetry Collector (otelcol) configuration file utilized by the OpenTelemetry demo application.
+
+Additionally, it highlights the observability data (traces and metrics) flow within the system.
+
+![](img/otelcol-data-flow-overview.png)
+
+---
+[OTEL DEMO](https://opentelemetry.io/docs/demo/architecture/) Describes the list of services that are composing the Astronomy Shop.
+
+They are combined of:
+ - [Accounting](https://opentelemetry.io/docs/demo/services/accounting/)
+ - [Ad](https://opentelemetry.io/docs/demo/services/ad/)
+ - [Cart](https://opentelemetry.io/docs/demo/services/cart/)
+ - [Checkout](https://opentelemetry.io/docs/demo/services/checkout/)
+ - [Currency](https://opentelemetry.io/docs/demo/services/currency/)
+ - [Email](https://opentelemetry.io/docs/demo/services/email/)
+ - [Feature Flag](https://opentelemetry.io/docs/demo/services/feature-flag/)
+ - [Fraud Detection](https://opentelemetry.io/docs/demo/services/fraud-detection/)
+ - [Frontend](https://opentelemetry.io/docs/demo/services/frontend/)
+ - [Kafka](https://opentelemetry.io/docs/demo/services/kafka/)
+ - [Payment](https://opentelemetry.io/docs/demo/services/payment/)
+ - [Product Catalog](https://opentelemetry.io/docs/demo/services/product-catalog/)
+ - [Quote](https://opentelemetry.io/docs/demo/services/quote/)
+ - [Recommendation](https://opentelemetry.io/docs/demo/services/recommendation/)
+ - [Shipping](https://opentelemetry.io/docs/demo/services/shipping/)
+ - [Fluent-Bit](../src/fluent-bit/README.md) *(nginx's otel log exported)* 
+ - [Integrations](../src/integrations/README.md) *(pre-canned OpenSearch assets)* 
+ - [DataPrepper](../src/dataprepper/README.md) *(OpenSearch's ingestion pipeline)
+
+Backend supportive services
+ - [Load Generator](http://load-generator:8089)
+   - See [description](https://opentelemetry.io/docs/demo/services/load-generator/)
+ - [Frontend Nginx Proxy](http://nginx:90) *(replacement for _Frontend-Proxy_)*
+   - See [description](../src/nginx-otel/README.md)
+ - [OpenSearch](https://opensearch-node1:9200)
+    - See [description](https://github.com/YANG-DB/opentelemetry-demo/blob/12d52cbb23bbf4226f6de2dfec840482a0a7d054/docker-compose.yml#L697)
+ - [Dashboards](http://opensearch-dashboards:5601)
+   - See [description](https://github.com/YANG-DB/opentelemetry-demo/blob/12d52cbb23bbf4226f6de2dfec840482a0a7d054/docker-compose.yml#L747) 
+ - [Prometheus](http://prometheus:9090)
+   - See [description](https://github.com/YANG-DB/opentelemetry-demo/blob/12d52cbb23bbf4226f6de2dfec840482a0a7d054/docker-compose.yml#L674)
+ - [Feature-Flag](http://feature-flag-service:8881)
+   - See [description](../src/featureflagservice/README.md)
+ - [Grafana](http://grafana:3000)
+   - See [description](https://github.com/YANG-DB/opentelemetry-demo/blob/12d52cbb23bbf4226f6de2dfec840482a0a7d054/docker-compose.yml#L637)
+
+### Services Topology
+The next diagram shows the docker compose services dependencies
+
+![](img/docker-services-topology.png)
+---
+
+## Purpose
+The purpose of this demo is to demonstrate the different capabilities of OpenSearch Observability to investigate and reflect your system.
+
+### Ingestion 
+The ingestion capabilities for OpenSearch is to be able to support multiple pipelines:
+  - [Data-Prepper](https://github.com/opensearch-project/data-prepper/) is an OpenSearch ingestion project that allows ingestion of OTEL standard signals using Otel-Collector
+  - [Jaeger](https://opensearch.org/docs/latest/observing-your-data/trace/trace-analytics-jaeger/) is an ingestion framework which has a build in capability for pushing OTEL signals into OpenSearch
+  - [Fluent-Bit](https://docs.fluentbit.io/manual/pipeline/outputs/opensearch) is an ingestion framework which has a build in capability for pushing OTEL signals into OpenSearch
+
+### Integrations -
+The integration service is a list of pre-canned assets that are loaded in a combined manner to allow users the ability for simple and automatic way to discover and review their services topology.
+
+These (demo-sample) integrations contain the following assets:
+ - components & index template mapping
+ - datasources 
+ - data-stream & indices
+ - queries
+ - dashboards
+
+Once they are loaded, the user can imminently review his OTEL demo services and dashboards that reflect the system state.
+ - [Nginx Dashboard](../src/integrations/display/nginx-logs-dashboard-new.ndjson) - reflects the Nginx Proxy server that routes all the network communication to/from the frontend
+ - [Prometheus datasource](../src/integrations/datasource/prometheus.json) - reflects the connectivity to the prometheus metric storage that allows us to federate metrics analytics queries
+ - [Logs Datastream](../src/integrations/indices/data-stream.json) - reflects the data-stream used by nginx logs ingestion and dashboards representing a well-structured [log schema](../src/integrations/mapping-templates/logs.mapping)
+
+Once these assets are loaded - the user can start reviewing its Observability dashboards and traces
+
+![Nginx Dashboard](img/nginx_dashboard.png)
+
+![Prometheus Metrics](img/prometheus_federated_metrics.png)
+
+![Trace Analytics](img/trace_analytics.png)
+
+![Service Maps](img/services.png)
+
+![Traces](img/traces.png)
+
+![ServiceGraph](img/service-graph.png)
+---
+
+### **Scenarios**
+
+How can you solve problems with OpenTelemetry? These scenarios walk you through some pre-configured problems and show you how to interpret OpenTelemetry data to solve them.
+
+- Generate a Product Catalog error for GetProduct requests with product id: OLJCESPC7Z using the Feature Flag service
+- Discover a memory leak and diagnose it using metrics and traces. Read more
+
+### **Reference**
+Project reference documentation, like requirements and feature matrices [here](https://opentelemetry.io/docs/demo/#reference)
+

README.md

@@ -54,14 +54,14 @@ adding a link below. The community is committed to maintaining the project and
 keeping it up to date for you.
 
 |                                                                                                                   |                                                                   |                                                                                              |
-| ----------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
-| [AlibabaCloud LogService](https://github.com/aliyun-sls/opentelemetry-demo)                                       | [Grafana Labs](https://github.com/grafana/opentelemetry-demo)     | [Sentry](https://github.com/getsentry/opentelemetry-demo)                                  |
-| [AppDynamics](https://www.appdynamics.com/blog/cloud/how-to-observe-opentelemetry-demo-app-in-appdynamics-cloud/) | [Helios](https://otelsandbox.gethelios.dev)                       | [Splunk](https://github.com/signalfx/opentelemetry-demo)                                    |
-| [Aspecto](https://github.com/aspecto-io/opentelemetry-demo)                                                       | [Honeycomb.io](https://github.com/honeycombio/opentelemetry-demo) | [Sumo Logic](https://www.sumologic.com/blog/common-opentelemetry-demo-application/)                                     |
-| [Coralogix](https://coralogix.com/blog/configure-otel-demo-send-telemetry-data-coralogix)                         | [Instana](https://github.com/instana/opentelemetry-demo)          | [TelemetryHub](https://github.com/TelemetryHub/opentelemetry-demo/tree/telemetryhub-backend)          |
-| [Datadog](https://github.com/DataDog/opentelemetry-demo)                                                          | [Kloudfuse](https://github.com/kloudfuse/opentelemetry-demo)      | [Teletrace](https://github.com/teletrace/opentelemetry-demo) |
+| ----------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- |----------------------------------------------------------------------------------------------|
+| [AlibabaCloud LogService](https://github.com/aliyun-sls/opentelemetry-demo)                                       | [Grafana Labs](https://github.com/grafana/opentelemetry-demo)     | [Sentry](https://github.com/getsentry/opentelemetry-demo)                                    |
+| [AppDynamics](https://www.appdynamics.com/blog/cloud/how-to-observe-opentelemetry-demo-app-in-appdynamics-cloud/) | [Helios](https://otelsandbox.gethelios.dev)                       | [Splunk](https://github.com/signalfx/opentelemetry-demo)                                     |
+| [Aspecto](https://github.com/aspecto-io/opentelemetry-demo)                                                       | [Honeycomb.io](https://github.com/honeycombio/opentelemetry-demo) | [Sumo Logic](https://www.sumologic.com/blog/common-opentelemetry-demo-application/)          |
+| [Coralogix](https://coralogix.com/blog/configure-otel-demo-send-telemetry-data-coralogix)                         | [Instana](https://github.com/instana/opentelemetry-demo)          | [TelemetryHub](https://github.com/TelemetryHub/opentelemetry-demo/tree/telemetryhub-backend) |
+| [Datadog](https://github.com/DataDog/opentelemetry-demo)                                                          | [Kloudfuse](https://github.com/kloudfuse/opentelemetry-demo)      | [Teletrace](https://github.com/teletrace/opentelemetry-demo)                                 |
 | [Dynatrace](https://www.dynatrace.com/news/blog/opentelemetry-demo-application-with-dynatrace/)                   | [Lightstep](https://github.com/lightstep/opentelemetry-demo)      | [Uptrace](https://github.com/uptrace/uptrace/tree/master/example/opentelemetry-demo)         |
-| [Elastic](https://github.com/elastic/opentelemetry-demo)                                                          | [New Relic](https://github.com/newrelic/opentelemetry-demo)      |                                                                                              |
+| [Elastic](https://github.com/elastic/opentelemetry-demo)                                                          | [New Relic](https://github.com/newrelic/opentelemetry-demo)      | [OpenSearch](https://github.com/opensearch-project/opentelemetry-demo)                               |
 |                                                                                                                   |                                                                   |                                                                                              |
 
 ## Contributing

add_hosts_locally.sh

@@ -0,0 +1,30 @@
+# Copyright The OpenTelemetry Authors
+# SPDX-License-Identifier: Apache-2.0
+
+#!/bin/bash
+# Local host that are used to help developing and debugging the OTEL demo locally
+
+# The IP address you want to associate with the hostname
+IP="127.0.0.1"
+
+# The hostname you want to associate with the IP address
+OPENSEARCH_HOST="opensearch-node1"
+OPENSEARCH_DASHBOARD="opensearch-dashboards"
+OTEL_STORE="frontend"
+FEATURE_FLAG_SERVICE="feature-flag-service"
+NGINX="nginx"
+OTEL_LOADER="loadgenerator"
+PROMETHEUS="prometheus"
+LOAD_GENERATOR="load-generator"
+GRAFANA="grafana"
+
+# Add the entry to the /etc/hosts file
+echo "$IP    $OPENSEARCH_HOST" | sudo tee -a /etc/hosts
+echo "$IP    $OPENSEARCH_DASHBOARD" | sudo tee -a /etc/hosts
+echo "$IP    $OTEL_STORE" | sudo tee -a /etc/hosts
+echo "$IP    $PROMETHEUS" | sudo tee -a /etc/hosts
+echo "$IP    $OTEL_LOADER" | sudo tee -a /etc/hosts
+echo "$IP    $NGINX" | sudo tee -a /etc/hosts
+echo "$IP    $LOAD_GENERATOR" | sudo tee -a /etc/hosts
+echo "$IP    $FEATURE_FLAG_SERVICE" | sudo tee -a /etc/hosts
+echo "$IP    $GRAFANA" | sudo tee -a /etc/hosts