Getting started otel (#174)

* update Getting started for open-telemetry integration Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update Getting started for open-telemetry integration Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update Getting started for open-telemetry integration Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update Getting started for open-telemetry integration Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update Getting started for open-telemetry integration add getting started dashboard asset + markdown Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update Getting started for open-telemetry integration add getting started dashboard asset + markdown Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update Getting started for nginx integration add getting started dashboard asset + markdown Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add schema steps section for the getting started Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update cli README.md Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update getting started with index-pattern creation phase Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update getting started with services-map index-pattern Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update getting started with services-map index-pattern Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update two workflows: - quickstart - connect to an existing collector Signed-off-by: YANGDB <yang.db.dev@gmail.com> * add savedObjectId to the gallery dashboard images links Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update otel assets for getting started Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update otel assets for getting started Signed-off-by: YANGDB <yang.db.dev@gmail.com> * update otel assets for getting started Signed-off-by: YANGDB <yang.db.dev@gmail.com> * move getting-started into its own folder to become an independed ndjson resource - getting started - getting_started.json - metadata - Getting-Started.md - markdown - sample.json - sample data Signed-off-by: YANGDB <yang.db.dev@gmail.com> * move getting-started into its own folder to become an independed ndjson resource - getting started - getting_started.json - metadata - Getting-Started.md - markdown - data - sample data Signed-off-by: YANGDB <yang.db.dev@gmail.com> * move getting-started into its own folder to become an independed ndjson resource - getting started - getting_started.json - metadata - Getting-Started.md - markdown - data - sample data Signed-off-by: YANGDB <yang.db.dev@gmail.com> * move getting-started into its own folder to become an independed ndjson resource - getting started - getting_started.json - metadata - Getting-Started.md - markdown - data - sample data Signed-off-by: YANGDB <yang.db.dev@gmail.com> --------- Signed-off-by: YANGDB <yang.db.dev@gmail.com>
opensearch-project · Jul 30, 2024 · e73b657 · e73b657
1 parent 54ef727
commit e73b657
Show file tree

Hide file tree

Showing 17 changed files with 2,648 additions and 31 deletions.
diff --git a/cli/README.md b/cli/README.md
@@ -24,6 +24,17 @@ $ pip install --editable .
 $ osints
 ```
 
+You can also install this as a python `venv` virtual python environment:
+
+```bash
+python3 -m venv venv && source ./venv/bin/activate && pip install -r requirements.txt
+````
+
+And then :
+```bash
+pip install .
+```
+
 Or you can skip the install entirely and run it as a module:
 
 ```bash

diff --git a/docs/getting-started/.env b/docs/getting-started/.env
@@ -0,0 +1,14 @@
+# OpenSearch version
+OPENSEARCH_VERSION=2.15.0
+OPENSEARCH_ADMIN_PASSWORD=my_%New%_passW0rd!@#
+OPENSEARCH_INITIAL_ADMIN_PASSWORD=my_%New%_passW0rd!@#
+
+# OpenSearch Node1
+OPENSEARCH_PORT=9200
+OPENSEARCH_HOST=localhost
+OPENSEARCH_ADDR=${OPENSEARCH_HOST}:${OPENSEARCH_PORT}
+
+# OpenSearch Dashboard
+OPENSEARCH_DASHBOARD_PORT=5601
+OPENSEARCH_DASHBOARD_HOST=localhost
+OPENSEARCH_DASHBOARD_ADDR=${OPENSEARCH_DASHBOARD_HOST}:${OPENSEARCH_DASHBOARD_PORT}
diff --git a/docs/getting-started/README.md b/docs/getting-started/README.md
@@ -0,0 +1,86 @@
+# OpenSearch Docker Compose Setup
+
+This repository contains a Docker Compose configuration to set up an OpenSearch cluster with OpenSearch Dashboards. The setup includes one OpenSearch node and an instance of OpenSearch Dashboards, designed for easy deployment and management of an OpenSearch environment.
+
+## Purpose
+
+The purpose of this setup is to provide a simple and efficient way to deploy and run OpenSearch and OpenSearch Dashboards using Docker. This setup is ideal for development, testing, and small-scale deployments.
+
+## Prerequisites
+
+Before you start, ensure you have the following installed on your machine:
+
+- Docker
+- Docker Compose
+
+## Services
+
+### OpenSearch Node 1
+
+- **Image:** `opensearchproject/opensearch:${OPENSEARCH_VERSION}`
+- **Container Name:** `opensearch-node1`
+- **Ports:** `9200` (OpenSearch API), `9600` (Metrics)
+- **Environment Variables:**
+ - `cluster.name=opensearch-cluster`
+ - `node.name=opensearch-node1`
+ - `discovery.type=single-node`
+ - `bootstrap.memory_lock=true`
+ - `plugins.query.datasources.encryption.masterkey`
+ - `OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m`
+ - `OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_ADMIN_PASSWORD}`
+- **Volumes:**
+ - `opensearch-data1:/usr/share/opensearch/data`
+- **Network Mode:** `host`
+- **Ulimits:** Memory lock set to unlimited
+
+### OpenSearch Dashboards
+
+- **Image:** `opensearchproject/opensearch-dashboards:${OPENSEARCH_VERSION}`
+- **Container Name:** `opensearch-dashboards`
+- **Ports:** `5601` (Dashboards UI)
+- **Environment Variables:**
+ - `OPENSEARCH_HOSTS=["http://localhost:9200"]`
+- **Volumes:**
+ - `./opensearch_dashboards.yml:/usr/share/opensearch-dashboards/config/opensearch_dashboards.yml`
+- **Network Mode:** `host`
+
+## Usage
+
+1. **Clone the Repository:**
+ ```sh
+ git clone https://github.com/your-repo/opensearch-docker-compose.git
+ cd opensearch-docker-compose
+2. **Set Environment Variables:**
+ Review the `.env` file in the root directory and with the following variables:
+ ```sh
+ # OpenSearch version
+ OPENSEARCH_VERSION=2.15.0
+ OPENSEARCH_ADMIN_PASSWORD=Open$earch123
+ OPENSEARCH_INITIAL_ADMIN_PASSWORD=Open$earch123
+ 
+ # OpenSearch Node1
+ OPENSEARCH_PORT=9200
+ OPENSEARCH_HOST=localhost
+ OPENSEARCH_ADDR=${OPENSEARCH_HOST}:${OPENSEARCH_PORT}
+ 
+ # OpenSearch Dashboard
+ OPENSEARCH_DASHBOARD_PORT=5601
+ OPENSEARCH_DASHBOARD_HOST=localhost
+ OPENSEARCH_DASHBOARD_ADDR=${OPENSEARCH_DASHBOARD_HOST}:${OPENSEARCH_DASHBOARD_PORT}
+3. **Start the Services:**
+ ```sh
+ docker-compose up -d
+4. **Access OpenSearch Dashboards:**
+Open your web browser and navigate to http://localhost:5601 to access OpenSearch Dashboards.
+
+5. **Server Configuration**
+The OpenSearch node is configured to run as a single-node cluster with memory locking enabled to improve performance. Java options are set to allocate 512MB of heap memory.
+
+6. **Dashboards Configuration**
+OpenSearch Dashboards is configured to connect to the OpenSearch node running on http://localhost:9200.
+You can modify the opensearch_dashboards.yml file to customize the Dashboards settings.
+7. **Troubleshooting**
+If you encounter any issues, please check the container logs for more details:
+ ```sh
+ docker-compose logs
+
diff --git a/docs/getting-started/opensearch_dashboards.yml b/docs/getting-started/opensearch_dashboards.yml
@@ -0,0 +1,5 @@
+server.host: "0.0.0.0"
+opensearch.hosts: ["http://localhost:9200"]
+opensearch.username: "admin"
+opensearch.password: "my_%New%_passW0rd!@#"
+vis_type_vega.enableExternalUrls: true
diff --git a/docs/getting-started/os-cluster.yml b/docs/getting-started/os-cluster.yml
@@ -0,0 +1,38 @@
+version: '3'
+services:
+ opensearch-dashboards:
+ image: opensearchproject/opensearch-dashboards:${OPENSEARCH_VERSION}
+ container_name: opensearch-dashboards
+ ports:
+ - 5601:5601
+ environment:
+ OPENSEARCH_HOSTS: '["http://localhost:9200"]'
+ volumes:
+ - ./opensearch_dashboards.yml:/usr/share/opensearch-dashboards/config/opensearch_dashboards.yml
+ network_mode: "host"
+
+
+ opensearch-node1:
+ image: opensearchproject/opensearch:${OPENSEARCH_VERSION}
+ container_name: opensearch-node1
+ environment:
+ - cluster.name=opensearch-cluster
+ - node.name=opensearch-node1
+ - discovery.type=single-node
+ - bootstrap.memory_lock=true
+ - plugins.query.datasources.encryption.masterkey=8e3f206ea7c07cc1bfc5cf40
+ - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
+ - "OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_ADMIN_PASSWORD}"
+ ulimits:
+ memlock:
+ soft: -1
+ hard: -1
+ volumes:
+ - opensearch-data1:/usr/share/opensearch/data
+ ports:
+ - 9200:9200
+ - 9600:9600
+ network_mode: "host"
+
+volumes:
+ opensearch-data1:
diff --git a/integrations/observability/otel-services/assets/getting-started-demo-1.0.0.ndjson b/integrations/observability/otel-services/assets/getting-started-demo-1.0.0.ndjson
diff --git a/integrations/observability/otel-services/assets/otel-index-patterns-1.0.0.ndjson b/integrations/observability/otel-services/assets/otel-index-patterns-1.0.0.ndjson
diff --git a/...lity/otel-services/info/GettingStarted.md → ...ervices/getting-started/GettingStarted.md b/...lity/otel-services/info/GettingStarted.md → ...ervices/getting-started/GettingStarted.md
@@ -8,13 +8,13 @@ The following web-pages can be directly access to review the load generator
 
 - Here the synthetic testing process can be started
 
-![](./img/load-generator.png)
+![](../info/img/load-generator.png)
 
 The following web-pages can be directly access to actually simulate store purchase
 - http://localhost:8080/ (front-end) OR http://localhost:90 (nginx-forntend proxy)
 - Here the actual shop demo can be used (simulated demo shop ...)
 
-![](./img/demo-app.png)
+![](../info/img/demo-app.png)
 
 
 ---
@@ -28,12 +28,12 @@ Once all services are up and running - log-in to the Dashboard and enter the nex
 - http://localhost:5601/
 - OpenSearch Dashboard login
 
-![dashboard-login.png](img/dashboard-login.png)
+![dashboard-login.png](../info/img/dashboard-login.png)
 
 ## Store URLs in session storage
 Enable Store URLs in session storage for allowing to view and edit Vega based visualizations
 - Once logged in, go to [Advanced Setting](http://localhost:5601/app/management/opensearch-dashboards/settings)
- ![store-url-in-session.png](img/store-url-in-session.png)
+ ![store-url-in-session.png](../info/img/store-url-in-session.png)
 
 ## Installing OTEL Services Demo
 
@@ -45,54 +45,54 @@ Install OpenTelemetry Dashboards workflow - navigate to the [Observability Catal
 This will load all the OpenTelemetry Services flow dashboards and you can immediately navigate to any of the imported dashboards and monitor the ingested telemetry data.
 >  _For detailed instructions see [here](https://github.com/opensearch-project/opensearch-catalog/releases/tag/otel_services_dashboard-1.0.0)_
 
-![integration-otel-services-setup.png](img/dashboard-mng.png)
+![integration-otel-services-setup.png](../info/img/dashboard-mng.png)
 
-![integration-otel-services-dashboards.png](img/import-savedObj.png)
+![integration-otel-services-dashboards.png](../info/img/import-savedObj.png)
 
 
 ## Ingestion Rate Dashboard
 This dashboard show the 3 signals ingestion rate as they are shipped via data-prepper into OpenSearch indices
 
-![otel-ingestion-rate-dashboard.png](img/otel-ingestion-rate-dashboard.png)
+![otel-ingestion-rate-dashboard.png](../info/img/otel-ingestion-rate-dashboard.png)
 
 ## Services High Level Dashboards View
 This dashboard show the 3 signals ingestion rate as they are shipped via data-prepper into OpenSearch indices
-![services-general-dashboard.png](img/services-general-dashboard.png)
+![services-general-dashboard.png](../info/img/services-general-dashboard.png)
 
 ## Single Service Details Dashboards View
 This dashboard show the specific service details including associated high level view of the metrics collected for the service
-![specific-service-dashboard.png](img/specific-service-dashboard.png)
+![specific-service-dashboard.png](../info/img/specific-service-dashboard.png)
 
 ## Service Metrics View
 
 ### Service Metrics system projection Dashboards
 This dashboard show the specific service details in particular the system related metrics collected for the service
 
-![amp-services-system-metrics-details.png](img/amp-services-system-metrics-details.png)
+![amp-services-system-metrics-details.png](../info/img/amp-services-system-metrics-details.png)
 
 ### Service Metrics network projection Dashboards
 This dashboard show the specific service details in particular the network related metrics collected for the service
 
-![amp-services-network-metrics-details.png](img/amp-services-network-metrics-details.png)
+![amp-services-network-metrics-details.png](../info/img/amp-services-network-metrics-details.png)
 
 ## Trace Analytics
 Traces can also be seen using the following [menu-item](http://localhost:5601/app/observability-traces#/traces)
-![traces-analytics-dialog.png](img/traces-analytics-dialog.png)
+![traces-analytics-dialog.png](../info/img/traces-analytics-dialog.png)
 - showing a table of traces with their duration/errors rates
 
-![traces-analytics-dialog-spans.png](img/traces-analytics-dialog-spans.png)
+![traces-analytics-dialog-spans.png](../info/img/traces-analytics-dialog-spans.png)
 - showing a specific trace's spans water fall chart
 ---
 ## Services Analytics
 Services can be also be seen using the following [menu-item](http://localhost:5601/app/observability-traces#/services)
 
-![service-analytics-dialog-list.png](img/service-analytics-dialog-list.png)
+![service-analytics-dialog-list.png](../info/img/service-analytics-dialog-list.png)
 - showing the table of services including their Group /Avg Duration / Errors
 
-![service-analytics-dialog-services_map.png](img/service-analytics-dialog-services_map.png)
+![service-analytics-dialog-services_map.png](../info/img/service-analytics-dialog-services_map.png)
 - showing the service map graph with the services relationships
 
-![service-analytics-dialog-trace_group.png](img/service-analytics-dialog-trace_group.png)
+![service-analytics-dialog-trace_group.png](../info/img/service-analytics-dialog-trace_group.png)
 - showing the services trace-group charts according to Avg Duration / Traces-Error rates / Traces Request rates
 
 
@@ -101,18 +101,18 @@ Services can be also be seen using the following [menu-item](http://localhost:56
 
 ### Setting Up Prometheus `datasource`
 - First [setup](http://localhost:5601/app/datasources#/new) the Prometheus datasource we wanted to connect with
- ![configure-prometheus-datasource.png](img/configure-prometheus-datasource.png)
+ ![configure-prometheus-datasource.png](../info/img/configure-prometheus-datasource.png)
 
 ### Query Prometheus OTEL metrics
 - [Select the OTEL metrics](http://localhost:5601/app/observability-metrics#/) (Prometheus / OpenSearch) we want to display
- ![metrics-analytics-prometheus.png](img/metrics-analytics-prometheus.png)
- ![metrics-analytics-prometheus-select-metrics.png](img/metrics-analytics-prometheus-select-metrics.png)
+ ![metrics-analytics-prometheus.png](../info/img/metrics-analytics-prometheus.png)
+ ![metrics-analytics-prometheus-select-metrics.png](../info/img/metrics-analytics-prometheus-select-metrics.png)
 
 ### Query OpenSearch OTEL metrics
 - Select the `ss4o_metrics-*-*` based index to view the OTEL metrics stored in OpenSearch:
- ![metrics-analytics-opensearch-otel-metrics.png](img/metrics-analytics-opensearch-otel-metrics.png)
- ![metrics-analytics-opensearch-otel-metrics-select-metrics.png](img/metrics-analytics-opensearch-otel-metrics-select-metrics.png)
+ ![metrics-analytics-opensearch-otel-metrics.png](../info/img/metrics-analytics-opensearch-otel-metrics.png)
+ ![metrics-analytics-opensearch-otel-metrics-select-metrics.png](../info/img/metrics-analytics-opensearch-otel-metrics-select-metrics.png)
 
 ## Discover - Log Exploration
 - [Discover](http://localhost:5601/app/data-explorer/discover) the OTEL logs stored inside OpenSearch
- ![logs-discovery-otel.png](img/logs-discovery-otel.png)
+ ![logs-discovery-otel.png](../info/img/logs-discovery-otel.png)