feat: add otel-demo tutorial (#4858)

* feat: add otel-demo tutorial

Co-authored-by: Suhyen Im <suhyenim.kor@gmail.com>
Co-authored-by: Jaeyeon Park <donionbs7@gmail.com>
Signed-off-by: Suhyen Im <suhyenim.kor@gmail.com>
Signed-off-by: Jaeyeon Park <donionbs7@gmail.com>

* feat: add otel-demo screenshots

Co-authored-by: Suhyen Im <suhyenim.kor@gmail.com>
Co-authored-by: Jaeyeon Park <donionbs7@gmail.com>
Signed-off-by: Suhyen Im <suhyenim.kor@gmail.com>
Signed-off-by: Jaeyeon Park <donionbs7@gmail.com>

* chore: update tutorial and architecture image

Co-authored-by: Suhyen Im <suhyenim.kor@gmail.com>
Co-authored-by: Jaeyeon Park <donionbs7@gmail.com>
Signed-off-by: Suhyen Im <suhyenim.kor@gmail.com>
Signed-off-by: Jaeyeon Park <donionbs7@gmail.com>

---------

Signed-off-by: Suhyen Im <suhyenim.kor@gmail.com>
Signed-off-by: Jaeyeon Park <donionbs7@gmail.com>
Co-authored-by: Jaeyeon Park <donionbs7@gmail.com>
Co-authored-by: Namkyu Park <53862866+namkyu1999@users.noreply.github.com>

monitoring/README.md

@@ -12,6 +12,10 @@ This directory contains chaos interleaved grafana dashboards along with the util
 
   > Contains utilities required to setup monitoring infrastructure on a kubernetes cluster.
 
+- [Tutorials](./tutorials)
+
+  > Contains tutorials for users on monitoring target applications under chaos using various tools.
+
 ## Setup the LitmusChaos Infrastructure
 
 - Install the litmus chaos operator and CRDs

monitoring/tutorials/README.md

@@ -0,0 +1,7 @@
+# Tutorials
+
+This directory contains tutorials for users on monitoring target applications under chaos using various tools.
+
+- [Otel-demo](./otel-demo)
+
+  > Contains a tutorial on injecting chaos into target applications using LitmusChaos and observing the chaos with OpenTelemetry.

monitoring/tutorials/otel-demo/README.md

@@ -0,0 +1,95 @@
+# Otel-demo tutorial
+
+This tutorial provides a step-by-step guide for injecting chaos into target applications using LitmusChaos and observing the chaos with OpenTelemetry.
+
+<img src="./screenshots/otel_demo_tutorial_architecture.png" alt="otel_demo_tutorial_architecture" width="800" />
+
+### 0. Prerequisites
+- Kubernetes 1.24+
+- 8 GB of free RAM
+- Helm 3.9+
+
+### 1. Install Litmus
+1. Create the `litmus` namespace.
+   ```bash
+   kubectl create ns litmus
+   ```
+2. Add the Litmus Helm repository.
+   ```bash
+   helm repo add litmuschaos https://litmuschaos.github.io/litmus-helm/
+   ```
+3. Install Litmus using Helm.
+   ```bash
+   helm install chaos litmuschaos/litmus \
+   --namespace=litmus \
+   --set portal.frontend.service.type=NodePort \
+   --set mongodb.image.registry=ghcr.io/zcube \
+   --set mongodb.image.repository=bitnami-compat/mongodb \
+   --set mongodb.image.tag=6.0.5
+   ```
+4. Verify the installation.
+   ```bash
+   kubectl get all -n litmus
+   ```
+5. Forward the Litmus frontend service port.
+   ```bash
+   kubectl port-forward svc/chaos-litmus-frontend-service 9091:9091 -n litmus
+   ```
+   Access the Litmus frontend at [http://localhost:9091](http://localhost:9091) and log in with `admin` / `litmus`.
+
+### 2. Set Up Litmus Environment
+1. Create a new environment.
+   - Environment Name: `local`
+   - Environment Type: `Production`
+2. Configure a new chaos infrastructure.
+   - Name: `local`
+   - Chaos Components Installation: `Cluster-wide access`
+   - Installation Location (Namespace): `litmus`
+   - Service Account Name: `litmus`
+3. Deploy the new chaos infrastructure.
+   ```bash
+   cd ~/Downloads
+   kubectl apply -f local-litmus-chaos-enable.yml
+   ```
+   Wait until the status shows `CONNECTED`.
+
+### 3. Install Otel-demo microservices & Observability tools
+1. Create the `otel-demo` namespace.
+   ```bash
+   kubectl create ns otel-demo
+   ```
+2. Add the OpenTelemetry Helm repository.
+   ```bash
+   helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
+   ```
+3. Install Otel-demo microservices and Observability tools using Helm.
+   ```bash
+   cd litmus/monitoring/tutorials/otel-demo
+   helm install my-otel-demo open-telemetry/opentelemetry-demo --namespace otel-demo --values custom_otel_demo_values.yml
+   ```
+   It contains Otel-demo microservices, OpenTelemetry(with chaos metrics), Prometheus, Jaeger and Grafana.  
+4. Verify the installation.
+   ```bash
+   kubectl get all -n otel-demo
+   ```
+5. Forward the Otel-demo frontend proxy port.
+   ```bash
+   kubectl port-forward svc/my-otel-demo-frontendproxy 8080:8080 -n otel-demo
+   ```
+6. Access the following services.
+   - Web store: [http://localhost:8080/](http://localhost:8080/)
+   - Grafana: [http://localhost:8080/grafana/](http://localhost:8080/grafana/)
+   - Load Generator UI: [http://localhost:8080/loadgen/](http://localhost:8080/loadgen/)
+   - Jaeger UI: [http://localhost:8080/jaeger/ui/](http://localhost:8080/jaeger/ui/)
+
+### 4. Add Grafana Panel
+Import the `chaos-experiments-dashboard.json` file into Grafana to visualize the results of chaos experiments.
+
+### 5. Observe chaos
+Explore the following experiments to observe chaos on the Otel-demo microservices.
+
+- [Pod Network Latency](./cart-service)
+  > Performs a pod network latency experiment on the cart service.
+
+- [Pod Delete](./recommendation-service)
+  > Performs a pod delete experiment on the recommendation service.

monitoring/tutorials/otel-demo/cart-service/README.md

@@ -0,0 +1,26 @@
+# cart service pod network latency
+## Description
+- This experiment injects network latency to the cart service pod.
+- The Probe checks Prometheus metrics Latency of cart service requests.
+## Steps
+### 1. Probe Settings
+- probe type: `Prometheus Probe`
+- name: `cart-service-pod-network-latency-probe`
+- timeout: 3s
+- interval: 3s
+- prometheus endpoint: `http://my-otel-demo-prometheus-server.otel-demo:9090`
+- prometheus query: `histogram_quantile(0.99, sum(rate(duration_milliseconds_bucket{service_name=\"cartservice\"}[5m])) by (le))/1000`
+- Data Comparison:
+  - Type: Float
+  - Criteria: `<`
+  - Value: `3.0`
+### 2. Make Experiment
+1. New Experimnet
+2. Complete Overview
+3. Start off by Upload YML(cart-service-pod-network-latency.yml)
+### 3. Run Experiment
+1. Click on the `Run` button
+2. Check Experiment Status and Logs
+3. Check the Resilience Score
+4. Check the Chaos Exporter metrics using Grafana and confirm if the experiment failed. ![cart_service_pod_network_latency_experiment_result_dashboard.png](../screenshots/cart_service_pod_network_latency_experiment_result_dashboard.png)
+5. Check cart service Spanmetrics Metrics using Grafana ![cartservice_spanmetrics.png](../screenshots/cartservice_spanmetrics.png)