Tutorial : Downsampling and Unlimited Metric Retention for Prometheus.

Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Added Intro File Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Adding Initial Setup File Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Files : Initial Setup and Verification Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Initial prometheus servers Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Updating 3 prometheus to 2 prometheus Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Updating Step-2 and Verification Signed-off-by: soniasingla <soniasingla.1812@gmail.com> bash file Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Update JSON file: Environment and files Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Details and verification Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Step3 Bash file Signed-off-by: soniasingla <soniasingla.1812@gmail.com> script changes added Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Step 2: Verification updated Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Update in step 3 Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Nit improvements Signed-off-by: soniasingla <soniasingla.1812@gmail.com> step 4 updated Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Updated Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Deployed Thanos Store Gateway Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Bash script as per step-3 Signed-off-by: soniasingla <soniasingla.1812@gmail.com> No image error Signed-off-by: soniasingla <soniasingla.1812@gmail.com> step 4, wohps Signed-off-by: soniasingla <soniasingla.1812@gmail.com> step 4, wohps Signed-off-by: soniasingla <soniasingla.1812@gmail.com> thanos quay Signed-off-by: soniasingla <soniasingla.1812@gmail.com> thanos quay Signed-off-by: soniasingla <soniasingla.1812@gmail.com> CE Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Changing 2 servers to 1 server Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Update in step-2 Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Update Prometheus version Signed-off-by: soniasingla <soniasingla.1812@gmail.com> update bash Signed-off-by: soniasingla <soniasingla.1812@gmail.com> add minio Signed-off-by: soniasingla <soniasingla.1812@gmail.com> add minio Signed-off-by: soniasingla <soniasingla.1812@gmail.com> add minio Signed-off-by: soniasingla <soniasingla.1812@gmail.com> configuration path Signed-off-by: soniasingla <soniasingla.1812@gmail.com> configuration added Signed-off-by: soniasingla <soniasingla.1812@gmail.com> update docker container Signed-off-by: soniasingla <soniasingla.1812@gmail.com> update docker container Signed-off-by: soniasingla <soniasingla.1812@gmail.com> update docker container Signed-off-by: soniasingla <soniasingla.1812@gmail.com> update store conatiner Signed-off-by: soniasingla <soniasingla.1812@gmail.com> added description, removed in progress Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Add automation file Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Add automation file Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Add automation files Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Adding metrics path Signed-off-by: soniasingla <soniasingla.1812@gmail.com> i did it Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Verification : Step-3 Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Remove double into Signed-off-by: soniasingla <soniasingla.1812@gmail.com> grammarly fixes Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Thanos Compactor Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Thanos Compactor Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Configuration update, Added Thanos Querier, and TODO steps Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Fix comma error in step-4 Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Fix errors in step-4 Signed-off-by: soniasingla <soniasingla.1812@gmail.com> nipticks Signed-off-by: soniasingla <soniasingla.1812@gmail.com> update docker container Signed-off-by: soniasingla <soniasingla.1812@gmail.com> update docker container Signed-off-by: soniasingla <soniasingla.1812@gmail.com> update docker container Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Update minio container Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Add minio verification Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Update sidecar container Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Update compactor container Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Add thanosbench to generate blocks Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Add thanosbench to generate blocks Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Add thanosbench to generate blocks Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Update thanosbench container Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Add the newly generated image into sh file Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Updated image for all platforms(non-linux users) Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Update container for better picture Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Let's add more info Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Updating docker image and added thanosbench Signed-off-by: soniasingla <soniasingla.1812@gmail.com> some nit fixes Signed-off-by: soniasingla <soniasingla.1812@gmail.com> some nit fixes Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Update: Content Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Mounting the data into host Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Prometheus container Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Update the deployment and metrics docker Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Disbale compaction and change minio with TSDB step Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Add step to stop sidecar Signed-off-by: soniasingla <soniasingla.1812@gmail.com> added thanos bucket and updated store container Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Updates sidecar container Signed-off-by: soniasingla <soniasingla.1812@gmail.com> IP address correctly added and minor changes Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Updated step-3 Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Updated step-3 with nit improvements Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Updated step-4 with nit improvements Signed-off-by: soniasingla <soniasingla.1812@gmail.com> cached, loaded Signed-off-by: soniasingla <soniasingla.1812@gmail.com> updated sh file Signed-off-by: soniasingla <soniasingla.1812@gmail.com> updated sh file Signed-off-by: soniasingla <soniasingla.1812@gmail.com> updated sh file Signed-off-by: soniasingla <soniasingla.1812@gmail.com> nit improvements Signed-off-by: soniasingla <soniasingla.1812@gmail.com> nit improvements Signed-off-by: soniasingla <soniasingla.1812@gmail.com> nit improvements Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Update tutorials/katacoda/thanos/2-lts/intro.md Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Update tutorials/katacoda/thanos/2-lts/step1.md Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Update tutorials/katacoda/thanos/2-lts/step1.md Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Update tutorials/katacoda/thanos/2-lts/step1.md Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Update tutorials/katacoda/thanos/2-lts/step1.md Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: soniasingla <soniasingla.1812@gmail.com> refactored states Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Revert "Replace sync/atomic with uber-go/atomic (thanos-io#2935)" This reverts commit 20a82b9. Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Revert "cleanup shipper NewWithCompacted function (thanos-io#2940)" This reverts commit 2a54885. Signed-off-by: soniasingla <soniasingla.1812@gmail.com> running querier Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Adding updated thanos version Signed-off-by: soniasingla <soniasingla.1812@gmail.com> tutorials/katacoda/thanos/2-lts/index.json ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~users querying data Signed-off-by: soniasingla <soniasingla.1812@gmail.com> Cleaning up Signed-off-by: soniasingla <soniasingla.1812@gmail.com>
soniasingla · Oct 2, 2020 · 885737a · 885737a
1 parent bb1662a
commit 885737a
Show file tree

Hide file tree

Showing 15 changed files with 482 additions and 8 deletions.
diff --git a/tutorials/katacoda/thanos-pathway.json b/tutorials/katacoda/thanos-pathway.json
@@ -11,7 +11,7 @@
     {
       "course_id": "2-lts",
       "title": "Intro: Downsampling and unlimited metric retention for Prometheus",
-      "description": "In progress. Stay Tuned!"
+      "description": "Learn how to extend your metric retention in a cheap and easy way with Thanos."
     },
     {
       "course_id": "3-meta-monitoring",

diff --git a/tutorials/katacoda/thanos/2-lts/courseBase.sh b/tutorials/katacoda/thanos/2-lts/courseBase.sh
@@ -1 +1,6 @@
 #!/usr/bin/env bash
+
+docker pull dockerenginesonia/thanosbench:v7
+docker pull quay.io/prometheus/prometheus:v2.19.0
+docker pull quay.io/thanos/thanos:v0.15.0
+docker pull minio/minio:RELEASE.2019-01-31T00-31-19Z
diff --git a/tutorials/katacoda/thanos/2-lts/finish.md b/tutorials/katacoda/thanos/2-lts/finish.md
@@ -0,0 +1,17 @@
+# Summary
+
+Congratulations! 🎉🎉🎉
+You completed our second Thanos tutorial. Let's summarize what we learned:
+
+* To preserve the data beyond Prometheus regular retention time, we used an object storage system for backing up our historical data.
+* The Thanos Store component acts as a data retrieval proxy for data inside our object storage.
+* With Sidecar uploading metric blocks to the object store as soon as it is written to disk, it keeps the “scraper” (Prometheus with Thanos Sidecar), lightweight. This simplifies maintenance, cost, and system design.
+* Thanos Compactor improved query efficiency and also reduced the required storage size.
+
+See next courses for other tutorials about different deployment models and more advanced features of Thanos!
+
+### Feedback
+
+Do you see any bug, typo in the tutorial or you have some feedback for us?
+
+let us know on https://github.com/thanos-io/thanos or #thanos slack channel linked on https://thanos.io
diff --git a/tutorials/katacoda/thanos/2-lts/index.json b/tutorials/katacoda/thanos/2-lts/index.json
@@ -1,17 +1,52 @@
 {
   "title": "Intro: Downsampling and unlimited metric retention for Prometheus",
-  "description": "Learn how to extend you metric retention in a cheap way with Thanos.",
+  "description": "Learn how to extend your metric retention in a cheap and easy way with Thanos.",
+  "difficulty": "Beginner",
+  "time": "15-20 Minutes",
   "details": {
     "steps": [
+      {
+        "title": "Configuring Initial Prometheus Server",
+        "text": "step1.md",
+        "verify": "step1-verify.sh"
+      },
+      {
+        "title": "Thanos Sidecars",
+        "text": "step2.md",
+        "verify": "step2-verify.sh"
+      },
+      {
+        "title": "Thanos Store Gateway",
+        "text": "step3.md",
+        "answer": "step3-answer.md"
+      },
+      {
+        "title": "Thanos Compactor",
+        "text": "step4.md"
+      }
     ],
     "intro": {
       "text": "intro.md",
-      "courseData": "courseBase.sh"
+      "courseData": "courseBase.sh",
+      "credits": "https://thanos.io"
+    },
+    "files": {
+      "text": "finish.md"
     }
   },
+  "files": [
+    "prometheus0_eu1.yml",
+    "bucket_storage.yml"
+  ],
   "environment": {
     "uilayout": "editor-terminal",
-    "uisettings": "yaml"
+    "uisettings": "yaml",
+    "showdashboard": true,
+    "dashboards": [
+      {"name": "Prometheus 0 EU1", "port": 9090},
+      {"name": "Minio", "port": 9000},
+      {"name": "Thanos Query", "port": 29090}
+  ]
   },
   "backend": {
     "imageid": "docker-direct"

diff --git a/tutorials/katacoda/thanos/2-lts/intro.md b/tutorials/katacoda/thanos/2-lts/intro.md
@@ -1,7 +1,14 @@
-# In progress
+[Thanos](thanos.io) is a set of components that can be composed into a highly available metric system with unlimited storage capacity. It can be added seamlessly on top of existing Prometheus deployments.
 
-🚧 This tutorial is in progress. 🚧
+This course uses docker containers with pre-built docker images.
 
-Do you want to see this soon or you want to help us?
+In this tutorial, you will learn about :
 
-Let us know on https://github.com/thanos-io/thanos or #thanos slack channel linked on https://thanos.io
+* How to start uploading your Prometheus data seamlessly to cheap object storage thanks to Thanos sidecar.
+* How to further query data in object storage thanks to Thanos Store Gateway: a metric browser that serves metric blocks stored in Object Store via *StoreAPI* gRPC API.
+* How to query both fresh and older data in easy way through Thanos Querier.
+
+
+All of this allows you to keep your metrics in cheap and reliable object storage, allowing virtually unlimited metric retention for Prometheus.
+
+Let's jump in!
diff --git a/tutorials/katacoda/thanos/2-lts/query.png b/tutorials/katacoda/thanos/2-lts/query.png
diff --git a/tutorials/katacoda/thanos/2-lts/step1-verify.sh b/tutorials/katacoda/thanos/2-lts/step1-verify.sh
@@ -0,0 +1,9 @@
+#!/usr/bin/env bash
+
+curl -s 127.0.0.1:9090/metrics > /dev/null || exit 1
+
+curl -s 127.0.0.1:19090/metrics > /dev/null || exit 1
+
+curl -s 127.0.0.1:29090/metrics > /dev/null || exit 1
+
+echo "\"done\""
diff --git a/tutorials/katacoda/thanos/2-lts/step1.md b/tutorials/katacoda/thanos/2-lts/step1.md
@@ -0,0 +1,155 @@
+# Step 1 - Initial Prometheus Setup
+
+Thanos is a set of components that adds high availability to Prometheus installations, unlimited metrics retention and global querying across clusters.
+
+Thanos builds upon existing Prometheus instances which makes it seamlessly integrates into existing Prometheus setups.
+
+In this tutorial, we will mimic the usual state with a Prometheus server running for several months. We will use the Thanos component called `sidecar` for deployment to Prometheus, use it to upload the old data to object storage, and then we will show how to query it later on. 
+
+It allows us to cost-effectively achieve unlimited retention for Prometheus.
+
+Let's start this initial Prometheus setup, ready?
+
+## Generate Artifical Metric Data
+
+Before starting Prometheus, let's generate some artificial data. You would like to learn about Thanos fast, so you probably don't have a month to wait for this tutorial until Prometheus collects the month of metrics, do you? (:  
+
+We will use our handy [thanosbench](link here) project to do so.
+
+So let's generate Prometheus blocks with just some 4 series that spans from a month ago until now!
+
+Execute the following command:
+
+```
+mkdir -p test && docker run -i dockerenginesonia/thanosbench:v7 block plan -p realistic-key-k8s-1d-small --labels 'cluster="one"' --max-time 2019-10-18T00:00:00Z | docker run -v /root/test:/test -i  dockerenginesonia/thanosbench:v7 block gen --output.dir test
+```{{execute}}
+
+## Prometheus Configuration Files
+
+Here, we will prepare configuration files for the Prometheus instance that will run with our pre-generated data.
+It will also scrape our components we will use in this tutorial.
+
+Click `Copy To Editor` for config to propagate the configs to file.
+
+<pre class="file" data-filename="prometheus0_eu1.yml" data-target="replace">
+global:
+  scrape_interval: 15s
+  evaluation_interval: 15s
+  external_labels:
+    cluster: eu1
+    replica: 0
+
+scrape_configs:
+  - job_name: 'prometheus'
+    static_configs:
+      - targets: ['127.0.0.1:9090']
+  - job_name: 'sidecar'
+    static_configs:
+      - targets: ['127.0.0.1:19090']
+  - job_name: 'minio'
+    metrics_path: /minio/prometheus/metrics
+    static_configs:
+      - targets: ['127.0.0.1:9000']
+  - job_name: 'store_gateway'
+    static_configs:
+      - targets: ['127.0.0.1:19095']
+</pre>
+
+## Starting Prometheus Instances
+
+Let's now start the container representing Prometheus instance. 
+
+Note `-v $(pwd)/test:/prometheus \` and `--storage.tsdb.path=/prometheus` that allows us to place our generated data in Prometheus data directory.
+
+Execute the following commands:
+
+### Prepare "persistent volumes"
+
+
+### Deploying "EU1"
+
+```
+docker run -d --net=host --rm \
+    -v $(pwd)/prometheus0_eu1.yml:/etc/prometheus/prometheus.yml \
+    -v $(pwd)/test:/prometheus \
+    -u root \
+    --name prometheus-0-eu1 \
+    quay.io/prometheus/prometheus:v2.19.0 \
+    --config.file=/etc/prometheus/prometheus.yml \
+    --storage.tsdb.path=/prometheus \
+    --storage.tsdb.max-block-duration=2h \
+    --storage.tsdb.min-block-duration=2h \
+    --web.listen-address=:9090 \
+    --web.external-url=https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.com \
+    --web.enable-lifecycle \
+    --web.enable-admin-api && echo "Prometheus EU1 started!"
+```{{execute}}
+
+## Setup Verification
+
+Once started you should be able to reach the Prometheus instance here:
+
+* [Prometheus-0 EU1](https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.com/)
+
+# Installing Thanos sidecar
+
+At the end of this step, we will have running Prometheus instance with sidecar deployed. You can read more about sidecar [here](https://thanos.io/tip/components/sidecar.md/).
+
+
+## Deployment
+
+Click snippets to add a sidecar to the Prometheus instance.
+
+### Adding sidecar to "EU1" Prometheus
+
+```
+docker run -d --net=host --rm \
+    -v $(pwd)/prometheus0_eu1.yml:/etc/prometheus/prometheus.yml \
+    -v $(pwd)/test:/prometheus \
+    --name prometheus-0-sidecar-eu1 \
+    -u root \
+    quay.io/thanos/thanos:v0.15.0 \
+    sidecar \
+    --http-address 0.0.0.0:19090 \
+    --grpc-address 0.0.0.0:19190 \
+    --reloader.config-file /etc/prometheus/prometheus.yml \
+    --prometheus.url http://127.0.0.1:9090 && echo "Started sidecar for Prometheus 0 EU1"
+```{{execute}}
+
+Now, you should have a sidecar running well. Since now Prometheus has access to sidecar metrics we can query for [`thanos_sidecar_prometheus_up`](https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.com/graph?g0.expr=thanos_sidecar_prometheus_up&g0.tab=1) to check if sidecar has access to Prometheus.
+
+## Problem statement:
+
+Let's try to play with this setup a bit.
+
+Grab a coffee (or your favorite tasty beverage). Let's verify whether the blocks were uploaded before or not? Interesting? 😉
+
+Tip: Look for `prometheus_tsdb_reloads_total` metric 🕵️‍
+
+* Check here <a href="https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.com/graph?g0.range_input=1h&g0.expr=prometheus_tsdb_reloads_total&g0.tab=1&g1.range_input=5m&g1.expr=prometheus_tsdb_head_series&g1.tab=0">`prometheus_tsdb_reloads_total`</a>
+
+## Deploying Thanos Querier
+
+Let' now start the Query component. As you remember [Thanos sidecar](https://thanos.io/tip/components/query.md/) exposes `StoreAPI`
+so we will make sure we point the Querier to the gRPC endpoints of the sidecar:
+
+Click below snippet to start the Querier.
+
+```
+docker run -d --net=host --rm \
+    --name querier \
+    quay.io/thanos/thanos:v0.15.0 \
+    query \
+    --http-address 0.0.0.0:29090 \
+    --query.replica-label replica \
+    --store 127.0.0.1:19190 \
+    --store 127.0.0.1:10906 && echo "Started Thanos Querier"
+```{{execute}}
+
+## Setup verification
+
+Thanos Querier exposes very similar UI to the Prometheus, but on top of many `StoreAPIs, you wish to connect to.
+
+To check if the Querier works as intended let's look on [Querier UI `Store` page](https://[[HOST_SUBDOMAIN]]-29090-[[KATACODA_HOST]].environments.katacoda.com/stores).
+
+This should list the sidecar, including the external label.
diff --git a/tutorials/katacoda/thanos/2-lts/step2-verify.sh b/tutorials/katacoda/thanos/2-lts/step2-verify.sh
@@ -0,0 +1,13 @@
+#!/usr/bin/env bash
+
+curl -s 127.0.0.1:9090/metrics > /dev/null || exit 1
+
+curl -s 127.0.0.1:19090/metrics > /dev/null || exit 1
+
+curl -s 127.0.0.1:29090/metrics > /dev/null || exit 1
+
+curl -s 127.0.0.1:9090/metrics > /dev/null || exit 1
+
+curl -s 127.0.0.1:19090/metrics > /dev/null || exit 1
+
+echo "\"done\""
diff --git a/tutorials/katacoda/thanos/2-lts/step2.md b/tutorials/katacoda/thanos/2-lts/step2.md
@@ -0,0 +1,68 @@
+# Step 2 - Object Storage Configuration
+
+In this step, we will configure the object store and change sidecar to upload to the object-store.
+
+## Running Minio
+
+Now, execute the command
+
+```
+mkdir -p /storage/thanos && docker run -d --name minio -v /storage:/data -p 9000:9000 -e "MINIO_ACCESS_KEY=minio" -e "MINIO_SECRET_KEY=minio123" minio/minio:RELEASE.2019-01-31T00-31-19Z server /data
+```{{execute}}
+
+## Verification
+
+Now, you should have minio running well.
+
+To check if the Minio is working as intended, let's check out [here](https://[[HOST_SUBDOMAIN]]-9000-[[KATACODA_HOST]].environments.katacoda.com/minio/)
+
+Enter the credentials as mentioned below:
+
+**Access Key** = `minio`
+**Secret Key** = `minio123`
+
+## Configuration :
+
+The configuration file content :
+
+Click `Copy To Editor` for config to propagate the configs to the file `bucket_storage.yml`:
+
+<pre class="file" data-filename="bucket_storage.yml" data-target="replace">
+type: S3
+config:
+  bucket: "thanos"
+  endpoint: "127.0.0.1:9000"
+  insecure: true
+  signature_version2: true
+  access_key: "minio"
+  secret_key: "minio123"
+</pre>
+
+Before moving forward, we need to stop the `sidecar container` and we can do so by executing the following command:
+
+```
+docker stop prometheus-0-sidecar-eu1
+```{{execute}}
+
+Now, execute the following command :
+
+```
+docker run -d --net=host --rm \
+    -v $(pwd)/bucket_storage.yml:/etc/prometheus/bucket_storage.yml \
+    -v $(pwd)/test:/prometheus \
+    --name sidecar \
+    -u root \
+    quay.io/thanos/thanos:v0.15.0 \
+    sidecar \
+    --tsdb.path                 /prometheus \
+    --objstore.config-file      /etc/prometheus/bucket_storage.yml \
+    --prometheus.url            http://127.0.0.1:9090 \
+    --http-address              0.0.0.0:19090    \
+    --grpc-address              0.0.0.0:19190 && echo "Store API exposed"
+```{{execute}}
+
+The flag `--objstore.config-file` loads all the required configuration from the file to ship the TSDB blocks to an object storage bucket, the storage endpoints, and the credentials used.
+
+## Verification
+
+We can check whether the data is uploaded into `thanos` bucket by visitng [Minio](https://[[HOST_SUBDOMAIN]]-9000-[[KATACODA_HOST]].environments.katacoda.com/minio/). The stored metrics will also be available in the object storage.
diff --git a/tutorials/katacoda/thanos/2-lts/step3-answer.md b/tutorials/katacoda/thanos/2-lts/step3-answer.md
@@ -0,0 +1,19 @@
+## Answer
+
+**In an HA Prometheus setup with Thanos sidecars, would there be issues with multiple sidecars attempting to upload the same data blocks to object storage?**
+
+This is handled by having unique **external labels** for all Prometheus, sidecar instances and HA replicas. To indicate that all replicas are storing same targets, they differ only in one label. 
+
+For an instance, consider the situation below:
+
+```
+First:
+"cluster": "prod1"
+"replica": "0"
+
+Second:
+"cluster":"prod1"
+"replica": "1"
+```
+
+There is no problem with storing them since the label sets are **unique**.