Katacoda Tutorial : Downsampling and Unlimited Metric Retention for P…

…rometheus (#2840)

* Tutorial : Downsampling and Unlimited Metric Retention for Prometheus.

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Added Intro File

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Adding Initial Setup File

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Files : Initial Setup and Verification

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Initial prometheus servers

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Updating 3 prometheus to 2 prometheus

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Updating Step-2 and Verification

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

bash file

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Update JSON file: Environment and files

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Details and verification

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Step3 Bash file

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

script changes added

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Step 2: Verification updated

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Update in step 3

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Nit improvements

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

step 4 updated

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Updated

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Deployed Thanos Store Gateway

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Bash script as per step-3

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

No image error

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

step 4, wohps

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

step 4, wohps

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

thanos quay

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

thanos quay

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

CE

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Changing 2 servers to 1 server

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Update in step-2

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Update Prometheus version

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

update bash

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

add minio

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

add minio

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

add minio

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

configuration path

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

configuration added

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

update docker container

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

update docker container

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

update docker container

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

update store conatiner

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

added description, removed in progress

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Add automation file

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Add automation file

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Add automation files

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Adding metrics path

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

i did it

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Verification : Step-3

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Remove double into

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

grammarly fixes

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Thanos Compactor

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Thanos Compactor

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Configuration update, Added Thanos Querier, and TODO steps

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Fix comma error in step-4

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Fix errors in step-4

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

nipticks

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

update docker container

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

update docker container

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

update docker container

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Update minio container

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Add minio verification

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Update sidecar container

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Update compactor container

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Add thanosbench to generate blocks

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Add thanosbench to generate blocks

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Add thanosbench to generate blocks

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Update thanosbench container

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Add the newly generated image into sh file

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Updated image for all platforms(non-linux users)

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Update container for better picture

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Let's add more info

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Updating docker image and added thanosbench

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

some nit fixes

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

some nit fixes

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Update: Content

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Mounting the data into host

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Prometheus container

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Update the deployment and metrics docker

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Disbale compaction and change minio with TSDB step

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Add step to stop sidecar

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

added thanos bucket and updated store container

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Updates sidecar container

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

IP address correctly added and minor changes

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Updated step-3

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Updated step-3 with nit improvements

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Updated step-4 with nit improvements

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

cached, loaded

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

updated sh file

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

updated sh file

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

updated sh file

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

nit improvements

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

nit improvements

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

nit improvements

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Update tutorials/katacoda/thanos/2-lts/intro.md

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Update tutorials/katacoda/thanos/2-lts/step1.md

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Update tutorials/katacoda/thanos/2-lts/step1.md

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Update tutorials/katacoda/thanos/2-lts/step1.md

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Update tutorials/katacoda/thanos/2-lts/step1.md

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

refactored states

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Revert "Replace sync/atomic with uber-go/atomic (#2935)"

This reverts commit 20a82b9.

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Revert "cleanup shipper NewWithCompacted function (#2940)"

This reverts commit 2a54885.

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

running querier

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Adding updated thanos version

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

tutorials/katacoda/thanos/2-lts/index.json
~
~
~
~
~
~
~
~
~
~
~
~users querying data

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Cleaning up

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Removing new line from compactor

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Formatting all the files

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Linting & Formatting

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

* fixes and updates

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

* Tiny modifications and improvements to LTS tutorial.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Tiny modifications and improvements to LTS tutorial.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

* Tiny modifications and improvements to LTS tutorial.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Nit Improvements

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

* Modifications and Improvements

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

* Updates added

Signed-off-by: soniasingla <soniasingla.1812@gmail.com>

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>

tutorials/katacoda/thanos-pathway.json

@@ -11,7 +11,7 @@
  {
  "course_id": "2-lts",
  "title": "Intro: Downsampling and unlimited metric retention for Prometheus",
- "description": "In progress. Stay Tuned!"
+ "description": "Learn how to extend your metric retention in a cheap and easy way with Thanos."
  },
  {
  "course_id": "3-meta-monitoring",

tutorials/katacoda/thanos/2-lts/courseBase.sh

@@ -1 +1,8 @@
 #!/usr/bin/env bash
+
+docker pull minio/minio:RELEASE.2019-01-31T00-31-19Z
+docker pull quay.io/prometheus/prometheus:v2.20.0
+docker pull quay.io/thanos/thanos:v0.16.0
+docker pull quay.io/thanos/thanosbench:v0.2.0-rc.1
+
+mkdir /root/editor

tutorials/katacoda/thanos/2-lts/finish.md

@@ -0,0 +1,17 @@
+# Summary
+
+Congratulations! 🎉🎉🎉
+You completed our second Thanos tutorial. Let's summarize what we learned:
+
+* To preserve the data beyond Prometheus regular retention time, we used an object storage system for backing up our historical data.
+* The Thanos Store component acts as a data retrieval proxy for data inside our object storage.
+* With Sidecar uploading metric blocks to the object store as soon as it is written to disk, it keeps the “scraper” (Prometheus with Thanos Sidecar), lightweight. This simplifies maintenance, cost, and system design.
+* Thanos Compactor improved query efficiency and also reduced the required storage size.
+
+See next courses for other tutorials about different deployment models and more advanced features of Thanos!
+
+### Feedback
+
+Do you see any bug, typo in the tutorial or you have some feedback for us?
+
+let us know on https://github.com/thanos-io/thanos or #thanos slack channel linked on https://thanos.io

tutorials/katacoda/thanos/2-lts/index.json

@@ -1,17 +1,53 @@
 {
  "title": "Intro: Downsampling and unlimited metric retention for Prometheus",
- "description": "Learn how to extend you metric retention in a cheap way with Thanos.",
+ "description": "Learn how to extend your metric retention in a cheap and easy way with Thanos.",
+ "difficulty": "Moderate",
  "details": {
  "steps": [
+ {
+ "title": "Configuring Initial Prometheus Server",
+ "text": "step1.md",
+ "verify": "step1-verify.sh"
+ },
+ {
+ "title": "Thanos Sidecars",
+ "text": "step2.md",
+ "verify": "step2-verify.sh"
+ },
+ {
+ "title": "Thanos Store Gateway",
+ "text": "step3.md",
+ "answer": "step3-answer.md"
+ },
+ {
+ "title": "Thanos Compactor",
+ "text": "step4.md"
+ }
  ],
  "intro": {
  "text": "intro.md",
- "courseData": "courseBase.sh"
+ "courseData": "courseBase.sh",
+ "credits": "https://thanos.io"
+ },
+ "files": {
+ "text": "finish.md",
+ "credits": "test"
  }
  },
+ "files": [
+ "prometheus0_eu1.yml",
+ "bucket_storage.yaml"
+ ],
  "environment": {
  "uilayout": "editor-terminal",
- "uisettings": "yaml"
+ "uisettings": "yaml",
+ "uieditorpath": "/root/editor",
+ "showdashboard": true,
+ "dashboards": [
+ {"name": "Prometheus 0 EU1", "port": 9090},
+ {"name": "Minio", "port": 9000},
+ {"name": "Thanos Query", "port": 9091}
+ ]
  },
  "backend": {
  "imageid": "docker-direct"

tutorials/katacoda/thanos/2-lts/intro.md

@@ -1,7 +1,30 @@
-# In progress
+# Intro: Downsampling and unlimited metric retention for Prometheus
 
-🚧 This tutorial is in progress. 🚧
+They say that [Thanos](thanos.io) is a set of components that can be composed into a highly available metric system with **unlimited storage capacity**
+and that it can be added **seamlessly** on top of existing Prometheus deployments. 🤔🤔
 
-Do you want to see this soon or you want to help us?
+In this course you can experience all of this yourself.
 
-Let us know on https://github.com/thanos-io/thanos or #thanos slack channel linked on https://thanos.io
+In this tutorial, you will learn about:
+
+* How to start uploading your Prometheus data seamlessly to cheap object storage thanks to Thanos sidecar.
+* How to further query your data in object storage thanks to Thanos Store Gateway.
+* How to query both fresh and older data in easy way through Thanos Querier.
+
+All of this allows you to keep your metrics in cheap and reliable object storage, allowing virtually unlimited metric retention for Prometheus.
+
+> NOTE: This course uses docker containers with pre-built Thanos, Prometheus, and Minio Docker images available publicly.
+> However, a similar scenario will work with any other deployment method like Kubernetes or systemd, etc.
+
+### Prerequisites
+
+Please complete first intro course about GlobalView before jumping into this one! 🤗
+
+### Feedback
+
+Do you see any bug, typo in the tutorial or you have some feedback for us?
+Let us know on https://github.com/thanos-io/thanos or #thanos slack channel linked on https://thanos.io
+
+### Contributed by:
+
+* Sonia Singla [@soniasingla](http://github.com/soniasingla)

tutorials/katacoda/thanos/2-lts/step1-verify.sh

@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+
+curl -s 127.0.0.1:9090/metrics >/dev/null || exit 1
+curl -s 127.0.0.1:19090/metrics >/dev/null || exit 1
+curl -s 127.0.0.1:9091/metrics >/dev/null || exit 1
+
+echo '"done"'

tutorials/katacoda/thanos/2-lts/step1.md

@@ -0,0 +1,145 @@
+# Step 1 - Initial Prometheus Setup
+
+In this tutorial, we will mimic the usual state with a Prometheus server running for... a year!.
+We will use it to seamlessly backup all old data in the object storage and configure Prometheus for continuous backup mode, which
+will allow us to cost-effectively achieve unlimited retention for Prometheus.
+
+Last but not the least, we will go through setting all up for querying and automated maintenance (e.g compactions, retention and downsampling).
+
+In order to showcase all of this, let's start with single cluster setup from the previous course. Let's start this initial Prometheus setup, ready?
+
+## Generate Artificial Metrics for 1 year
+
+Actually, before starting Prometheus, let's generate some **artificial data**. You most likely want to learn about Thanos fast,
+so you probably don't have months to wait for this tutorial until Prometheus collects the month of metrics, do you? (:
+
+We will use our handy [thanosbench](https://github.com/thanos-io/thanosbench) project to do so! Let's generate Prometheus
+data (in form of TSDB blocks) with just 5 series (gauges) that spans from a year ago until now (-6h)!
+
+Execute the following command (should take few seconds):
+
+```
+mkdir -p /root/prom-eu1 && docker run -i quay.io/thanos/thanosbench:v0.2.0-rc.1 block plan -p continuous-365d-tiny --labels 'cluster="eu1"' --max-time=6h | docker run -v /root/prom-eu1:/prom-eu1 -i quay.io/thanos/thanosbench:v0.2.0-rc.1 block gen --output.dir prom-eu1
+```{{execute}}
+
+On successful block creation you should see following log lines:
+
+```
+level=info ts=2020-10-20T18:28:42.625041939Z caller=block.go:87 msg="all blocks done" count=X
+level=info ts=2020-10-20T18:28:42.625100758Z caller=main.go:118 msg=exiting cmd="block gen"
+```
+
+Run `ls -lR /root/prom-eu1` to see dozens of generated TSDB blocks.
+
+## Prometheus Configuration File
+
+Here, we will prepare configuration files for the Prometheus instance that will run with our pre-generated data.
+It will also scrape our components we will use in this tutorial.
+
+Click `Copy To Editor` for config to propagate the configs to file.
+
+<pre class="file" data-filename="prometheus0_eu1.yml" data-target="replace">
+global:
+ scrape_interval: 5s
+ external_labels:
+ cluster: eu1
+ replica: 0
+ tenant: team-eu # Not needed, but a good practice if you want to grow this to multi-tenant system some day.
+
+scrape_configs:
+ - job_name: 'prometheus'
+ static_configs:
+ - targets: ['127.0.0.1:9090']
+ - job_name: 'sidecar'
+ static_configs:
+ - targets: ['127.0.0.1:19090']
+ - job_name: 'minio'
+ metrics_path: /minio/prometheus/metrics
+ static_configs:
+ - targets: ['127.0.0.1:9000']
+ - job_name: 'querier'
+ static_configs:
+ - targets: ['127.0.0.1:9091']
+ - job_name: 'store_gateway'
+ static_configs:
+ - targets: ['127.0.0.1:19092']
+</pre>
+
+## Starting Prometheus Instance
+
+Let's now start the container representing Prometheus instance.
+
+Note `-v /root/prom-eu1:/prometheus \` and `--storage.tsdb.path=/prometheus` that allows us to place our generated data
+in Prometheus data directory.
+
+Let's deploy Prometheus now. Note that we disabled local Prometheus compactions `storage.tsdb.max-block-duration` and `min` flags.
+Currently, this is important for the basic object storage backup scenario to avoid conflicts between the bucket and local compactions.
+Read more [here](https://thanos.io/tip/components/sidecar.md/#sidecar).
+
+We also extend Prometheus retention: `--storage.tsdb.retention.time=1000d`. This is because Prometheus by default removes all data older
+than 2 weeks. And we have a year (:
+
+### Deploying "EU1"
+
+```
+docker run -d --net=host --rm \
+ -v /root/editor/prometheus0_eu1.yml:/etc/prometheus/prometheus.yml \
+ -v /root/prom-eu1:/prometheus \
+ -u root \
+ --name prometheus-0-eu1 \
+ quay.io/prometheus/prometheus:v2.20.0 \
+ --config.file=/etc/prometheus/prometheus.yml \
+ --storage.tsdb.retention.time=1000d \
+ --storage.tsdb.path=/prometheus \
+ --storage.tsdb.max-block-duration=2h \
+ --storage.tsdb.min-block-duration=2h \
+ --web.listen-address=:9090 \
+ --web.external-url=https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.com \
+ --web.enable-lifecycle \
+ --web.enable-admin-api
+```{{execute}}
+
+## Setup Verification
+
+Once started you should be able to reach the Prometheus instance here and query.. 1 year of data!
+
+* [Prometheus-0 EU1](https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.com/)
+
+## Thanos Sidecar & Querier
+
+Similar to previous course, let's setup global view querying with sidecar:
+
+```
+docker run -d --net=host --rm \
+ --name prometheus-0-eu1-sidecar \
+ -u root \
+ quay.io/thanos/thanos:v0.16.0 \
+ sidecar \
+ --http-address 0.0.0.0:19090 \
+ --grpc-address 0.0.0.0:19190 \
+ --prometheus.url http://127.0.0.1:9090
+```{{execute}}
+
+And Querier. As you remember [Thanos sidecar](https://thanos.io/tip/components/query.md/) exposes `StoreAPI`
+so we will make sure we point the Querier to the gRPC endpoints of the sidecar:
+
+```
+docker run -d --net=host --rm \
+ --name querier \
+ quay.io/thanos/thanos:v0.16.0 \
+ query \
+ --http-address 0.0.0.0:9091 \
+ --query.replica-label replica \
+ --store 127.0.0.1:19190
+```{{execute}}
+
+## Setup verification
+
+Similar to previous course let's check if the Querier works as intended. Let's look on
+[Querier UI `Store` page](https://[[HOST_SUBDOMAIN]]-9091-[[KATACODA_HOST]].environments.katacoda.com/stores).
+
+This should list the sidecar, including the external labels.
+
+On graph you should also see our 5 series for 1y time, thanks to Prometheus and sidecar StorAPI: [Graph](https://[[HOST_SUBDOMAIN]]-9091-[[KATACODA_HOST]].environments.katacoda.com).
+
+Click `Continue` to see how we can move this data to much cheaper and easier to operate object storage.

tutorials/katacoda/thanos/2-lts/step2-verify.sh

@@ -0,0 +1,9 @@
+#!/usr/bin/env bash
+
+curl -s 127.0.0.1:9090/metrics >/dev/null || exit 1
+curl -s 127.0.0.1:19090/metrics >/dev/null || exit 1
+curl -s 127.0.0.1:9091/metrics >/dev/null || exit 1
+
+curl -s 127.0.0.1:19090/metrics >/dev/null || exit 1
+
+echo '"done"'

tutorials/katacoda/thanos/2-lts/step2.md

@@ -0,0 +1,97 @@
+# Step 2 - Object Storage Continuous Backup
+
+Maintaining one year of data within your Prometheus is doable, but not easy. It's tricky to
+resize, backup or maintain this data long term. On top of that Prometheus does not do any replication,
+so any unavailability of Prometheus results in query unavailability.
+
+This is where Thanos comes to play. With a single configuration change we can allow Thanos Sidecar to continuously upload blocks of metrics
+that are periodically persisted to disk by the Prometheus.
+
+> NOTE: Prometheus when scraping data, initially aggregates all samples in memory and WAL (on-disk write-head-log). Only after 2-3h it "compacts"
+> the data into disk in form of 2h TSDB block. This is why we need to still query Prometheus for latest data, but overall with this change
+> we can keep Prometheus retention to minimum. It's recommended to keep Prometheus retention in this case at least 6 hours long, to have safe buffer
+> for a potential event of network partition.
+
+## Starting Object Storage: Minio
+
+Let's start simple S3-compatible Minio engine that keeps data in local disk:
+
+```
+mkdir /root/minio && \
+docker run -d --rm --name minio \
+ -v /root/minio:/data \
+ -p 9000:9000 -e "MINIO_ACCESS_KEY=minio" -e "MINIO_SECRET_KEY=melovethanos" \
+ minio/minio:RELEASE.2019-01-31T00-31-19Z \
+ server /data
+```{{execute}}
+
+Create `thanos` bucket:
+
+```
+mkdir /root/minio/thanos
+```{{execute}}
+
+## Verification
+
+To check if the Minio is working as intended, let's [open Minio server UI](https://[[HOST_SUBDOMAIN]]-9000-[[KATACODA_HOST]].environments.katacoda.com/minio/)
+
+Enter the credentials as mentioned below:
+
+**Access Key** = `minio`
+**Secret Key** = `melovethanos`
+
+## Sidear block backup
+
+All Thanos components that use object storage uses the same `objstore.config` flag with the same "little" bucket config format.
+
+Click `Copy To Editor` for config to propagate the configs to the file `bucket_storage.yaml`:
+
+<pre class="file" data-filename="bucket_storage.yaml" data-target="replace">
+type: S3
+config:
+ bucket: "thanos"
+ endpoint: "127.0.0.1:9000"
+ insecure: true
+ signature_version2: true
+ access_key: "minio"
+ secret_key: "melovethanos"
+</pre>
+
+Let's restart sidecar with updated configuration in backup mode.
+
+```
+docker stop prometheus-0-eu1-sidecar
+```{{execute}}
+
+[Thanos sidecar](https://thanos.io/tip/components/sidecar.md/) allows to backup all the blocks that Prometheus persits to
+the disk. In order to accomplish this we need to make sure that:
+
+* Sidecar has direct access to the Prometheus data directory (in our case host's /root/prom-eu1 dir) (`--tsdb.path` flag)
+* Bucket configuration is specified `--objstore.config-file`
+* `--shipper.upload-compacted` has to be set if you want to upload already compacted blocks when sidecar starts. Use this only
+when you want to upload blocks never seen before on new Prometheus introduced to Thanos system.
+
+Let's run sidecar:
+
+```
+docker run -d --net=host --rm \
+ -v /root/editor/bucket_storage.yaml:/etc/thanos/minio-bucket.yaml \
+ -v /root/prom-eu1:/prometheus \
+ --name prometheus-0-eu1-sidecar \
+ -u root \
+ quay.io/thanos/thanos:v0.16.0 \
+ sidecar \
+ --tsdb.path /prometheus \
+ --objstore.config-file /etc/thanos/minio-bucket.yaml \
+ --shipper.upload-compacted \
+ --http-address 0.0.0.0:19090 \
+ --grpc-address 0.0.0.0:19190 \
+ --prometheus.url http://127.0.0.1:9090
+```{{execute}}
+
+## Verification
+
+We can check whether the data is uploaded into `thanos` bucket by visitng [Minio](https://[[HOST_SUBDOMAIN]]-9000-[[KATACODA_HOST]].environments.katacoda.com/minio/).
+It will take couple of seconds to synchronize all blocks.
+
+Once all blocks appear in the minio `thanos` bucket, we are sure our data is backed up. Awesome! 💪