Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Updated detailed steps for migrating blocks from Thanos to Mimir. #3290

Merged
merged 10 commits into from
Nov 4, 2022
Merged

Updated detailed steps for migrating blocks from Thanos to Mimir. #3290

merged 10 commits into from
Nov 4, 2022

Conversation

Leegin-darknight
Copy link
Contributor

@Leegin-darknight Leegin-darknight commented Oct 24, 2022 •
edited
Loading

The detailed steps on how to migrate the blocks from Thanos to Mimir isn't available and the current documentation also doesn't seems to be helpful. Therefor I have formulated the steps I have followed for migrating the blocks from Thanos to Mimir so that it would be helpful for people who are looking to move from Thanos.

What this PR does

Which issue(s) this PR fixes or relates to

Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@Leegin-darknight
Updated detailed steps for migrating blocks from Thanos to Mimir.
d526622 
The detailed steps on how to migrate the blocks from Thanos to Mimir isn't available and the current documentation also doesn't seems to be helpful. Therefor I have formulated the steps I have followed for migrating the blocks from Thanos to Mimir so that it would be helpful for people who are looking to move from Thanos.
@Leegin-darknight Leegin-darknight requested review from osg-grafana and a team as code owners October 24, 2022 08:38
@CLAassistant
Copy link

CLAassistant commented Oct 24, 2022 •
edited
Loading

CLA assistant check
All committers have signed the CLA.

@osg-grafana
Copy link
Contributor

Relates to #3288

@osg-grafana osg-grafana added the type/docs Improvements or additions to documentation label Oct 24, 2022
@osg-grafana
Copy link
Contributor

@Leegin-darknight, thank you for the contribution. I am going to restructure is a bit to conform to common style.

pstibrany
pstibrany reviewed Oct 24, 2022
View reviewed changes
Copy link
Member

@pstibrany pstibrany left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you very much for working on this. I focused on technical accuracy and use of the scripts in my review, and left some comments.

docs/sources/migration-guide/migrating-from-thanos-or-prometheus.md Outdated Show resolved Hide resolved
docs/sources/migration-guide/migrating-from-thanos-or-prometheus.md Outdated Show resolved Hide resolved
docs/sources/migration-guide/migrating-from-thanos-or-prometheus.md Outdated Show resolved Hide resolved
docs/sources/migration-guide/migrating-from-thanos-or-prometheus.md Outdated Show resolved Hide resolved
docs/sources/migration-guide/migrating-from-thanos-or-prometheus.md Outdated Show resolved Hide resolved
docs/sources/migration-guide/migrating-from-thanos-or-prometheus.md Outdated Show resolved Hide resolved
docs/sources/migration-guide/migrating-from-thanos-or-prometheus.md Outdated Show resolved Hide resolved
@Leegin-darknight Leegin-darknight requested review from osg-grafana and pstibrany and removed request for osg-grafana October 24, 2022 16:51
@Leegin-darknight Leegin-darknight requested review from osg-grafana and pstibrany and removed request for pstibrany and osg-grafana October 25, 2022 07:26
@Leegin-darknight
Updated the documentation with the changes suggested.
719f180 
Updated the documentation with the changes suggested with the for loop and also included steps to remove the external labels from the meta.json as well.
pstibrany
pstibrany reviewed Oct 31, 2022
View reviewed changes
docs/sources/migration-guide/migrating-from-thanos-or-prometheus.md Outdated Show resolved Hide resolved
docs/sources/migration-guide/migrating-from-thanos-or-prometheus.md Outdated Show resolved Hide resolved
docs/sources/migration-guide/migrating-from-thanos-or-prometheus.md Outdated Show resolved Hide resolved
docs/sources/migration-guide/migrating-from-thanos-or-prometheus.md Outdated Show resolved Hide resolved
docs/sources/migration-guide/migrating-from-thanos-or-prometheus.md Outdated Show resolved Hide resolved
docs/sources/migration-guide/migrating-from-thanos-or-prometheus.md Outdated Show resolved Hide resolved
docs/sources/migration-guide/migrating-from-thanos-or-prometheus.md Outdated Show resolved Hide resolved
docs/sources/migration-guide/migrating-from-thanos-or-prometheus.md Outdated Show resolved Hide resolved
docs/sources/migration-guide/migrating-from-thanos-or-prometheus.md Outdated Show resolved Hide resolved
docs/sources/migration-guide/migrating-from-thanos-or-prometheus.md Outdated Show resolved Hide resolved
@Leegin-darknight
Updated with the changes suggested
92a7d9f 
Updated the script to remove labels from meta.json, retention of the downsampled blocks and other changes suggested.
pstibrany
pstibrany approved these changes Nov 1, 2022
View reviewed changes
Copy link
Member

@pstibrany pstibrany left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you very much, this looks great!

@osg-grafana this is ready for "docs squad" review.

docs/sources/migration-guide/migrating-from-thanos-or-prometheus.md Outdated
@@ -88,6 +88,8 @@ This may cause that incorrect results are returned for the query.
1) Copy the blocks from Thanos's bucket to an Intermediate bucket

Create an intermediate object storage bucket(S3,GCS,etc.) in your cloud provider where you can copy the historical blocks and work on them before uploading to the Mimir bucket.

> **Note**: It is recommeneded to run all the above steps in a **screen** command to avoid any interruptions as these may take time depending on the amount of data to be processed.
Copy link
Member

@pstibrany pstibrany Nov 1, 2022 •
edited by osg-grafana
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
> **Note**: It is recommeneded to run all the above steps in a **screen** command to avoid any interruptions as these may take time depending on the amount of data to be processed.
> **Tip:** Run the commands within a `screen` or `tmux` session, to avoid any interruptions because the steps might take time depending on the amount of data being processed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pstibrany and @Leegin-darknight, I have updated this suggestion to a tip and tweaked the phrases.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@osg-grafana I have updated the PR with the changes suggested. Can you please review?

docs/sources/migration-guide/migrating-from-thanos-or-prometheus.md Outdated Show resolved Hide resolved
docs/sources/migration-guide/migrating-from-thanos-or-prometheus.md Outdated Show resolved Hide resolved
docs/sources/migration-guide/migrating-from-thanos-or-prometheus.md Outdated Show resolved Hide resolved
docs/sources/migration-guide/migrating-from-thanos-or-prometheus.md Outdated Show resolved Hide resolved
docs/sources/migration-guide/migrating-from-thanos-or-prometheus.md Outdated Show resolved Hide resolved
docs/sources/migration-guide/migrating-from-thanos-or-prometheus.md Outdated Show resolved Hide resolved
osg-grafana
osg-grafana approved these changes Nov 2, 2022
View reviewed changes
Copy link
Contributor

@osg-grafana osg-grafana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unblocking with several nits that conform to style and phrasing, as well as clarify some abiguity.

@pracucci
Copy link
Collaborator

pracucci commented Nov 3, 2022

Fixes #3120

pracucci
pracucci approved these changes Nov 3, 2022
View reviewed changes
Copy link
Collaborator

@pracucci pracucci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You really did a fantastic job! I would love to merge it asap. I just left a couple of last comments.

docs/sources/migration-guide/migrating-from-thanos-or-prometheus.md Outdated Show resolved Hide resolved
docs/sources/migration-guide/migrating-from-thanos-or-prometheus.md Outdated Show resolved Hide resolved
docs/sources/migration-guide/migrating-from-thanos-or-prometheus.md Outdated
Copy the blocks from the intermediate bucket to the Mimir bucket using the below gsutil command.

```bash
gsutil -m cp -r gs://<mimir-intermediate-bucket> gs://<mimir-gcs-bucket>/<TENANT>/
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mind showing the AWS command as well, like you did at the top of the section?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pracucci I have updated the doc with the command for AWS as well. Thanks.

@Leegin-darknight
Updated with the changes suggested.
bec8db1 
I have updated the PR with the AWS command to copy blocks and corrected the typos.
pracucci
pracucci approved these changes Nov 4, 2022
View reviewed changes
Copy link
Collaborator

@pracucci pracucci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@pracucci pracucci enabled auto-merge (squash) November 4, 2022 11:15
@pracucci
Reformatted markdown running 'make doc'
7d71c4d 
Signed-off-by: Marco Pracucci <marco@pracucci.com>
@pracucci pracucci merged commit 374877d into grafana:main Nov 4, 2022
masonmei pushed a commit to udmire/mimir that referenced this pull request Dec 16, 2022
@Leegin-darknight
Updated detailed steps for migrating blocks from Thanos to Mimir. (gr…
116693e 
…afana#3290)

* Updated detailed steps for migrating blocks from Thanos to Mimir.

The detailed steps on how to migrate the blocks from Thanos to Mimir isn't available and the current documentation also doesn't seems to be helpful. Therefor I have formulated the steps I have followed for migrating the blocks from Thanos to Mimir so that it would be helpful for people who are looking to move from Thanos.

* Updated the changes suggested after review

* Replaced xargs with for loop as suggested

* Updated the documentation with the changes suggested.

Updated the documentation with the changes suggested with the for loop and also included steps to remove the external labels from the meta.json as well.

* Updated with the changes suggested

* Updated with the changes suggested

Updated the script to remove labels from meta.json, retention of the downsampled blocks and other changes suggested.

* Updated with the changes suggested.

* Updated the changes suggested

* Updated with the changes suggested.

I have updated the PR with the AWS command to copy blocks and corrected the typos.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pstibrany pstibrany pstibrany approved these changes

@pracucci pracucci pracucci approved these changes

@osg-grafana osg-grafana osg-grafana approved these changes

Assignees
No one assigned
Labels
type/docs Improvements or additions to documentation
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@Leegin-darknight @CLAassistant @osg-grafana @pracucci @pstibrany