New blog about Thanos adoption at Aiven #6461
Conversation
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
| * Thanos with six months retention: $19,703 | ||
| * Thanos with a years retention: $22,447 | ||
| * Thanos with 2 years retention: $27,955 | ||
| * Thanos with 3 years retention: $33,423 |
There was a problem hiding this comment.
Hm, Is your obj storage cost alone ~18k USD for 3y data? Sounds bit off and expensive 🤔
There was a problem hiding this comment.
We have a lot of metrics and manage a large fleet. We also use Thanos (m3) for our billing data which is quite detailed at the service level. This is really the costs, but look how it compares to the cost of M3.
|
|
||
| We are also paying roughly 25% for the storage costs. M3DB has a total of 54TB of storage provisioned today, at a cost of $4320 per month. We could house 216TB of storage for the same cost with Thanos. We are currently generating about 750GB per day, which means we can keep almost a year of metrics for the same cost as M3DB. Additionally, we are backing up M3DB which is using 33TB of object storage at a cost of $1,320 per month. With object storage, we have the added cost for the networking, this is around $1800 per month in additional costs. Here are our estimated costs: | ||
|
|
||
| * M3: $38480 |
There was a problem hiding this comment.
I will add details on that. Thanks.
|
|
||
| ## Performance Gains | ||
|
|
||
| The performance normally was much better in our ongoing reporting and alerting needs as well. Today we are using vmalert to drive our alerting pipeline, since M3DB is limited, there is no such thing as an alertmanager integration. This brings me to another issue we found with vmalert it would sometimes execute rules twice within the same group evaluation period. This, by default, would realign the result timestamp with the group evaluation start time, which would lead to failed and rejected writes. The timestamp issue was caused by samples with same timestamp but different value ), this was fixed by disabling this query time alignment ( datasource.queryTimeAlignment ). |
There was a problem hiding this comment.
Why vmalert? Any specific reason why not Thanos Ruler?
There was a problem hiding this comment.
M3DB doesn't have a similar capability, so we are using vmalert. It was easier than switching everything over. We are happy with vmalert right now even though it's pull vs push.
There was a problem hiding this comment.
Awesome! 🚀
LGTM mod small nit about the diagram and @bwplotka's comments.
Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Jonah Kowall <jkowall@kowall.net>
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
|
Not sure why the docs is giving an error on outstanding commits, can someone assist on it? (@saswatamcode) |
|
Yes, @jkowall! Could you run |
|
Looks good now, thanks @saswatamcode ! |
|
Still can't merge it due to a CI failure, any advice would be helpful. |
This is a new blog I have written with two other co-authors here at Aiven on our adoption of Thanos at Aiven. I think it's a great contribution, and we hope to keep doing so upstream and pushing Thanos forward in the community. I would appreciate any reviews on the blog.
Thanks!