Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New blog about Thanos adoption at Aiven #6461

Merged
merged 29 commits into from
Jun 27, 2023
Merged
Changes from 1 commit
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
63e696c
Add aiven as adopter... more soon!
jkowall Jun 8, 2023
adcdcb7
First draft of the Aiven adoption story
jkowall Jun 8, 2023
c5d4ea4
Adding graphic for blog showing architecture
jkowall Jun 17, 2023
6ba1b1e
Fixing mermaid graph
jkowall Jun 17, 2023
6bcde9a
Fixing mermaid graph again
jkowall Jun 17, 2023
13e1650
Remove image and fix mermaid for other flowchart
jkowall Jun 17, 2023
9c09ba6
Fixing prod flowchart
jkowall Jun 17, 2023
42ee068
Fixing prod flowchart title
jkowall Jun 17, 2023
af82a8b
Fixing md table
jkowall Jun 17, 2023
bab3f8d
Fixing md table again
jkowall Jun 17, 2023
4537b6d
Adding clarification on cost breakdowns
jkowall Jun 17, 2023
90624ad
Adding clarification on cost breakdowns
jkowall Jun 17, 2023
f0c5aef
Some corrections on text and links
jkowall Jun 19, 2023
ade2584
Fixing links
jkowall Jun 19, 2023
720a2ff
Fixing links
jkowall Jun 19, 2023
926b8b7
Fixing links
jkowall Jun 19, 2023
b594ecf
Fixing links
jkowall Jun 19, 2023
e67ea64
Fixing links
jkowall Jun 19, 2023
09c0084
Fixing diagams
jkowall Jun 19, 2023
0114706
Fixing currencies
jkowall Jun 19, 2023
ef8f2d9
Fixing diagams
jkowall Jun 19, 2023
0c333c1
Fixing images
jkowall Jun 19, 2023
3075c85
Merge branch 'thanos-io:main' into aiven-blog
jkowall Jun 20, 2023
c1311c2
Update docs/blog/2023-06-08-thanos-at-aiven.md
jkowall Jun 23, 2023
723f75e
Merge branch 'thanos-io:main' into aiven-blog
jkowall Jun 23, 2023
a60b7d8
Add retention for M3
jkowall Jun 23, 2023
dd7f34f
Add image versus mermaid diagram. Couldn't comment it out unfortunately
jkowall Jun 23, 2023
88f1f30
Adding text around link to fix linter
jkowall Jun 23, 2023
9b514ed
make docs
jkowall Jun 24, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fixing diagams
Signed-off-by: Jonah Kowall <jkowall@kowall.net>
jkowall committed Jun 19, 2023

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
commit 09c00848ca2911eb6ff1269f59af29a16dcad2a8
9 changes: 6 additions & 3 deletions docs/blog/2023-06-08-thanos-at-aiven.md
Original file line number Diff line number Diff line change
@@ -47,10 +47,9 @@ flowchart
907649 --> 4
209978 --> 141231

%% Mermaid Flow Diagram Link
%% Keep this link to make future edits to your diagram
%% https://www.mermaidflow.app/flowchart#N4IgZgNg9g7iBcoB2UAmBTAzgg2qGAlqgC4AWCATAAxUA0Ip6BA5qcQgIxUAs9RCIDiHoAHKJgLECUJAlAAPBNyoA6DgDYK3AMwcKegKwBOABwUD9AJ6d9Ko9u1Gj6p1wDsB9W4C+9VAENifzkQCH8AI3QIAQBhAFdMYigAW3QAJwACABFAiP9MLAAecLSAPmEQNP8kAGtMLKqYWURfEBF-NPQkYgAlapqEOhBURoBlYksIdAQwfwgC+kTJrFwQRLSoGvQAWkISUngMihFFAF1WiZFp+GHc8Pz0ADk0adFxSWkkAEFwzCgIOLEa4KJSqDRaXT6cymcxWGwcOwOJwuIzuTw+RZRdAAYyBqBmcwWwyqzGYBCQzAJ83QvnwRDInG06nojBYbEZzJA-BuFAqYgkUhkIUU8Ao6gRbg4RgMFFl3HsOjhooMCPU3GM2h46ioTLcVBMrQCQRCYUi0RuAGl-GAasF6FVavVGs1gK12p1un1aoM-GMJlMEMQ0nEafRLtdhlA4uEpjECGlsVNnhg+e9Bd9fv9AcCQCKxRKpTK5QrtErzKr1fYtTqvPrWgUprj0Pj4LNqb7-KTyZTW4TQ3T9pQaCymKx2PAuLwuS2QNpUwLPsKEAYqG4VMY3OoTBxtFCDCYTEqdxQVFoTC4dGKVyYdYbciaIlEBAAVUjVcQZHpRqQUz84gjoAAbukxRlBUDp1A0-hNHIbodF0vT9D6xLQeMyyBsGoYgOGAgiAQEAQMmrxtGmnw-H8AJAku8ArmuG5bsehgHkeu6ntw55GJe6jXremKNniVJEiMnZkhSgn9iAewMqKw4MKO7ITjwfAztw84fEKiC5suq4qG43BSpoYpeNoBhTtY8C7twKguAYJm1m4N4mAYd7GppppPjcr7vpgGQAJIUlgP7MH+2JMMBaSgeU9r9E60EunBHqId68BDMJMBoQG8BBiGFyWFcuH4YRLxqem5FZlRmkirRun6c4srqMZplKpZ1nSnZeoOfqzl8TiAm9u2KFdmJ-UFLSkn0uQMlDKyY6cEp04CBw+kULoJWLpV2lGHYYqbvq0oUG4jhKtwHgqLoMpGFQ+76tqzgucEbmPuaIAAOJQFAzBTBkMTQHEqAZOMUAkugGQABQAPLhAAVr1ANJJ0ACUkUVEsUzYPAOCnNFjpQTBLSiPBnpISlHbpf61zZVhOE3EaeQFERKNYk2LZtkS-LqRmFHZtRK5bUYO3aiY+2HUYx2nedFCXdd55UHdHZDT2rMSVJk3UIeclsuOHB6Mpi07nqvJvAuGkgqK7FneY+gqrL3DcIZYuqFopn2Bw26otr90PmaL5vigPkxCk7S4kDyOLOT6OY9jkHOrBBOJV6AwkyhZPoVlmG5flNydLi1SfU8xU9cz4lGxzZWUTmebm7uMocNbnF27KDtsc7uhu9rHCGiSomK32Y0q0O03yVr83coIng0HOJfptRenqGobgu8454mauR62Lb2iXju+nKFunuPd7nm+x+ACKIZpJYGQAGIbN0XSoKHazh7gWOVDFuPxXHCEJ8haUZRT6cwx5QjHhAiDNC59SVlPMimZy4zztvPReW51ArzcGvBEG8t6byWlQPe8tu7iT7hNAeI5NZzSnKPYwmgvBrRNlpeATgtoNVRELFB2gnLbjXiedUIsNA7irBiW4rlQDuWei9KoswkD+EfqjFYGNX4QVinjV0X8ibJVSn6VOlMM4RkYPITsMhwFrCZpAvs0CZBl25htBhThrIuFds4BwHCOBcJUDwxwfDHCakEcJBWhDaADmkmrUhs0Jw6wWjcbcygNC0JdPQrgNBrImF0PKbWTJ2IuJAOZA6VkDAql0CZMwOgVT7xEU9AQAA1AAsl8KYaR2Bh2WBHBR78Y74zaITJKicNGoXJhhHKQDM6VF6rnJMBdjH8WbMXEixtOblQrnNRJW4Unu3SZwrJlA9LrnyTufcEISn4O7P4wJqt9QhIUtrCgutIl6DlLE6iCTVDLKWqstUyTjqeBUFdGg2o7bsRXNwUpoRyk3FqekYgyRqidnSCjZ+8io5KM-h0+OxMekp0ytowZEZs5BApGMlMECpkjXQIc4aUCZml1gVY02jykkrLSW80sGz4Dqjnt83BWp-k8G8EQwcMl1YzQueE0el1NzynudY2eiDdBLxQbRZqHA1z63FIqhw4o1RAtET7byGQz7pEsDIuFkc344zaSo5F39UWk3-v0qmwCBA4tGfnAlEzepEvJezUqVKKqmylYqpBy95XMt0Eq7Qm5taHSZBoQFpKe7Ul5UEs5GtQmXOuSAagTgHISt9eqNQ3EmRUGoNxbUnJzJ2zXNQKNJ1uDmGUJuTVIKQBeT9nDIG6BDVNJfgij+scLVqO6davpacBnYXtVnEZeLnXEQbG6lmZjBoEOJeYuZcDJW5o0LZbURbx6lqUF4U8NYlp6VraudQPLX7NmYHIvAo6hnYgSEkZIFRIVpC2GkcYHRxwgCgFcWQ9AX1voAKJIBnGIckDSuR3yqLiT4AB1YhMlGmZVAOsTY6B4N8o7mmy9Oxa6XX5lKLQV0UZRgTBGIQYYOhXq-YbNYpHQoAAlqioEyoICoQQ0jUaYyB1jFGhEPTKYfJtWJmASIqAQTAgHUAfDJb3egAAvfyGARQJMJXO+NATb0RnvYkFIz6Ohvo-fUgQP6uj6dfekYDoGoDgfE1B-wMGZAYcTUhnMqGtjOcmpk6TmARBhHMpBiA5JpitFHjh7YHFZQ0ElpxIwJG4hkYELRjj1GBCTzowlxjzHePsao+gYg3GWPkYqNUAgkLTEDVpiECTUmZNxqJFqzyImxOtEUyB9AKnZIzqLsSsa1MQA6cfeZwzHGv2mb-SAADlmQO4Rs90OzQJoPpk80OVzIR3PoYQxQULM5wsqh3LoGgZhLrxcSzcdLKX8sCFUosej6BCs5co5x-LD3it+HvJpGr0mgrTMayAL4AAtbY0F4IZC2EESFGQ3yYFIGkI5rWlMdcWWp45WmBCDb0-+gz6QjNjd-cN6b1nbN8Hs45pAK3ENP1TihoMaGKdYYy2dwQ+tC2nayzxiMyW8tfqWnoVaT2uPZc5+J3bqAr3bDyXkpa2pHA6hK0gMrgR3Xzqq5pWRqwNvwAAMThA4Lr8I6gQDnAU4jzrQxusVdGpp-rGOn1Y4s++0bJn8f26AzNm4YH5sk8Ww55bW3zdDppxsDzCGGd-EyxGG7jP2dFaS7l57PPlr8+wtz17i0RcCHCyYQ6kpzpmGSfLxXluSX8fW0OnAT9g-oG1-rvXhvjcgDa8p5HrqetKz62OgbD7MeTex47z9zuzOu8J7N4nkGfdk4p9QNbavach8w-WO7AgqE7TZ-doX6eBdXZuLXbUcut8FY3zyDPNw9vynPDWsNtd0ulfK8ryrH2aepwr5rnXeu69G4R+1s3KPevW877bgTv3sZjcONkAVZqPl7uPukL7nBv7jPkHnTqHovhHgIFElQDErdhHmnjvvHqljvuPPvingnjgemifiAOFlKOdAvLnueIXnfupkJI-lTplC-nPtXm-vrvXl-s3opObiYvflbqADbt3nbr3g7rjoPhNlNmkBAR7nNhBuBjAZPvASwW5uwfTigUzugZgdHuvhznHgfmgbcloHgS9kfmxthmLjsBeGqLgheOwvQUrowSXqrk-qwZXmhjXu-gbp-ibt-i3hboIRJCIbpmITIZIaAS7uIW7kTlAUoUtnAXytPmoWXlXpoVgUzrvhPGvqQVzgngIKKmqHFgfqQXxmFtYdsMoEyOwjbDqFQE4cXu9sIqkRjJ4VsN4VwX4Y3qboEQIS4R3neqIUAZEd+tETIXISRGPgkbAU5qoerrPukcgZkaFIUaeuKisfobHsfkYTyLLEYFmqURYeUaLuLsktxHzsZEyI0cEc0QJq0WwVXp0R-g3k3kjnwb-u3v-kMWESMU7lEUPjESPvIdMaTn7skQHtTu0ZtgvpsWsWKiUXoWUWYV+lHpdofgYbgVYeLk4LbKYIdjqHxrfs4dMm4Q8dCc8b4a8b0R8a3sXoMdpsMcPsAXjoCRMe7lMfEWCUkS5q0YgfPtJGHkvnsZmiYLkccSiYtEnnxuicidiTsGKNCGYCuIRjcS4XcWXs-hSZwS8Twe8apnScEd4K-IBABDAGICAabNsK7AiJdGqOxKkhQCYDWkeA5KeE6SqKZMkrLm4Ggo3u9E+ilHYOeCtAdDwPKA4PuAaIaAQJ2FUMkFUvlv4DkC0dJjiutE2gAEIVBgC3zEBVKBCLYhBSDECsZfAEDARIAZDNofjxBpCJQZDIwxnpnATNL0DYgpCpDdBtkDZhCYAFDNKhaSbyAWnEAAAK1QUQ4M+Ow6Jen0UA9wEAUmV6z4Y6oAqAcQcwv2wWq5QyFE-A9AOGBZDuu5ICChFQHZ0AaQAgOpvhrQ85i5REp5xZnejqk6Ril5QMN5NAP5DR9A9w2INQomUYIGAcV5N5YAkFUFFQ4QQMGAaQYFX5NwWuv5NAMFcF6Qr4BAgFSAWA6MdGQW+IfA9Q+QjAAx3gQAA
%% Mermaid Flow Diagram
```
This is the high level overview of our current Thanos architecture.

As you can see in our architecture, we are using Telegraf since it supports monitoring the many technologies which Aiven provides in a smaller footprint. Although we support Prometheus scraping for our users, internally we push metrics to M3DB via influx line protocol. We use additional technologies together, as you can also see. We had several options where we could introduce Thanos into the mix. Furthermore, we decided to continue using Telegraf, but sending directly via remote write, versus using the Influx protocol. This created one of our first challenges, which is that some of our metrics are delayed due to the number of clouds we support. When a metric is written in the future, the ingesters would crash. It just so happened someone was just fixing this upstream (https://github.com/thanos-io/thanos/pull/6195).

@@ -65,6 +64,9 @@ We take great care to manage the hash ring in a way that ensures no failed write
The community has been critical for us. One example we ran into around this portion of the implementation which we received some help from the other users and maintainers on the CNCF Slack. We also ran into some other issues which would create replication loops causing a crash as well. We addressed this issue by moving to a routing-ingesting receiver topology as suggested by the community.

```mermaid
---
title: "Aiven Thanos Scale Out Architecture"
---
flowchart TD
subgraph ThanosQueryCluster
Query1
@@ -115,6 +117,7 @@ The community has been critical for us. One example we ran into around this port
Store3
end
```
This is how we would scale out the components in our architecture outlined above.

## Cost Savings