document startup procedure #1186
Conversation
Dieterbe
requested review from
replay,
DanCech,
woodsaj,
jimnagy and
robert-milan
docs/startup.md
Outdated
| | start input plugin(s) | starts backfill (kafka) or listening (carbon, prometheus) and maintain priority based on input lag | if backfilling: above-normal CPU and RAM usage | | ||
| | mark ready state | immediately (primary) or after warmup period (secondary) (combined with priority for clustering) | no | | ||
|
|
||
| The CPU increase during the kafka backfilling is very significant. typically 10x compared to normal usage. |
There was a problem hiding this comment.
I think typically 10x is a statement which includes many assumptions. In our case it might be 10x but in other cases that number could be completely different. I'd rather articulate that in a way which does not make the reader assume that in their own environment it will probably also be 10x. Something like
It can easily be 10x of the normal usage.
There was a problem hiding this comment.
our sizing recommendations (admittedly, not well documented) account for being able to replay a +- 7hr backlog in 30minutes, so 10x to 15x is expected for a properly set up ("typical") cluster. could be higher if the cluster is overprovisioned or lightly loaded, maybe i can clarify that.
docs/startup.md
Outdated
| | create Input(s) | open connections (kafka) or listening sockets (carbon, prometheus) | no | | ||
| | start cluster | starts gossip, joins cluster | no | | ||
| | create Index | creates instance and starts write queues | minor RAM increase ~ queue size | | ||
| | start API server | opens listening socket and starts handling requests | no | |
There was a problem hiding this comment.
This is a little confusing because it sounds like MT will already handle queries, but it loads a lot of data after this step. Maybe it should be mentioned that even though the sockets are listening, it will reject queries at this point until it is ready.
FYI to on call people.