add the mt-index-prune utility

[replay]

only did some basic testing with this in a local setup, will do more testing

will also need to add docs for it

fixes #1069

[woodsaj]

looks good to me. the docs need to be updated so that the build checks pass.

[replay]

Added a few more minor changes, made tests pass, and did some more testing in my local env. Seems to all be working fine as fas as i can tell.
Please take another look and approve if you think it's ok to merge @woodsaj

[shanson7]

Would it be possible to do just 1 partition, or a range? It looks like this would take quite a long time to run sequentially on our data, so there might be utility in doing a partition at a time so we can stop/continue easily.

[replay]

@shanson7 definitely, added that

[Dieterbe]

i didn't check this in depth (or think about it much because i'm on vacation \o/ ), but running this as a tool seems harder to operate (requires setting up extra jobs etc)
wouldn't it be simpler for MT to, upon loading the index, do this automatically and/or a background routine within MT periodically scan for stuff that can be moved? that way there's no additional orchestration overhead.

[woodsaj]

wouldn't it be simpler for MT to, upon loading the index, do this automatically and/or a background routine

A background routine would be a really bad idea, for large instances a huge amount of ram is needed the data from cassandra.
Doing this at startup is an interesting idea, but to get a real benefit you would have to restart the MT instances regularly. Additionally it would increase the startup time of MT and the main goal of the archiving is to reduce startup time.

Doing the archiving out-of-band is definitely the most efficient approach and solves the problems we are trying to address.

Perhaps we could make archiving at startup an optional thing, but doing so is outside the scope of this PR.

[shanson7]

Why would it require any more RAM than MT is already using? MT already has exactly what time-series are still alive in the memory index.

I will point out, though, that doing it within MT would be harder for me. We run 1 set of write instances which prune the index aggressively (different set of index rules) so they wouldn't be able to do this. Our read instances are replicated, meaning they would duplicate the effort if they both tried to do the same partitions.

Just my 2 cents