Implement rate limiting

[tomwilkie]

Users will pay for samples/s (indirectly), so we should hold them to this.

We should also limit the cardinality of metrics.

[tomwilkie]

https://github.com/youtube/doorman

[rade]

We should also limit the cardinality of metrics.

#47.

[juliusv]

@tomwilkie Doorman is about voluntary client-side load-balancing between a client process and a server, but I guess we could repurpose it by registering each Cortex user as a resource with it (because we want a global max sample ingestion rate per user), and then have the distributors request capacity with Doorman for every user they get samples for, so every distributor would get a proportional share of each user's total allowed rate. Does that work? Do we want to do that? :)

[tomwilkie]

Yeah thats what I was thinking too. Yes we want to do it, not sure its the highest priority right now.

[juliusv]

Ok, cool. I would have assumed fairly high priority, given that it's a trivial (potentially even accidental) DoS opportunity against Cortex right now.

[tomwilkie]

Yeah, hence "not sure".

[juliusv]

So Doorman needs etcd. But we already have Consul. Options:

• a) Run both etcd and consul.
• b) Run only etcd and make Cortex's ring use etcd as well.
• c) Build our own distributed rate-limiting.

I don't like any of these options much, although if we do a), we could still decide later to do b).

Or when building our own solution, we could keep it simple: keep the state in each distributor and gossip state between the distributors (e.g. for each user, what's the current rate in each distributor). Then each distributor can determine the global rate for a user by adding up all distributor's rates and throttle accordingly?