ingester: support chunk transfers on ingester shutdown.

[rfratto]

This commit introduces chunk transfers, borrowing the mechanism from Cortex's implementation: when an ingester is shut down with claim on rollout enabled, the ingester will find a pending ingester and transfer all of its chunks to it.

When the ingester is being shut down, it is switch to a read-only mode where only Query and Label operations are supported. Push will return an error.

TODO:

• Add tests for transfers
• Clean up code

[rfratto]

This is ready for review; the only thing I'm not sure about here is my changes to logproto.proto - it feels weird to add a new service just for transferring chunks with four new messages types. WDYT @Kuqd @slim-bean?

[cyriltovena]

LGTM ! Great PR.

Quick Question: Will this cause error to be logged when running a single ingester ? or the Lifecycler know how many ingesters are in the replicaset and won't trigger transfer for those type of deployment ?

[rfratto]

thanks! if there's only one ingester in the ring the user would probably be running the ingesters with claim_on_rollout: false in the lifecycler config; as long as that's false, none of this code will be hit

[rfratto]

I'll merge this first thing in the morning

[cyriltovena]

If we have some doc on ingesters, see if you can add a quick paragraph about transfers that would be useful please.