Support keys with TTL

[robertmircea]

1. Please add ability to set TTL on keys for the k/v store subsystem. After expiration, the keys should not be available any more.
2. Allow to unset TTL for a key in order to prevent expiration using HTTP API
3. Allow touching the key in order to reset TTL period to its initial value
4. Allow changing TTL for a key to an another value before expiration.

[Telmo]

This addition would be very welcome in a situation where you have a number of containers coming up and registering their IPs to consul so there is some smart routing done. if they system doesn't re-register its ip by the time that the TTL hits the ip is out of the routing logic.

[awheeler]

The lack of this feature is the only thing giving me pause about replacing etcd.
If this is inconsistent with the thinking behind consul, can someone educate us on the correct methodology?
If not, I may take a stab at it.

[armon]

This is something we want to add. So far, my current thinking is actually to add TTL support to Sessions, not Keys directly. This enables a single TTL to apply to many keys, and has a broader set of uses. Additionally, Session behavior can be modified to support ephemeral keys. Stay tuned...

[awheeler]

Excellent news! Is this likely to be part of the 0.4 release?

[armon]

We've locked down the 0.4RC, so likely in 0.5.

[rboyer]

If you add TTL support to Sessions would it work similarly to "session leases" as outlined in the Google Chubby paper (section 2.8)?

[armon]

@naelyn Yes exactly.

[rboyer]

I had some free time and started trying to figure out how to modify Consul to implement Session TTLs in the manner of chubby leased sessions.

It wasn't clear if it was better to make the agent maintain the session (similar to how Checks appear to be plumbed) or to introduce a priority queue of soon-to-be-expired sessions at the Leader.

Doing it at the agents can reuse more of the existing plumbing, but I'm not sure it's semantically correct to defer KeepAlives to the agent in this situation rather than requiring clients to directly issue them.

On the other hand doing it with a straight priority queue at the leader requires that all sessions exist directly in main memory (without modifications to MDBIndex for time-indexing), and that's one more thing for the leader to be tracking and adjusting.

[armon]

So I think the answer in a way is "both". We already have agent-based TTL checks, which in effect are the same thing. The agent maintains the TTL and updates the check status with the servers. However, with this ticket I really meant support for server side TTLs. This allows a session to be created not associated with any agent. As you mentioned, it does mean all sessions will be tracked in-memory. Luckily, I think we can "compress" the amount of state we maintain to be fairly low, and even with thousands of sessions, it will be in the low kilobytes of memory usage.

[armon]

First pass at design: https://docs.google.com/document/d/1Y5-pahLkUaA7Kz4SBU_mehKiyt9yaaUGcBTMZR7lToY/edit?usp=sharing

[JeanMertz]

👍 to this feature. Looking forward to the implementation.

[amalaviy]

Merged 29afa88

[armon]

Closed by @amalaviy via #524