Unable to set a TTL on a key

[omribahumi]

We've started working on flannel-io/flannel#233 (tl;dr - flannel uses etcd, we're adding a consul backend) and we've hit a dead-end.
According to #172, TTL should be supported with Sessions. However, the only reference to session Id we found in the KV PUT API was in acquire and release. We just want a simple Key TTL.

The line we're trying to port is:
https://github.com/coreos/flannel/blob/master/subnet/registry.go#L95

Also, flannel uses 24h TTLs, but it seems Consul session TTL is 10-3600 seconds.

[ryanuber]

@omribahumi by design Consul uses sessions to apply TTL's to keys in the K/V store. By creating a session, you get back a session ID, which you can pass in as the ?acquire value when setting a key. When creating the session, you can pass in the TTL and behavior for when the session expires (either release or delete). If you use delete as the behavior, then the key will be deleted when the session expires. Docs on the session create endpoint can be found here.

I realize its not as simple as "create a key with this TTL", but by tying key TTL's to sessions, we get some added benefits, like having multiple keys under the same session handle, and ephemeral keys.

Consul doesn't impose a cap on session lifetime. The 10-3600s is just the time range in which a session renewal must happen to keep the session alive. Basically a client must renew its session within the window if it wants to extend the session expiration. You can renew with the session renew endpoint.

Does that help?

[highlyunavailable]

@ryanuber There is actually a check to determine both min and max TTL.

There is no explicit cap on session lifetime, but there is a cap on TTL length.

[omribahumi]

@ryanuber I see. I missed the release/delete part.

What about having higher TTL values, for those cases you just want to "create a key with a TTL".
Also, would I be able to rewrite that key with a new session, without a prior release?
I'm thinking about that the app (flannel in this case) might be killed and lose its session. Will I have to wait for the TTL to expire prior to re-acquiring it?

Again, the use-case here is https://github.com/coreos/flannel/blob/master/subnet/registry.go#L95

[highlyunavailable]

You have 2 options there (I'm assuming you're using the Go API - it's pretty much directly translatable to HTTP if you're doing that):

1. From your new flannel process, you can call session.Destroy on the old session ID that has acquired the Key (which you can find by calling kv.Get on your key). This will cause the session behavior to trigger - e.g. if the behavior is set to delete, the key will be deleted. This is the same as the session expiring normally.
2. You can call kv.Release on your key from your new flannel process. This will disassociate the session from the key. You may then re-associate a new session. The old session will expire normally due to TTL timeout and will only affect keys it is still associated to.

Also, session->key association has nothing to do with writing data to the key, technically. You can write to a key from any node no matter what session is associated with it with kv.Put. There is no "lock" that is taken out on the key by associating a session. However, the caveat is that kv.Put won't update the session. To do that, you use kv.Acquire. If you call kv.Acquire on a key that already has a session, it will return false if there is a session already associated with that key. On the other hand, if the key does not exist, it will create the key and also associate the session in one step.

[omribahumi]

@highlyunavailable thanks!

I think this should be pointed out in the documentation.

As for the options you suggested, 1 is problematic because, theoretically, the address pool might be re-assigned to another node during the time it's deleted.
About the session TTL, having it limited to 1h limits the maximum node downtime possible. The current etcd implementation uses 24h, which makes sense in this scenario.

Why was the 1h value chosen?

[highlyunavailable]

You might be able to resurrect the session by ID as long as it's on the same node. I'm not sure what the Consul authors think of that, but I know that you can send an arbitrary session ID to the renew endpoint, so as long as you recover the session ID somehow (e.g. call client.Session.Node(localNodeName) and compare the list with the session ID already attached to a key) it shouldn't be a big deal to take over renewal again.

[omribahumi]

No need for that. When a node tries to write to a key for the first time, it would pass ?cas=0.

There's still a problem with the short TTL

[omribahumi]

Closing this one. Opening a longer TTL case instead.

[omribahumi]

Reopened as #1104

[kyrias]

Unless #6828 is fixed there is still a need for being able to set a TTL on a key independent of sessions, since due to that regression it's currently impossible to do this cross-DC.

[naughtyGitCat]

setting TTL on the key without a session needed will be more and more convenient, maybe a little more work with the hashcorp consul developers, but less ugly noodle code we will have