More granular timeouts

[Suor]

Redis supports millisecond timeouts on SET with PX, but BLPOP understands only integer seconds. This is still achievable by using publish/subscribe with a bit more hustle.

[ionelmc]

Do you need this feature for cacheops?

[Suor]

Yes. I plan to expose this timeout as user setting. It also makes sense to set it to same time as calculation will take, which is usually under second in web apps.

[ionelmc]

Just to understand this correctly, I wonder - wouldn't that create a smaller dogpile effect if the database and management overhead (query serialization, deserialization and cache storage) takes more than 1 second?

If we use pubsub then all the clients would be woken up, even if they can't get the lock.

On the other hand, this slightly less efficient technique could be active only if the timeout is a fraction.

[Suor]

Look if I set expire to 0.5 second then I am still sure that calculation is made no more than twice a second, which should be enough to mitigate dogplie. OTOH, in normal operation things are unlocked as soon as possible, so no need to make timeout that smaller. So for cacheops this should be ok current way.

[ionelmc]

Now I'm not sure if you still want this feature. Should I take this as a "not for now"?

[Suor]

I don't, at least for cacheops. Probably, you do? )

[jhorman]

Sorry to hijack a thread for a question, but here goes. From what I understand, when Redis runs low on memory, it will randomly sample keys, and evict the ones with the shortest remaining ttl. That means that you cannot rely on your expire ttl to be accurate when churning the cache. That means under memory constraints, locks might disappear if using expires. It might be safer to not use redis expires, and maintain your own expirations.

Or, I have this all wrong.

[zoni]

@jhorman (and anyone else coming by this issue), what Redis does when running out of memory depends on the specific eviction policy you have configured.

For this use-case (distributed locking, especially with low-TTL keys), setting a noeviction policy might be wise.

[dimaqq]

Please document noeviction requirement.