Change TTL resolution to nanoseconds and allow subsecond TTLs

[deepakjois]

Currently, Badger allows setting TTL values on key-value pairs. We store these TTL values as unix timestamps with a per-second granularity. This was probably an oversight, but it prevents setting sub-second TTL values. Changing the way timestamps are stored is a backwards incompatible change.

I am creating this issue to track feedback and comments on how urgently this change is required, and whether it is useful. We would also need to put some thought into how we backup/restore a Badger database that stores the timestamps currently in seconds, so that it works with a version of Badger that stores the timestamps in nanoseconds.

[brunotm]

+1

[magik6k]

What I see could be done to avoid breakage would be allowing the user to interact with those values more directly - Add getTime field to options use it's return as 'time' when possible, fall back to seconds when nil.

This would allow subsecond resolution with the side effect of enabling custom time definitions - like 'n app launches', 'n inserts' , etc

[deepakjois]

Add getTime field to options use it's return as 'time' when possible, fall back to seconds when nil.

Interesting idea. Let me try it out.

[deepakjois]

Pls see #342 and let me know what you think.

[manishrjain]

Let's postpone this until we're ready for v2.0 launch. For now, second level TTL should suffice. We've tied into it for this version anyway.

[deepakjois]

We thought more about this, and there are some inconvenient tradeoffs to think about:

• Like we have already discussed, to implement this we need to change the way we store timestamps on disk. It will break any existing Badger DBs that store TTL values. The only way out for those using TTLs now would be to backup and restore using a special version of our backup tool that would rewrite timestamps.
• Strictly speaking, a change in the on-disk format means we should be increment the Badger manifest version as well. This would be very disruptive for all existing Badger users. Given 1.0 just came out, this is not something we want to do in a hurry.

For now we think it is better to leave things as they are. We would like to hear more about real-world use cases for a sub-second TTL. If there is a convincing case for us to make such a disruptive change in the future, we could do it. Otherwise, we could just wait and slip these changes in, in the future when we are compelled to upgrade the Badger manifest version.

[jiminoc]

I wouldn't expect badger to support this. #1 just because something TTL's just means it's not available for viewing vs actually being deleted from disk (that's compaction's job). It would seem more appropriate to put a field in your stored data type or wrap your data with a TS field you can filter on yourself. Less complexity in Badger = better IMO

[JeanMertz]

I was looking for this as well.

In my case, this isn't needed for our production use-case, but since we have a wrapper around the database calls, we wanted to add sufficient tests to make sure what we're doing is actually working as expected. So, we write tests. We saw random failures, and were unsure what was causing this, until we noticed that Badger doesn't support sub-second TTLs. We fixed this for now by slowing down the tests that validate TTL behaviour, meaning those tests take multiple seconds to run.

So; for us it's less about the requirement for production, and more about the improvement in testing speed if this went in. (I saw this was already merged in the 2.0 branch, so that's great!)

[manishrjain]

Hey guys, I decided to not move to subsecond level TTL for the latest v1.5.0 release. The only thing which was a data level breaking change would have been this sub-second feature for TTLs, which would require us to move to 2.0. I'm not convinced there's a lot of need for it out there. Second level TTL should be sufficient for most users, I reckon.

A lot of changes have gone into v1.5.0, which benefits data reclaim, both at value log GC level and at LSM tree level.

But, there's a clear use case for sub-second TTLs, do bring it up. So, we can decide if this needs to go in as 2.0.