-
Notifications
You must be signed in to change notification settings - Fork 182
mv bucket expiry
- merged to master -
- code complete - November 18, 2016
- development started - 11, 2016
This branch is a partial implementation of Basho's expiry by bucket types for use within Riak. This branch contains the subset of the total design that is sufficient for an immediate customer need. Subsequent branches will complete the expiry by bucket type feature.
The complete feature is intended for Basho's enterprise edition products, not open source. Enterprise edition products required paid support. Therefore a substantial portion of this feature is isolated within the private leveldb_ee repository. Portions of the feature are also within Basho's eleveldb open source repository. A successful build requires the mv-bucket-expiry branch from all three repositories: basho/eleveldb, basho/leveldb, and basho/leveldb_ee. The term "this branch" refers to the collective set of code changes within all three repositories.
The key feature of this branch is to derive the expiry "write time" from data within a Riak Object that is being written to leveldb. The branch adds two write time sources: Riak Object's LastModTime field and an optional customer X-Riak-Meta-Expiry-Base-Seconds field. The priorities for write time sources are:
| Priority | Source | Where valid |
|---|---|---|
| 1 | X-Riak-Meta-Expiry-Base-Seconds | enterprise only |
| 2 | LastModTime | enterprise only |
| 3 | "current minute" | enterprise and open source |
The "write time" is used in conjunction with a user specified expiry minutes. leveldb considers an object expired (logically deleted) once the current time is past the "write time" plus the expiry minute setting.
X-Riak-Meta-Expiry-Base-Seconds is property that the user can set when the data being loaded needs to be "back dated" for the purpose of expiry. Example: loading 12 months of previous data to facilitate computations against new live data, but desire rolling 12 month expiry. The previous data needs this explicit property to facilitate properly timed expiry. The value of this property is seconds since "epoch", Jan 1, 1970. The valid range of dates is Jan 1, 1980 to Jan 1, 2080. Examples:
**X-Riak-Meta-Expiry-Base-Seconds: 1245494500** is June 9, 2009 at 7am
**X-Riak-Meta-Expiry-Base-Seconds: 1478342700** is November 5, 2016 at 6:45am
The "current minute" uses the current system clock, but is only updated roughly every 60 seconds. The expiry clock is not intended to be precise and overall throughput improves using the imprecise clock.
The distinction between enterprise edition and open source expiry was unclear prior to this branch. The open source files (leveldb_os/expiry_os.cc, leveldb_os/expiry_os.h, leveldb_os/expiry_os_test.cc) were directly copied into the enterprise edition repository with only the file names changed (leveldb_ee/expiry_ee.cc, leveldb_ee/expiry_ee.h, leveldb_ee/expiry_ee_test.cc). This branch moves the open source files to the utils directory. Then the enterprise edition derives is class (ExpiryModuleEE) from the open source class (ExpiryModuleOS). The new organization eliminates having the same code in two places.