-
Notifications
You must be signed in to change notification settings - Fork 76
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Integrate the bucket types functionality. basho/riak#362 Overall ------- Bucket types are the future of namespacing and property creation in Riak. They allow efficient storage of "bucket properties" outside of the Ring and 2-level namespacing of `Type` and `Name`. Essentially the bucket type can now be either a lone `binary()` (legacy) or a 2-tuple of `{Type :: binary(), Name :: binary()}`. Internally, when the legacy version is encountered it is considered to live under the `default` bucket type. For example the bucket `<<"my_bucket">>` would become `{<<"default">>, <<"my_bucket">>}`. Up until this point Yokozuna has used the bucket property `yz_index` to determine where to index data. This commit changes that in some ways. Legacy users will have existing data in buckets. Those buckets, in 2.0, will be considered to live under the default type as described above. For legacy buckets (the default type) Yokozuna will NOT respect the `yz_index` property. Rather it will act like Riak Search and use an index named the same as the bucket AS LONG AS the `search` property is set to true. Once users upgrade to 2.0 they should start opting into non-default bucket types since it is more efficient and newer features require the use of non-default type. For these types of buckets Yokozuna will still use the `yz_index` property. This property will typically be set at the type level but can also be overridden per name under a type. Yokozuna doesn't care. If that `{Type, Name}` has a `yz_index` property then it will be indexed. In summary: * Legacy buckets (default type) will act like Riak Search. The index used must have same name as bucket and `search` property must be true. This is used to users migrating from Riak Search. * All new users MUST use new style buckets made of Type + Name. In most cases the `yz_index` property will be set on the type and thus inherited by all names under it (many buckets to one index). The index DOES NOT have to have the same name. Handoff ------- Another important change revolves around handoff. Since Yokozuna leeches off the KV vnode it doesn't have control over handoff like it would if it were a true vnode. When a node joins KV can start shipping data before the bucket type data has been shipped over. In that case there will be no `yz_index` property and indexes will be missing. AAE would eventually catch this but it is poor form that node join would cause a degradation in harvest, especially in a quiescent cluster. To fix this Yokozuna needs more control over the lifecycle of the KV vnode. Yokozuna needs to hook into the `handoff_starting` stage and verify that the bucket types data is shipped before data handoff begins. This is accomplished by adding the `yz_kv:should_handoff` hook which is hard-coded in the KV vnode for now. This is important for removing the hack around index creation as well. Currently Yokozuna has a pretty horrible busy-wait hack in its index hook to make sure indexes are created on joining nodes before doing the first write of a handoff. This busy-wait blocks the KV vnode and is dangerous for vnode latency. In a future commit this busy-wait will be replaced with a check in this new handoff hook. Removal of Automatic AAE Tree Clearing -------------------------------------- Remove all functionality around automatic clearing of trees when adding or removing the `yz_index` property on a bucket with data. This was referred to as `sync_data` in the `yz_events` module. Also called "flags" harking back to when Yokozuna had a one-to-one bucket-to-index mapping. The original intention was that adding an index to a bucket with data should clear the AAE trees so that exchanges would start repairing missing indexes. If setting the index property to the tombstone value (removal) then a) data for that bucket should be purged from the index and b) AAE trees should be cleared. After much thought I think this implicit behavior hurts more than helps. Actions like clearing all AAE trees can be very expensive. It will not be obvious to all users that adding or changing `yz_index` could cause expensive operations to occur. For example, clearing the AAE trees for a database with billions or trillions of objects will be expensive to rebuild. Rather than relying on AAE a more direct operation could be offered that allows the user to re-index a bucket or subset of data. When removing an index it makes more sense to let the user delete the index entirely rather than do an implicit delete-by-query which is doing a bunch of extra work for a index that is going to be deleted anyways. Misc Changes ------------ * Update all tests to work with bucket types. * Update Basho Bench driver to work with bucket types. * Make map-reduce extraction more efficient. This is the ugly hack found in `yokozuna:positions`.
- Loading branch information
Showing
27 changed files
with
443 additions
and
469 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.