Add 2I to Memory backend

[seancribbs]

As it says, this adds secondary indexes to the memory backend. Why?

• The memory backend is useful when running automated test suites because it is less expensive than creating and deleting LevelDB instances. (A special "test" mode has also been added, with a corresponding reset() function.)
• ETS is capable of providing this feature with minimal additional work.

Caveats:

• Index postings are not considered when computing the size of an object, and therefore do not count towards its eviction when used as an LRU. This could be added later but is potentially expensive to compute without introducing a coupling to riak_object.
• It is necessary to use ets:match_delete/2 and ets:select/1,3 operations to efficiently select items from the index, as ETS has no concept of cursors beyond ets:first/1 and ets:next/2 which do not let you specify a starting key. match and select have been known to be locky in certain scenarios. ets:match_delete/2 must be used when deleting all postings for a bucket/key from the index table, due to the natural ordering of index keys. This penalty is only incurred when removing objects in LRU mode.

[slfritchie]

I'm a bit puzzled by the comment about ets:next/2, because that func can give you cursor-like traversal. But if you've got an implementation using match_delete and select that works, I don't have anything to complain about. "Locky" is a term of art for "don't expect this to perform well". :-)

[seancribbs]

@slfritchie "which do not let you specify a starting key" is the main point there, sorry. In contrast, LevelDB lets you seek to a starting key, then iterate. How would you do the same in ETS?

[slfritchie]

With an ordered_set table, ets:next(Tab, AnyKeyOfYourChoosing) should choose the next key. For other types, you need to use safe_fixtable() before getting deterministic results from next/2. IIRC.

[seancribbs]

A few tidbits from the Efficiency Guide:

Select/Match operations on Ets and Mnesia tables can become very expensive operations. They usually need to scan the complete table. You should try to structure your data so that you minimize the need for select/match operations.

...

There are exceptions when the complete table is not scanned, for instance if part of the key is bound when searching an ordered_set table, or if it is a Mnesia table and there is a secondary index on the field that is selected/matched.

In this case, because a large portion of the key is bound in the match specs, it should be less likely to need a full-table scan.

[seancribbs]

@slfritchie Aha, so it could be done with next/2 (the table is an ordered_set). Is it worth rewriting it?

[slfritchie]

If this new feature is only being used for testing, and it's not buggy, then I'd leave it as-is. If the new feature is meant for general use (including by customers), then it's worth getting rid of as many match_delete() and select() calls as possible: they're both too heavy-handed in an SMP world to use except when nothing else suffices.

[seancribbs]

@slfritchie Thanks. It might be difficult to remove match_delete(), but removing select() is do-able. Although this is useful to testing environments, some will undoubtedly use the memory backend in production. Better to head this one off at the pass.

[seancribbs]

@slfritchie Ok, I've switched as much as possible to use lookup() and next() over select(). Can you give it a good thrashing?

[slfritchie]

Sean, things look pretty sane. If the Dialyzer isn't angered more by these patches, then +1 -- ideally, the module would be warning-free.

[seancribbs]

Ok, I will dialyze and fix.

[seancribbs]

@slfritchie One final look if you will? Thank you so much for your help on this.

[slfritchie]

+1 to merge, many thanks!