Adding KV tombstones to fix non-monotonic index on deletes

[armon]

Fixes #195. The visible issue from the outside of Consul is that if you are watching a prefix for changes (say service/web/config) for changes, and a key is deleted from the folder then the X-Consul-Index value either doesn't change or goes backward. This means that long-polling for changes breaks during a deletion. The reason for this is that the index is calculated as the Max(ModifyTime) of all the entries. The Max() function however is only monotonic if the inputs are also monotonic. A delete breaks this by removing entries, causing the Max value to slide backward or to stagnate. Because of this, blocking queries get "stuck". This is what a user experiences.

To fix this, we need to ensure X-Consul-Index is monotonic, even with deletes. This is tricky because you need to detect the absence of data. One way to do this is to never actually delete, but only soft delete and filter the results. Unfortunately, this causes unbounded data usage. The trade off we make is to soft delete by creating a "tombstone" entry. These are kept around for a configurable amount of time (default 15 minutes) and then GC'd later. This prevents unbounded data growth. However, while the tombstones are around, we use them to calculate the X-Consul-Index value and ensure the inputs are monotonic. Because of the tradeoff with unbounded storage growth, this only papers over the issue until the GC happens, at which point the index may slide backwards again.

In practice, this fixes the issues with replication and change detection and so works well enough. The only clients that would be affected by the GC are ones that are doing blocking queries extremely infrequently (less often than the GC runs). This is an edge case, and the behavior will be no worse than it currently is for them.

[ryanuber]

[mitchellh]

Straight outta India (literally).