Supporting Deletes in FASTER

[badrishc]

We are adding native support for the record Delete operation in FASTER:

• If the record is in the mutable region, it will be marked with the tombstone bit in header, and value will be overwritten with default.
• If the record is in memory and is pointed to directly by the hash index, we elide the record from the hash chain using a CAS on the index entry to the previous record. The elided record is marked with both a tombstone and an invalid bit in this case.
• If the record is in read-only or earlier region, we will add a new tombstone record to the end of the hash chain to identify the deleted record.

Note that from the perspective of recovery and CPR, deletes behave similarly to Upserts.

This checkin will replace the prior experimental method of DeleteFromMemory.

[badrishc]

This will be a breaking change, because the user callback functions will need an extra method to be implemented. This is called when the delete is completed, in the rare case when a delete operation goes pending (this can happen during a CPR checkpoint).

        public void DeleteCompletionCallback(ref Key key, Context ctx) { }

[gunaprsd]

Also, reads must make sure they are not reading a tombstone record. The liveness of a record needs to be checked everywhere.

[badrishc]

Correct, reads have already been updated accordingly to return NOTFOUND.

For log scans, right now I am returning deleted records as well. This is a debatable matter for resolution:

• We could treat tombstoned records as invisible to the scan (similar to Invalid records)
• We could expand the GetNext API to have a boolean IsTombstone out parameter so that the scanner will know they are scanning a deleted record.