Recovery optimization for persistent channels

[krasserm]

Durable queues on top of Cassandra are known as anti-pattern. The issues related to reading a large number of tombstones can be addressed by introducing optimizations on persistent channel level and on journal level.

Possible optimizations on persistent channel level are (copied from a code comment in PersistentChannel.scala):

// TODO: avoid scanning over large number of tombstones during recovery
//
// Introduce an optimization to address issues mentioned in
// http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets
// (which is also relevant when using a local LevelDB).
//
// This requires that recovery should start from the first
// non-deleted message rather than from sequence number 1.
// This can be achieved by taking empty snapshots (savepoints) 
// to set a recovery starting point. During recovery
//
// - when the first replayed message is received, take its
//   sequence number n and write a savepoint at n - 1.
//   This ensures that the next recovery skips a possible
//   expensive scan over n - 1 messages with a tombstone.
// - when the savepoint has been successfully written, delete
//   all savepoints that are older than n - 1.
//
// Writing a savepoint at lastSequenceNr - 1 requires a direct
// interaction with the snapshot store actor.
//
// Alternative/addition:
//
// The RequestReader actor of a persistent channel could
// also process confirmation messages and compute starting
// points for recovery. At periodic intervals, these starting
// points are then persisted as savepoints.

One optimization on journal level is to delete entire rows if all columns have been marked as deleted (i.e. have a tombstone). A prerequisite is to have row splitting which is already implemented by #1.

Furthermore, reads form the journal with no lower bound (i.e. starting from sequence number 1) are only done during recovery. All other reads (done by View or PersistentChannel) have a lower bound which are fast. Assuming infrequent persistent channel recoveries, Cassandra can be configured in a way that tombstone garbage collection is likely to occur between recoveries.

[krasserm]

Wrong reference from commit.

[rkuhn]

@bantonsson: this might interest you as well

[krasserm]

Obsolete with deprecation of persistent channels in Akka 2.3.4.