Separate HdfsSnapshotStore as akka-persistence-hdfs

[ktoso]

Snapshots can be also stored directly to HDFS, if they are really big or you need "easy takeout".

[dispalt]

@ktoso do you plan on making the accumulation of events work on hdfs too, or just snapshots?

[ktoso]

Putting events onto HDFS is not optimal I think, HBase performs way better for such operations (SCANs). I'd be inclined to say no to the feature of events directly on HDFS.

(see my talk about avoiding hot-spotting to get more insight how hbase is a better target for events: http://www.slideshare.net/ktoso/hbase-rowkey-design-for-akka-persistence )

[dispalt]

I was thinking since journaling is writing a consistent stream (nothing random), it would be an good fit. In my mind it would function similar to HBase's use of HDFS for it's WAL, since they are essentially the same thing. But you'd probably end up to write a lot of code to make it efficient and preserve the semantics.

Yeah I am familiar with region hotspotting, was more curious abt using HDFS directly.

[ktoso]

I think I'd end up reimplementing a lot of what hbase does when trying to implement this.
I always viewed hbase as "the best way to scan across HDFS".
If we'd naively just append to a file in HDFS I think we'd bump into hot-spotting anyway - directly overloading one datanode more than another one.

I may be wrong (that happens from time to time :-)), please correct me (code very welcome) if that's the case here :-)