Use Gluster/Ceph for build storage instead of tarballs

[ghost]

Originally reported by: Brent Tubbs (Bitbucket: btubbs, GitHub: btubbs)

---

A thought occurred to me the other day. What if instead of making tarballs out of builds, then pushing them into GridFS, then pulling them back out somewhere else, we just built the app directly into a networked folder like you get from Gluster or Ceph, and then the app hosts could be part of the same cluster with the same mounted folders?

I'm pretty sure we could still use overlayfs so that multiple instances of the same app/build on the same host would be isolated from each other.

The big advantage is that the bits from the build would start syncing to the hosts as soon as they came out of the build tools (pip, gcc, babel, whatever) at build time. All the time spent creating the tarball would be gone. All the time spent unpacking the tarball would be gone. And much of the time spent transferring the build would run concurrently with the build itself.

It would require that Gluster or Ceph support trans-datacenter clusters, which I haven't looked into (though I'm optimistic that they do).

There would be some dev/sysadmin tradeoffs:

• VR would no longer need a concept of storage backends (currently only a GridFS backend is implemented). This would be replaced with just writing to the local filesystem.
• VR would no longer be dependent on Mongo.
• VR would no longer be dependent on an app like Khartoum/Khargo to serve files out of Mongo.
• VR would become dependent on the new clustered file server (GlusterFS or Ceph).

I think this change is potentially a big win for deployment time performance.

---

• Bitbucket: https://bitbucket.org/yougov/velociraptor/issue/184

[ghost]

Original comment by Brent Tubbs (Bitbucket: btubbs, GitHub: btubbs):

---

I think I overestimated the capabilities of Gluster and Ceph. Googling turns up lots of recommendations against using them over a WAN. There are some hints that a master/slave asynchronous DRBD setup might work for this use case, but it's not obvious.