Failures using gocryptfs-on-NFS as ElasticSearch snapshot store #156
Description
This is distinct from #39 because it proposes using advisory locks to simulate Windows single-writer filesystem semantics (which are safer when having a writer can invalidate concurrent reads), as opposed to letting applications make their own use of advisory locks.
I'm using gocryptfs to encrypt an ElasticSearch snapshot store.
This works perfectly when the ElasticSearch cluster consists of only one node. With multiple nodes, however, there are various race conditions and timing errors that take place as one node tries to read content that another has not fully written, or when a node attempts to delete a presently-invalid/partial file (resulting in the unlink() attempt returning ENOENT despite the file actually existing).
Since ElasticSearch's snapshot support is designed to work with Microsoft's filesystem semantics -- where only a single writer to a given file is allowed and no concurrent reads are permitted during writes -- it should be possible to avoid these errors by using POSIX advisory locks to prevent any node from reading or writing to a file which any other node is actively writing to, or to prevent a file from being opened for write while any other node has it opened for read.
Obviously this would not be appropriate as default behavior, but would be useful to provide as an available option.
Thoughts on feasibility? Guidance on where to start with an implementation?