Support for Parallel File Systems

[henoby]

Hi,

thanks a lot for your great work!
We would like to use GoCryptFS from multiple nodes on our HPC system concurrently to write to our Parallel File Systems (e.g. BeeGFS, Lustre, GPFS, NFS). However, when multiple processes are writing to a single file concurrently we seem to get a race condition.

[rfjakob]

Work is ongoing in the https://github.com/rfjakob/gocryptfs/tree/LockSharedStorage branch.

This uses "Open file description locks" / F_OFD_SETLKW which have both a sensible API and work on NFS acc. to
https://gavv.net/articles/file-locks/#differing-features
Note: the article http://0pointer.de/blog/projects/locking.html is from 2010 where F_OFD_SETLKW did not exist yet.

I will comment again once I have something ready to test.

[rfjakob]

Hi, this is ready to test. When mounting with -sharedstorage, all concurrent R/W and W/W should be safe now. As long as the storage supports F_OFD_SETLKW byte-range locks.

[henoby]

Hi,
thanks a lot for your very quick response. I have quickly tested it on BeeGFS from two client nodes using the following ior command:
mpirun -n 2 ior --dataPacketType=timestamp -C -Q 1 -g -G=-1367929591 -k -e -o /gocryptfs/mount/file -O stoneWallingStatusFile=/local/path/ior-hard.stonewall -t 47008 -b 47008 -s 10000000 -w -D 300 -a POSIX -O saveRankPerformanceDetailsCSV=/local/path/ior-hard-write.csv -O stoneWallingWearOut=1
Without the -sharedstorage flag I get the expected ERROR: write(17, 0x770000, 47008) message.
Using the flag, it works!
I will do some benchmarking, to see the "overall cost" of using this lock. I will share the detailed results with you as soon as I have them.
Thanks a lot! :)

[rfjakob]

Ping @henoby