proposal: add new flags to syscall package for mapping persistent memory files

[jerrinsg]

This is a proposal to add the following flags to the syscall package for Linux/amd64 - MAP_SYNC and MAP_SHARED_VALIDATE.

Persistent memory is a new memory technology that provides DRAM-like access speeds and disk-like persistence. Applications using persistent memory benefit in a number of ways - faster restart time due to its data being readily available to consume in persistent memory, better performance as a lot of code that does data marshalling/unmarshalling between storage and memory can be avoided, etc. More details about persistent memory is available here.

Applications use persistent memory by memory-mapping a file on a persistent memory filesystem into its address space. Applications can then use load/store instructions to read/write data into it. Since applications get byte-level access to the persistent memory device, having a page-cache in the kernel that buffers reads and writes to this file is unnecessary. So a dax mode mount option [1] is used while mounting a persistent memory filesystem to indicate to the kernel to avoid the page cache.

When an application issues a store instruction to write data into persistent memory, the store may get cached in CPU caches. The write can be guaranteed to be persisted only if the CPU caches are flushed as well. Newer generation processor architectures may make this unnecessary if CPU caches are also part of the power-fail protection domain, but that is an aside. An application can issue flushes from the userspace only if the filesystem guarantees that it is safe to do so. This is what the MAP_SYNC flag guarantees. A file system that honors MAP_SYNC flag ensures that any metadata needed for data consistency is flushed before an application writes to that particular region of the file [2]. Quoting the man page -

Shared file mappings with this flag provide the guarantee that while some memory is mapped writable in the address space of the process, it will be visible in the same file at the same offset even after the system crashes or is rebooted.

MAP_SYNC was added in Linux kernel version 4.15. Older kernels ignore this flag and there is no way for userspace to identify that it is unsafe to do application-level cache flushes. This is where MAP_SHARED_VALIDATE helps. Linux defines MAP_SHARED_VALIDATE as MAP_SHARED| MAP_PRIVATE, which on older kernels return an error and newer kernels validates all additional flags passed to mmap [3]. It can be seen that a simple program that mmaps a persistent memory file with flags MAP_SYNC|MAP_SHARED_VALIDATE on kernel versions lower than 4.15 fails with error EINVAL but succeeds on kernel versions >= 4.15.

This proposal is submitted following up from an earlier proposal (#43810) that suggested adding native support for programming persistent memory in Go. The consensus on that proposal was to not accept it. @rsc suggested (#43810 (comment)) adding separate proposals aimed towards adding low-level persistent memory support in Go, and this is the first among them. This proposal will help applications map a persistent memory file into its address space. Applications can then directly issue load/store instructions to that address range. I intend to follow up on this with another proposal that adds an API to flush CPU caches. These two proposals in conjunction will help add a low-level persistent memory support in Go.

[1] https://www.kernel.org/doc/Documentation/filesystems/dax.txt
[2] https://lwn.net/Articles/731706/
[3] https://lwn.net/Articles/758594/

[ianlancetaylor]

The syscall package is mostly frozen. These flags should normally be added to golang.org/x/sys/unix instead. And in fact they are already there. So I'm not sure there is anything to do here.

[jerrinsg]

Thanks Ian. I didn't notice these constants were defined in golang.org/x/sys/unix and searched only in syscall package. There isn't anything to do, so closing this proposal.