atomic file writing

[vtjnash]

When writing out files, often we want to do this atomically, so that it gets a new inode, the original can be replaced even though read-only (maybe), and interruptions can't leave behind a broken file. We might be able to do this with an extra kwarg to open that specifies a temporary file, and then rename the new file on top of the old file.

open("replace.txt", write=true, atomic=true) do f
    println(f, "nothing")
end
open("replace.txt", read=true, write=true, atomic=true) do f
    println(f, replace(readchomp(f), "no" => "some"))
end

Other names might include atomic::Bool, tmpsuffix::Union{String,Function}, tmpfile::Bool. This option cannot be used with append=true.

This could be used to fix #35217. Might also want better copyfile support from upstream for completeness (libuv/libuv#3126).

(if O_TMPFILE is available, we can potentially even do fancier things, see https://lwn.net/Articles/619146/ and https://manpages.debian.org/stretch/manpages-dev/openat.2.en.html)

[mgkuhn]

How will this work under the hood regarding the type returned?

Will open("replace.txt", write=true, atomic=true) then return another type than the current IOStream, say a wrapper IOStreamTmp that stores both the temporary and the final name of the file being written, until close is called, such that close(::IOStreamTmp) then knows what to rename at the end?

Will open("replace.txt", read=true, write=true, atomic=true) return yet another type, say a wrapper IOStreamReplace for two IOStream objects oldfile and newfile, one for the original file being read and one for the temporary new file being written? And all the write methods on IOStreamReplace redirect to newfile and all the read methods on it redirect to oldfile? But what about IO methods that normally affect both read and write, such as seek?

[vtjnash]

This is only for the do block form.

[vtjnash]

Prior art here may include llvm::writeFileAtomically: https://llvm.org/doxygen/namespacellvm.html#a98697d7ad6deeffd48a7dc33b3a85606
https://llvm.org/doxygen/FileUtilities_8cpp_source.html#l00298

[mgkuhn]

@vtjnash OK, but the second form (read-write) will have to return a Tuple{IOStream,IOStream}, as in

open("replace.txt", read=true, write=true, atomic=true) do fin, fout
    println(fout, replace(readchomp(fin), "no" => "some"))
end

because we are dealing with two separate files here.

[mgkuhn]

A related (additional) option atomic=:compare that I frequently would find useful: a mode to atomically replace an existing file but not change anything if the content of the new file equals the content of the existing old file.

So basically: after closing the temporary new file, compare the contents of the old and the temporary new file, and then just delete the temporary new file if its content is identical to that of the old file.

Use cases include regularly running jobs (e.g., cron jobs to update a static web page), where this can help to avoid churn on backup systems, rsync mirrors, make builds, and similar environments where an unnecessary change of timestamp or inode number of a file can be unwanted.

The comparison could be done either by reading and comparing both files, or by calculating both their SHA-256 hashes. The latter could presumably even be done on the fly via a new type IOStreamHashed{sha2_256} that calculates a hash value of the file content as it is being read or written (which for most hash algorithms will mean that seek cannot be allowed).

[elextr]

IIUC Windows does not allow renames over open files, so the situation where something else has the file open and the rename fails needs to be addressed. This also occurs for a Windows filesystem mounted on a Linux system, so the system Julia is running on isn't a reliable guide. Also some remote filesystems don't do rename.

Also since the newly created file gets creation metadata (eg protection, ownership, execute bit) it may be different to the old file, so if possible those should be copied to the new file.

[mgkuhn]

@elextr The default mandatory-locking file-system semantics of Windows are a well-known nightmare inherited from MS-DOS, but there is little we can do at the time of the rename other than perhaps reminding users in the documentation of the fact that fully atomic replace is not guaranteed on Windows-like file systems. I'm not aware of any nice fall-back algorithm that can remedy the situation here, other than to fail properly (i.e., remove the temporary file, then rethrow the exception). (One particularly horrid workaround would be the MoveFileEx option MOVEFILE_DELAY_UNTIL_REBOOT which can postpone the rename until after the next reboot! It keeps a list of files to be moved in the registry.)

One thing that the process that still has opened the file that is to be replaced atomically could have done is to also set in CreateFile the FILE_SHARE_DELETE flag. Julia (libuv) could always set that, when it opens a file, to be a good citizen. But other applications may not do so.

Anyway, none of this should distract from the fact that the proposed extension is a perfectly useful and simple extension for Unix-style file systems.

See also:

• Moving and Replacing Files, Win32 API documentation
• File system operations, libuv documentation

[elextr]

Anyway, none of this should distract from the fact that the proposed extension is a perfectly useful and simple extension for Unix-style file systems.

For sure, it just needs to clearly documented as such making sure all known situations it does not work are listed, simply saying "only Linux filesystems", whilst technically correct, is likely to be unhelpful to people who operate in one of the problematic configurations, "but I am on Linux", "Oh, my central file server/storage appliance isn't a Linux filesystem, who knew".

And of course its use does cause code portability issues and that should be highlighted, but I suspect there will still be questions/issues about it not working in various situations.

[oschulz]

Go went this route to get atomic file replacement on Windows, it seems: golang/go#8914