contact file creation: fsync parent directory

[matthewrmshin]

On file system such as an NFS mounted file system, we may have a delay
before the file becomes visible from other process running on other
host. We already have a 1st fsync to ensure that the data of the contact
file is written to disk. This change adds a 2nd fsync to ensure that the
file metadata of the contact file is written as well - by doing an fsync
on its directory.

(The issue is causing rose suite-run to fail to launch cylc gui after calling cylc run successfully to start up a suite.)

(Need to back port to 7.8.2.)

[oliver-sanders]

Unable to replicate issue, change looks sensible.

[matthewrmshin]

metomi/rose#2290 is companion of this PR.

[kinow]

+1, also cannot replicate, but looks OK. Also spent some minutes reading about fsync, NFS, NFS' cache, why I never had to bother about it with Java (I believe calling .flush() on OutputStream has a fsync hidden somewhere in the JVM/JDK libs).

All checks passed, merging!

[kinow]

@matthewrmshin watching the Fosdem talks from this year, and found this good one about an old issue in Postgres with fsync: PostgreSQL vs. fsync - How is it possible that PostgreSQL used fsync incorrectly for 20 years, and what we'll do about it.

Give away, they are planning to tackle this issue over 2 or 3 years 🙂 and quite interesting the issues with Kernel, memory pages, fsync, and even different behaviour with different linux file systems.

Decided to see if anybody ever had issues with fsync in Python, and looks like there is a 8 years old bug: Change os.fsync() to support physical backing store syncs, which doesn't seem to have had any work since 2013, and doesn't look to have an easy fix.

So I think that if some day some user comes with some crash in Cylc, and complains that it looks like there was an error writing to the disk, and he is using NFS, then it is possible that there was an error on flush/fsync/write/etc, but it depends on operating system/file system/kernel version/python version/NFS versions/etc. And there are situations when we just cannot explain what happened.

[hjoliver]

(The issue is causing rose suite-run to fail to launch cylc gui after calling cylc run successfully to start up a suite.)

BTW I've noticed this many times in the past, but never looked into it (easy enough to manually fire up cylc gui...)

[matthewrmshin]

Yes, but users panic and keep blaming the new version.

[hjoliver]

Not saying it shouldn't be fixed - just highlighting my own tardiness on never following this one up!

[matthewrmshin]

Watched the video at the link posted by @kinow - all 40-something minutes of it. While not the problem here, it does highlight how little understanding I have with fsync. Yes, what happens if fsync returns an error? (I think we are OK here - the suite should fail to start.) But what if fsync does not return an error when there is one? (Then we are stuffed - but the suite should die on health check.)

[kinow]

At least we are not alone @matthewrmshin . I am reading the Notebook documentation around security, authentication, etc. And found the note

We are aware that SQLite doesn’t work well on NFS and we are working out better ways to do this.

The "working out better ways" links to this issue: jupyter/notebook#1782