Efficient change detection using NTFS USN numbers

[m5x]

Is there a technical reason why Unison does not use USN number or change journal on NTFS file system to efficiently detect changes and do not have to resort to comparing timestamps or scanning the full contents of all files every sync run?

[bcpierce00]

Nope — the implementation could probably be extended to do this, if someone cared to put in the effort. But I’d be surprised if the effort was worth it — checking timestamps is pretty fast even for fairly large filesystems.

…

On Jul 20, 2018, at 11:46 AM, Martin Dvořák ***@***.*** ***@***.***>> wrote: Is there a technical reason why Unison does not use USN number or change journal on NTFS file system to efficiently detect changes and do not have to resort to comparing timestamps or scanning the full contents of all files every sync run? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#209>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGsIC848Zzhy4cxK-nVcklrXnEMzxKxZks5uIftwgaJpZM4VYP_k>.

[m5x]

That's a surprising opinion. Checking timestamps involves rescanning of the whole tree every backup run (which is not efficient and in case of trees with millions of folders and files not even fast) but more importantly it's not reliable method at all. Timestamps are not changed in many scenarios (access through memory mapped files etc.) so full content scan must be periodically performed as well to make sure these files are backed up at least sometimes.

I am no expert on NTFS change journal but from what I can see other backup tools are doing I understood that it allows to detect changes without rescanning the whole tree and with such a reliability that full content scans are never needed anymore. If that's true it would make unison much more reliable and efficient with NTFS.

[gdt]

I wonder how this relates to the concept of watcher programs, where there is an interface to tell unison about changes. I am not enthused about adding os-specific mechanisms to a program that tries to steer to POSIX, but someone building an NT-specific watcher that uses this feature seems useful.

Also, the issue of changes not being detected y the normal scan should be addressed separately; perhaps unison should, on systems that have changed files without mod time changes, rescan contents. (I am unclear on mmap/mod-times on unix.)

[gdt]

Moved to https://github.com/bcpierce00/unison/wiki/Feature-Requests

(At this point it is difficult to have people even test on Windows, and no one working on unison is interested in OS-specific improvements.)