proposal: file system notifications library

[nathany]

Propose new package(s) under golang.org/x/ that provide wrappers around x/sys (syscall) or cgo implementations for the following file notification systems:

• inotify and/or fanotify on Linux/Android
• FSEvents on OS X
• ReadDirectoryChangesW and/or USN Journals on Windows
• kqueue on *BSD, OS X, and iOS
• FEN on Solaris
• (optionally) an efficient polling library intended for watching file shares and for Plan 9

Compared to syscall:

• Provide a slightly higher level API.
  For example, using EpollCreate1 with the file descriptor returned by InotifyInit.
• Specifically for file system notifications.
  For example, only supporting EVFILT_VNODE for kqueue.
• Rigorous test suite.
  Checks for compatibility across architectures (Go builders).
• Go-friendly APIs.
  Write or NoteWrite rather than NOTE_WRITE.

Compared to fsnotify:

• No attempt to provide common behaviour across notification systems.
• Expose the full underlying API. For example InCloseWrite and InMoveTo for inotify.
• Not a common API. Each notification system would have a slightly different API.
• High quality reviewed code. The Windows code in fsnotify is particularly difficult to follow.

The intent is that libraries such as fsnotify and @rjeczalik's notify could then depend on fsn instead of depending on syscall directly. Such libraries could offer common APIs and behaviour, including user-space recursive watching, filtering, etc. Various APIs could be explored in this space, while relying on the same solid underpinnings.

related: #4068

[nathany]

@minux suggested x/sys/fsn which makes sense since it would utilize x/sys/windows and x/sys/unix.

[bradfitz]

Yes, I think x/sys is the correct place for this. I'm not terribly keen on fsn, though. The fs part is obvious. The n is not. If there are no plans to put a common API in place, why do we even need a new paclage? It can just be added to x/sys/unix or x/sys/windows? Or: we can add x/sys/linux, x/sys/freebsd, etc.

[nathany]

Yah, separate packages might be better.

Maybe x/sys/inotify, x/sys/fanotify, x/sys/fsevents, etc.? I was thinking of grouping everything under a folder called something (but not fsnotify if it's not trying to provide the API of fsnotify).

Nathan.

[ppknap]

I'm the author of windows and linux part of @rjeczalik's notify package, so I think that my comment may be helpful:

Not a common API. Each notification system would have a slightly different API.

Actually, they can and should have a common API. All file notification systems uses one(or more) watching strategies(according to my knowledge):

	file watcher	directory watcher	recursive watcher
Inotify	x	x
FSEvents			x
ReadDirectoryChangesW		x	x
kqueue	x

One can use file watcher to mimic directory watcher and recursive watcher and directory watcher to mimic recursive watcher and file watcher etc. That's what @rjeczalik notify package tries to do.

Now, we have three interfaces. Each notification system can implement some of them and we can create generic package that can mimic missing interfaces using common strategies.

Of course, each notification system package can expose its own methods. Then, we may end up with a "driver" like structure where we have independent low-level notification systems, maybe agnostic driver that selects the proper driver based on underlying operating system, and, of top of that, fsnotify package that registers a driver and tries to mimic missing interfaces.

I think this may be the right approach for usable file system notification library.

All in all, I'm not sure if x/sys is the right place for this(higher level) stuff. Nothing that can be build with methods from x/sys should be present in x/sys...

[nathany]

@ppknap Thanks for your insights and for your work on notify.

notify is much further along than fsnotify in this regard, as fsnotify still doesn't support recursive watches.

What I am trying to think through is how to build usable file system notifications library in layers.

Ultimately I want a common API with common behaviour across the selected notification systems. As a user, I want a library where I don't even need to know what inotify is, and can just get on with building whatever it is I'm building.

At the top: common behaviour by filling in the gaps

This provides an easy to API where the user doesn't need to know or care about the underlying file notifications systems on different platforms. The library makes the experience as consistent as possible across every platform that is supported.

This top layer comes with more opinions on how notifications should work and which features of the low-level APIs are exposed. For example, one opinion may be to use polling on BSD as with Plan 9 and NFS shares -- rather than trying to emulate both directory and recursive watching on top of kqueue's file-handle limited watcher. Another could be which underlying flags to expose and how to expose them, such as inotify's InCloseWrite which fsnotify doesn't currently offer or use (but probably should).

Part of the idea behind my original proposal is to enable multiple experiments at this layer, but without having to rewrite the lower layers each time. The internals of fsnotify v1.x could also be gutted, pulling in the low-level parts from a dependency, while maintaining its current API and set of behaviours.

A driver model: common API but not common behaviour

At a layer below that, contributors to fsnotify and the fsevents package have been discussing a "driver" like model for sometime. By the sounds of it, notify already has one.

My thinking here is to provide a common API but not common behaviour.

• A single platform may have multiple drivers (FSEvents & kqueue, inotify & fanotify, etc.)
• Not all drivers are available on all platforms, but a polling driver always is.
• The user selects which driver to use. This helps align expectations with reality -- each system behaves a little differently, like choosing between MySQL and PostgreSQL.
• Additional drivers could be added over time, such as FEN on Solaris, USN Journals on Windows, and fanotify on Linux.

At this layer, a kqueue driver should be accessible from a generalized API, but behave just like kqueue. No extras or emulations of other platforms. But it's important to think about how higher level abstractions could be built above the drivers.

When I first started on fsnotify, I was working on a pipeline for processing events, a bit like a middleware stack for adding user-space functionality like recursive watching and filtering. That project was discarded, but it's an approach I would consider trying again for fsnotify v2. I was also considering a capabilities API of sorts, so that the underlying file notification systems could be utilized for various features when available.

I was originally going to propose a driver model here, but I'm not sure how a common API should expose differences in flags and features that are available. How would you do it?

If the common API is already squeezing out some features and choosing which flags are available, it would seem that there is still another level down to go.

At the bottom

This comes back to my original proposal for "fsn" or whatever it's called. Really it is just a collection of packages for each notification system using cgo or syscall, along with cross-architecture testing.

It shouldn't take much to wrap these up in a driver interface.

But I don't know, maybe this layer isn't needed. I'm very curious how others imagine building a stack for useable file notifications. Cheers.

[nathany]

Contributing

Another goal I have is to help divide up the work and make it easier to contribute. Right now it is quite daunting to contribute to fsnotify. I think that has a lot to do with it supporting multiple platforms with all the low-level bits tangled in with the user-space extras intended to make it more consistent and usable for users.

By splitting this into layers, Linux users interested interested the low-level bits could work on a package for inotify or even a driver for inotify without worrying so much about cross-cutting concerns. Meanwhile, other people could work on filling in the gaps with user-space code across multiple notification systems, working with the inotify package/driver rather than directly with InotifyInit and Epoll.

Testing

Finally, I really want to improve the testing story. A better architecture will help, no doubt. Also, I'd like to build out a set of test helpers especially for integration tests that exercise the file system notifications in a myriad of ways. Make a subdirectory, touch a file, rename a file, etc., etc. Building up a library of things to check, and making it easier and cleaner to add additional tests when bugs arise.

I'd like to experiment with each file notification system independently to see whether a "golden file" approach with an -update flag could be used to capture and verify behaviour reliably across architectures. Right now this is just a theory, but if it works, we could have one set of tests that expect different behaviours from each driver, but expect the behaviours to be consistent over time. It could also help catalog the odd behaviours of each different file notification system, so we know which gaps need to filled to build a consistent experience at the top layer.

[ppknap]

I was originally going to propose a driver model here, but I'm not sure how a common API should expose differences in flags and features that are available. How would you do it?

Yes, I do have an idea how this may be resolved. However, this is not an official proposal for the file system notification library, but rather loose thoughts about the design, so all variable/type names should not be taken as an official naming.

fsnotify package - it's like sql package from std. It defines:

• An event as int64 type.(int32 was sufficient for notify package. However, we sometimes used it to carry additional watcher state information (example). It would be nice to have extra 32 bits only for that)
• Three interfaces:

type FileWatcher interface {
    WatchFile(path string, events ...Event)
}

type DirectoryWatcher interface {
    WatchDirectory(path string, events ...Event)
}

type RecursiveWatcher interface {
    WatchRecursively(path string, events ...Event)
}

The package Watcher type will use drivers and implement all three interfaces. Eventually, it will mimic these interfaces which were not defined in imported driver.

func Use("driver_name", ...) (*Watcher, err) // similar to sql.DB.Open

It would be nice if Watcher supports filtering etc.

This approach makes it easy to mock and test this package since it doesn't know anything about underlying notification systems.

driver packages -they are pure system dependent things. So:

• there are no generic events like inotify.Create etc.
• only system specific event are defined (`inotify.InCreate, inotify.InDeleteSelf etc.)
• they implement only these interfaces they can support eg. fsnotify.DirectoryWatcher and fsnotify.RecursiveWatcher for ReadDireectoryChangesW on windows.
• they may expose extra system dependent functionality.
• they have their own system dependent tests.

Since no one who writes system independent code wants to import different drivers for different systems. There is one special driver:

agnostic driver - it defines a common behavior across different systems:

• defines generic events like agnostic.Create, agnostic.Delete etc.
• defines a wrapper that transforms system dependent events to system independent ones.
• tests are system independent.
• the interfaces it implements still depend on underlying operating system.

So:

• fsnotify package acts like tree from rjeczalik/notify package.
• agnostic driver package acts like notify and fsnotify.
• driver packages are just low level implementations.

If you want to use fsnotify in system independent way:

import (
    "whatever/fsnotify"
    "whatever/fsnotify/agnostic"
)
// ...
w, err := fsnotify.Use("agnostic", ...)
// ...
w.WatchDirectory("something/path", agnostic.Create)
// ...

If you want to use system specific thing you can use eg. inotify directly or if you still want to have benefits of filtering etc:

import (
    "whatever/fsnotify"
    "whatever/fsnotify/inotify"
)
// ...
w, err := fsnotify.Use("inotify", ...)
// ...
w.WatchDirectory("something/path", inotify.InCreate)
// ...

However, just as I said, these are just my thoughts after implementing different notification watchers to one common interface.

[nathany]

Thanks for sharing. None of this is an official proposal yet, just thoughts towards writing a proposal document if we proceed.

So the WatchDirectory, etc. interfaces are essentially a way of exposing the capabilities of the underlying drivers, which only satisfy the interfaces they support?

It would be nice if Watcher supports filtering etc.

Yes, I would like to take advantage of the kernel's filtering. It's quite different between Windows and Linux, so this again is a place where capabilities matter.

The fall back in a top-level API is to watch more events than necessary and filter them out before the user sees them. That provides a consistent behaviour, but at the cost of the kernel waking up to mention an event that is subsequently thrown away. So I would want to give the kernel some hints about what to watch.

I will keep thinking on these things and I hope you will too.

My initial idea here is to build out the lower level packages and test suite. But we do need a better idea of how they will be used (as drivers or in some other way) to do this right.

[ppknap]

So the WatchDirectory, etc. interfaces are essentially a way of exposing the capabilities of the underlying drivers, which only satisfy the interfaces they support?

exactly

My initial idea here is to build out the lower level packages and test suite. But we do need a better idea of how they will be used (as drivers or not) to do this right.

Yes, I agree. If we have at least the sketch of how this may look like, I can implement ReadDirectoryChangesW driver(or not) as a proof of concept.

[nathany]

I think we can simplify this proposal.

What I feel is needed right now is aninotify package with a test suite that is tested across all Linux platforms that Go supports. Whether that is x/inotify, syscall/inotify, or somewhere else the Go builders can reach.

We don't have a problem testing fsnotify on macOS and Windows with third-party CI services, and we don't yet have a Solaris implementation.

This separation would be a step towards fsnotify (and alternatives) focusing more on building a common high-level API and less on the low-level InotifyInit and EpollCreate1 details.

Perhaps more importantly, it would really help improve confidence that changes to fsnotify aren't breaking platforms that we don't have CI testing for. That has been making me nervous about merging any pull request we get.

[nathany]

closed in favour of #17312.