Auditbeat's file_integrity module deadlocks under Windows

[adriansr]

When the file_integrity module is used under Windows to monitor large and/or busy filesystems, it can end up in a deadlock.

When using this configuration:

- module: file_integrity
  paths:
  - C:/windows
  - C:/windows/system32
  - C:/Program Files
  - C:/Users
  recursive: true
  hash_types: [sha256]

Auditbeat will not generate any events whatsoever.

The reason for this is that the Windows implementation of fsnotify uses a single goroutine to forward events to auditbeat and to install watches. Communication with this goroutine is done via channels. In this case, what's happening is that the queue on channel to forward events fills and the whole goroutine blocks, so its unable to install new watches.

This is because as watches are installed, events start to be received but auditbeat is not consuming them, as it first installs the watches and then starts the consumer.

Auditbeat use of fsnotify looks like this:

  # setup phase
  for dir in config.paths:
     installWatch(dir)
  # run phase
  consumeEvents()

The recursive feature makes the problem much likely to appear for two reasons:

• it involves installing a watch for every subdirectory inside a watched directory. That's thousands of watches for C:\windows.
• it does not only install watches during the setup phase, but also when a subdirectory is created inside a watched directory. This means consuming events also involves installing more watches.

The simpler way to fix it requires two changes:

• Don't install any watches during setup phase. Use a goroutine inside consumeEvents() so that events are being consumed while watches are installed.
• Modify fsnotify to use OS support for recursive watches. This makes installing watches for new subdirectories unnecessary and simplifies things.

[adriansr]

This caused #6796