Allow glob patterns for native file watcher excludes

[ghuser]

Update from bpasero

⚠️ VSCode SUPPORTS glob patterns as files.watcherExclude. You can use any glob pattern and file events will be ignored properly. ⚠️

This issue is about supporting these glob patterns on the level of the file watcher library natively. The only reason we want this is to help Linux users with the "out of handles" error when many file handles are used for file watching.

⚠️ ignore this issue if you are not using Linux or when you are not seeing the "out of handles error" ⚠️

Original issue from ghuser

Does this issue occur when all extensions are disabled?: Yes/No

• VS Code Version: 1.62.3
• OS Version: Ubuntu 18.04.6 LTS

(First of all a big acknowledgement on the great work done with watcher on linux!! It works great in most cases.)

One Issue: I have a multiroot workspace. One of the roots contains the following subdir ./node/node_modules (instead of ./node_modules). The existing exclusion pattern **/node_modules/*/** does not exclude the nested node_modules directory from being watched.

I have tried adding the following exclusion patterns:

• **/*/node_modules/*/**
• **/**/node_modules/*/**
• */**/node_modules/*/**
• */node_modules/*/**

but without luck.

What works though is: **/node/node_modules/*/**, but this implies that we know the exact subdirectory path inside which the node_modules directory will exist.

Steps to Reproduce:

1. Make sure a multi-root workspace has been loaded to vscode before closing it. Also at least one of the root workspace directories should contain inside a node/node_modules directory.
2. execute: strace -f -etrace=inotify_add_watch /path/to/code 2>&1 | grep node_modules

You will see paths containing node_modules being printed. e.g.

[pid  4634] inotify_add_watch(58, "/projects/myproject/node/node_modules/modulename", IN_MODIFY|IN_ATTRIB|IN_MOVED_FROM|IN_MOVED_TO|IN_CREATE|IN_DELETE|IN_DELETE_SELF|IN_MOVE_SELF|IN_ONLYDIR|IN_DONT_FOLLOW|IN_EXCL_UNLINK) = 3024

[bpasero]

@ghuser the file watcher we are moving to does currently not yet support glob patterns for native excludes, so we have some code on our end to try best to convert glob patterns to absolute file paths for the watcher.

If you configure a relative path such as node/node_modules it should work better. Relative paths will automatically be joined with the workspace folder path.

[ghuser]

Indeed node/node_modules works and it is simpler, though it does not seem to have any difference to **/node/node_modules/*/**. It still requires the user to know the full path to the excluded dir, even if it is relative.

To sum up, the issue reported here is: the lack of a way to exclude a directory that could exist anywhere in the path.
Imagine a maven project with multiple sub-modules (something very common) where we need to exclude the target/ directory from each sub-module. Currently this does not seem to be possible unless we manually enter each one of the sub-modules' target/ directory to that exclusion list.

[bpasero]

Needs parcel-bundler/watcher#64

[unional]

FYI the default files.watcherExclude is still using glob pattern.

https://code.visualstudio.com/docs/getstarted/settings#_default-settings

  // Configure paths or glob patterns to exclude from file watching.
  "files.watcherExclude": {
    "**/.git/objects/**": true,
    "**/.git/subtree-cache/**": true,
    "**/node_modules/*/**": true,
    "**/.hg/store/**": true
  },

The tooltips is also still saying that.

Configure paths or glob patterns to exclude from file watching. Paths that are relative (for example build/output) will be resolved to an absolute path using the currently opened workspace. Glob patterns must match on absolute paths (i.e. prefix with **/ or the full path and suffix with /** to match files within a path) to match properly (for example **/build/output/** or /Users/name/workspaces/project/build/output/**). When you experience the file watcher process consuming a lot of CPU, make sure to exclude large folders that are of less interest (such as build output folders).

[unional]

Adding a use case for glob pattern here:

I'm working on a large repository that has code in C.

The build system builds the binary to a relative .bld folder.
There are hundreds of them.
So having a glob **/.bld/** is likely the only option.

[bpasero]

We have a heuristic to convert "simple" glob patterns into absolute paths given that. See this test for what is supported:

vscode/src/vs/platform/files/test/node/parcelWatcher.integrationTest.ts

Line 576 in b03fe7a

// simple globs (relative)

[bpasero]

Btw just to clarify: we do support glob patterns for watcher excludes. This issue is about supporting them on the native layer in the library itself (parcel watcher). Supporting them natively means you are not hitting the ENOSPC issue on Linux. But conceptually file changes are properly excluded even when you use glob patterns, just on a different layer.

[jaypea]

I had to downgrade and pin VSCode to version 1.62 on my Ubuntu system to keep using the monorepo i'm working with daily.
I also had to turn this setting on to keep using the older filewatcher.

"files.legacyWatcher": "on"

[bpasero]

Just to clarify: the previous file watcher behaved the same as the current file watcher in that excludes were not supported natively. So for this issue (running out of file handles), it makes no difference whether you use old VSCode or new VSCode. But when you are able to configure non-glob patterns for excludes, the new VSCode will behave better.

Edit
Actually not entirely true, for as long as we did use chokidar, exclude patterns would work even when glob patterns are used and reduce the handle count on Linux.

[jaypea]

@bpasero I run in to the "out of handles" error as soon as I change to the parcel watcher. my system setting is 524288 already.
I've also tried non-glob patterns as far as its possible in my workspace without any luck.

[bpasero]

Hm, does the workspace contain many folders with large structures or just a few? You would have to build absolute path patterns to exclude them unfortunately. I am open for suggestions how to make this any easier.

[ghuser]

@jaypea If possible use the command described on top to find out whether you have missed some exclusion pattern and also be able to provide some detailed feedback to the team with specific examples on when a directory is not excluded.

For example the specific command helped me fix my problem because I found out that I had to also add **/node/node_modules/*/** in the exclusion list.

What is described on this issue is that it would be even better to be able to exclude a directory (e.g node_modules) anywhere in the path without the need to specify the relative path to it.

[bpasero]

Given parcel-bundler/watcher#106 landed and we updated to 2.1.0, glob patterns for files.watcherExclude setting are now supported all the way down to the file watcher. You can:

• use relative paths: they are applied relative to the watch root
• use absolute paths: they are applied absolute
• use globs: they are applied relative to the watch root

Thanks @joaomoreno for the clever idea of using a node.js module for the heavy lifting of creating the glob regex and then plugging that into parcel watcher. This was a fun journey that even resulted in a StackOverflow question regarding RegEx on different platforms that remains unanswered and a mystery: https://stackoverflow.com/questions/74796098/c-regular-expression-with-negative-lookahead-and-matches-on-macos-but-not-l