Gatsby should not watch for changes deep in node_modules

[discordianfish]

Hi,

by default gatsby seems to setup inotify watchers for every single directory in a project, including node_modules and sub directories. Even with modest setups this causes ENOSPC errors in linux because the user runs out of inotify watches (8192 by default). This can be increased by modifying fs.inotify.max_user_watches with sysctl but shouldn't be necessary.

Instead gatsby should only watch one level of sub directories in node_modules (and keep watching every directory elsewhere), so it will detect updated dependencies without watching directories deep in the tree that should never change.

PS: This same feature request applies to many other projects like yest or grunt-watch but going to start here :)

[jquense]

do you have a sense of which watcher is is doing that? There are quite a few scattered around, not including the ones webpack sets up. As far as I know though they aren't watching node_modules

[discordianfish]

This is the error I get with the default sysctl settings:

error UNHANDLED EXCEPTION


  Error: watch /home/fish/latency.at/blog/node_modules/sockjs/examples/multiplex ENOSPC
  
  - nodefs-handler.js:37 createFsWatchInstance

[KyleAMathews]

Yeah if we're watching node_modules that'd be bad :-) but yeah, dunno which part of Gatsby would be doing this. It isn't something we're doing on purpose so…

[discordianfish]

Just reproduced this by creating a new project with gatsby new then run gastby develop in there. It also seems to watch node_modules:

$ gatsby new test
c2017-08-30T10:49:27+0200 <log> init-starter.js:91 (clone) Cloning git repo git://github.com/gatsbyjs/gatsby-starter-default.git to test...
d 2017-08-30T10:49:28+0200 <log> init-starter.js:97 () Created starter directory layout
2017-08-30T10:49:28+0200 <log> init-starter.js:29 (install) Installing packages...
test

> uws@0.14.5 install /tmp/test/node_modules/uws
> node-gyp rebuild > build_log.txt 2>&1 || exit 0

added 1379 packages in 43.853s

$ cd test
$ gatsby develop
Something is already running at port 8000 
Would you like to run the app at another port instead? [Y/n] y
success delete html files from previous builds — 0.010 s
success open and validate gatsby-config.js — 0.035 s
success copy gatsby files — 0.013 s
success source and transform nodes — 0.016 s
success building schema — 0.064 s
success createLayouts — 0.018 s
success createPages — 0.004 s
success createPagesStatefully — 0.008 s
success extract queries from components — 0.063 s
success run graphql queries — 0.012 s
success write out page data — 0.002 s
success update schema — 0.057 s

info bootstrap finished - 6.585 s

error UNHANDLED EXCEPTION


  Error: watch /tmp/test/node_modules/postcss-pseudo-class-any-link/node_modules/postcss-selector-parser/dist/selectors ENOSPC
  
  - nodefs-handler.js:37 createFsWatchInstance
    [test]/[chokidar]/lib/nodefs-handler.js:37:15
  
  - nodefs-handler.js:80 setFsWatchListener
    [test]/[chokidar]/lib/nodefs-handler.js:80:15
  
  - nodefs-handler.js:228 FSWatcher.NodeFsHandler._watchWithNodeFs
    [test]/[chokidar]/lib/nodefs-handler.js:228:14
  
  - nodefs-handler.js:407 FSWatcher.NodeFsHandler._handleDir
    [test]/[chokidar]/lib/nodefs-handler.js:407:19
  
  - nodefs-handler.js:455 FSWatcher.<anonymous>
    [test]/[chokidar]/lib/nodefs-handler.js:455:19
  
  - nodefs-handler.js:460 FSWatcher.<anonymous>

[discordianfish]

Oh and

$ gatsby --version
1.9.14

[jquense]

maybe here?

gatsby/packages/gatsby/src/utils/develop.js

Line 155 in e876fd0

const watchGlobs = [`src/html.js`, `**/gatsby-ssr.js`].map(directoryPath)

[KyleAMathews]

@jquense yup! That was it. Made a quick PR to change the glob #1973

[sebastienfi]

Still having errors about that.

https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby/src/reporter/prepare-stack-trace.js#L15

Happens on npm run build exclusively.

[jquense]

I think that's a different error tho?

[KyleAMathews]

Yeah different error. @sebastienfi this issue was node.js was watching too many files. Your stack trace does look like a new bug so opening another issue would be great.

[christroutner]

This Issue pops up as the first result for a google search for 'gatsby ignore node_modules ENOSPC'. I'm running the following software and constantly running into the ENOSPC error:

• Ubuntu: v20.4
• node.js: v14.15.5
• npm: v7.24.0
• Gatsby CLI: v2.12.21
• Gatsby: v4.1.0

From reading the history of this old Issue, it appears there was supposed to be a PR that fixed this issue. But it clearly still exists.

Running this command is a brief reprieve:

• echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p

But it only works once, and then I have to reboot my compute if I want to run npm run develop

I'd LOVE to find a permanent solution to this issue.