Latest hugo fails in watch mode on "too many open files" on OS X

[mattstratton]

Since updating to 0.55.6, when we run hugo server -w on our site, we get the following error on OS X:

ERROR 2019/07/24 09:31:04 Error: listen tcp 127.0.0.1:1313: socket: too many open files

We have had this happen in the past, and solved it by setting the following:

sudo launchctl limit maxfiles 10240 unlimited
sudo sysctl -w kern.maxfiles=65000

However, with 0.55.6, this doesn't seem to help. If I run with version v0.54.0-B1A82C61 it works fine.

For reference, this is the output of the build command (it builds fine, it just can't work in watch mode):

                   |  EN
+------------------+------+
  Pages            | 6436
  Paginator pages  |    2
  Non-page files   |   70
  Static files     | 8034
  Processed images |    0
  Aliases          |  462
  Sitemaps         |    1
  Cleaned          |    0

To reproduce this, clone the repo at https://github.com/devopsdays/devopsdays-web and run
hugo server -w --baseUrl="http://localhost:1313"

Mac OS Version 10.14.5

[bep]

This issue surfaced after Go in 1.12 started to use MacOS' C api to adjust these limits. I did some investigation into this very confusing area that I have now forgotten, but you may look at this thread:

golang/go#30401 (comment)

I suspect you can edit /Library/LaunchDaemons/limit.maxfiles.plist and reload that.

The real fix for this issue is, of course, to add a poll based watcher alternative to fsnotify. There are implementations ready to use (Docker has one), and there is an open issue about it, but I have been hoping that others would do the work ...

/cc @nathany

[mattstratton]

OK, so with some poking around, we came across the following workaround. Running these commands allows it to work now:

hugo check ulimit
sudo sysctl -w kern.maxfiles=65536
sudo sysctl -w kern.maxfilesperproc=65536
ulimit -n 65536 65536

These changes don't persist past an OS reboot, but at least watch mode works now. I don't know if this issue should stay open, based upon the fact that @bep mentioned there is an existing issue to fit the underlying problem. Feel free to close it if you think that is best.

[deanishe]

The real fix for this issue is, of course, to add a poll based watcher alternative to fsnotify

On macOS, the best way to watch a directory tree is with FSEvents (which fsnotify has an unstable implementation of), as it doesn't require opening a whole bunch of file handles.

I'm guessing it would have to be an extended feature, though, as it requires cgo.

[nathany]

If fsnotify were to integrate FSEvents, how would that impact things for Hugo, being that FSEvents requires cgo?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. The resources of the Hugo team are limited, and so we are asking for your help.
If this is a bug and you can still reproduce this error on the master branch, please reply with all of the information you have about it in order to keep the issue open.
If this is a feature request, and you feel that it is still relevant and valuable, please tell us why.
This issue will automatically be closed in the near future if no further activity occurs. Thank you for all your contributions.

[Mebus]

Hallo,

the workaround does not work for me on Fedora 32:

sudo sysctl -w kern.maxfiles=65536
sudo sysctl -w kern.maxfilesperproc=65536
ulimit -n 65536 65536
sysctl: cannot stat /proc/sys/kern/maxfiles: No such file or directory
sysctl: cannot stat /proc/sys/kern/maxfilesperproc: No such file or directory

Mebus

[mohin7]

I found this solution from here
https://www.docsy.dev/docs/getting-started/#workaround

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.