OSX: Reasonably large number of directories causes "FSEventStreamStart: register_with_server: ERROR"

[minikomi]

Boot Bug Report

Platform details

• Platform (macOS, Linux, Windows): Mac OSX
• Platform version: Sierra 10.12.6
• JRE/JDK version: "1.8.0_121"

Boot details

BOOT_CLOJURE_NAME=org.clojure/clojure
BOOT_CLOJURE_VERSION=1.8.0
BOOT_VERSION=2.7.1

build.boot content (if applicable)

(set-env!
 :resource-paths #{"resources"})

Description

When there are a reasonably large number of directories in resources, running boot, even without any tasks, causes many errors to the tune of:

2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2017-09-20 11:45 java[9766] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)

More directories (eg. 3000) causes the task to never finish / error out.

                                                            Boot.main                         Boot.java:  257
                                                                  ...
                                                        boot.App.main                          App.java:  491
                                                     boot.App.runBoot                          App.java:  399
          org.projectodd.shimdandy.impl.ClojureRuntimeShimImpl.invoke       ClojureRuntimeShimImpl.java:  150
          org.projectodd.shimdandy.impl.ClojureRuntimeShimImpl.invoke       ClojureRuntimeShimImpl.java:  159
                                                                  ...
                                                      boot.main/-main                          main.clj:  206
                                                   boot.main/-main/fn                          main.clj:  206
                                                                  ...
                                                     boot.user/eval32  boot.user3887463377436593005.clj:    5
                                                                  ...
                                                   boot.core/set-env!                          core.clj:  873
                                                                  ...
                                                   clojure.core/swap!                          core.clj: 2253
                                                   clojure.core/swap!                          core.clj: 2263
                                                                  ...
                                                   boot.core/init!/fn                          core.clj:  776
                                                boot.core/configure!*                          core.clj:  311
                                                                  ...
                                                         boot.core/fn                          core.clj:  797
                                             boot.core/set-user-dirs!                          core.clj:  196
                                                                  ...
                                                 boot.core/watch-dirs                          core.clj:  744
                                                                  ...
          org.projectodd.shimdandy.impl.ClojureRuntimeShimImpl.invoke       ClojureRuntimeShimImpl.java:  117
          org.projectodd.shimdandy.impl.ClojureRuntimeShimImpl.invoke       ClojureRuntimeShimImpl.java:  124
                                                                  ...
                                            boot.watcher/make-watcher                       watcher.clj:  125
                                                 boot.watcher/service                       watcher.clj:   94
                                           boot.watcher/service/doreg                       watcher.clj:   93
                                      boot.watcher/register-recursive                       watcher.clj:   65
                                      boot.watcher/register-recursive                       watcher.clj:   62
                                                    boot.watcher/fn/G                       watcher.clj:   25
                                                      boot.watcher/fn                       watcher.clj:   52
              com.barbarysoftware.watchservice.WatchableFile.register                WatchableFile.java:   43
              com.barbarysoftware.watchservice.WatchableFile.register                WatchableFile.java:   34
com.barbarysoftware.watchservice.MacOSXListeningWatchService.register  MacOSXListeningWatchService.java:   52
                                               java.lang.Thread.start                       Thread.java:  714
                                              java.lang.Thread.start0                        Thread.java
java.lang.OutOfMemoryError: unable to create new native thread
clojure.lang.ExceptionInfo: unable to create new native thread
    file: "/var/folders/dm/63qgv3gx0_96qkqbz6xpdzgh0000gn/T/boot.user3887463377436593005.clj"

Steps to reproduce

cd resources
seq 600 | xargs mkdir
cd ..
boot

[yannvanhalewyn]

Thanks for the ref, @arichiardi, removing ~/.boot/cache solved it for me (for now)

[arichiardi]

This also looks like the cache does not evenly distribute files in some corner case. Compared to a poor key choice while sharding maybe.

[yannvanhalewyn]

Care to expand on that? (new to boot)
Sadly, it came back after 2-3 restarts and unable to build bigger projects (multi figwheel builds for example). A more permanent solution would be appreciated. Is there a way for me to not have boot monitor the resource paths for example?

[dustingetz]

boot watch --help documents the regex pattern for the watch

[dustingetz]

i solved this by only having package.json in resource-paths and then running "cd target; yarn" after the resources are copied (assuming node_modules was the problematic resource)

[yannvanhalewyn]

Thanks, I was already checking out exclusion options for the watch task and didn't get it to work. Do you know if the repl task uses the watch task behind the scenes? Because I don't when launching the repl

[yannvanhalewyn]

So indeed no watch task was running, that wasn't the issue. I had a stroll through the boot source, and this is what I seem to understand:

• Boot syncs src and resources paths out to the tempdirs (filesets)
• This happens once when the task starts and then a watcher takes over (this one is causing the FSEvent errors)

Now boot apparently has support for ignore files, .bootignore. Which is a great solution! Sadly these are only taken into account in the first step - the watchers just recursively watch entire directories.

So for now (and future readers) I fixed this issue for me using boot --disable-watchers ..., which is great for figwheel like things, but could potentially break other building blocks. I will open a PR to attempt to use .bootignore regexes in the watchers.