Zeus doesn't reload code after change on macOS sierra #589

pavel-d · 2016-09-09T13:45:13Z

Description of Problem

Zeus doesn't reload code after changes on macOS sierra

System details

$ uname -a
Darwin Pavels-MacBook-Pro.local 16.0.0 Darwin Kernel Version 16.0.0: Mon Aug 29 17:56:20 PDT 2016; root:xnu-3789.1.32~3/RELEASE_X86_64 x86_64

$ ruby -v 
ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-darwin15]

Steps to Reproduce

zeus start in a new rails project
It boots fine, but one error is printed to the stdout:

zeus start
Starting Zeus server v0.15.10
[ready] [crashed] [running] [connecting] [waiting]
boot
└── default_bundle
    ├── development_environment
    │   └── prerake
    └── test_environment
        └── test_helper

Available Commands: [waiting] [crashed] [ready]
zeus rake
zeus runner (alias: r)
zeus console (alias: c)
zeus server (alias: s)
zeus generate (alias: g)
zeus destroy (alias: d)
zeus dbconsole
zeus test (alias: rspec, testrb)
2016-09-09 15:12 zeus-darwin-amd64[61609] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-22)

touch config/application.rb

Observed Behavior

Nothing happens

Expected Behavior

zeus should reload the code

The text was updated successfully, but these errors were encountered:

metcalf · 2016-09-09T21:46:00Z

I wonder if something in https://github.com/fsnotify/fsevents/blob/master/fsevents.go needs to be changed to support Sierra? Sadly that project has no tests so it's hard to isolate :-/. Zeus' test should catch this but I'd need a VM running Sierra, I'll see if I can spin one up.

pavel-d · 2016-09-11T17:01:39Z

@metcalf, seems so, I've recompiled the gem locally using go1.7.1 darwin/amd64, but the error is still there and the code reloading doesn't work :(

metcalf · 2016-09-12T23:51:25Z

@pavel-d: I got Zeus running inside a Sierra VM and can't reproduce this issue. What version of the Sierra preview are you running? Can you try creating a new, clean project with Zeus in it and see if the same issue occurs?

metcalf · 2016-09-14T19:12:58Z

@pavel-d: I upgraded my VM to the latest GM release of Sierra and still can't reproduce this. It'd be helpful to know if this repros for you on the latest Sierra release and with a new, clean project.

pavel-d · 2016-09-14T19:15:13Z

@metcalf, thanks! Sure, I'll dig a bit later this week

salamagd · 2016-09-16T04:20:28Z

Hmm, having the same problem with an existing project in Sierra, will see if I can reproduce it on a new project too.

metcalf · 2016-09-16T17:23:50Z

I just uncovered a slew of other bugs and edge cases in Zeus' file monitoring code (e.g. if you replace a watched file by moving another on top of it, it's no longer watched because the underlying inode changed). The work required to handle all these edge cases is great enough that I'm looking into ways to use Facebook's watchman in place of the current file monitoring code. I'm not sure when I'll be able to land that (or if it will work) so it'd still be great to have a repro of the Sierra issues, but if you can't repro then hopefully a solution is in the works!

metalelf0 · 2016-09-22T11:04:48Z

I'm having the same exact issues as @pavel-d, Sierra 10.12 and zeus 0.15.10. Being stuck on Rails 3 never felt so bad. :(

avegancafe · 2016-10-03T14:07:24Z

Is this still open? It seems like it's more of an issue with fsevents

mix86 · 2016-10-03T16:50:54Z

I'm having the same issue too. Sierra 10.12 (16A323), ruby 2.1.6p336, zeus 0.15.10.
After start i see

2016-10-03 19:46 zeus-darwin-amd64[27141] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-22)

and zeus does not reload code

uname -a
Darwin mixael-osx.local 16.0.0 Darwin Kernel Version 16.0.0: Mon Aug 29 17:56:20 PDT 2016; root:xnu-3789.1.32~3/RELEASE_X86_64 x86_64

zubin · 2016-10-03T19:33:41Z

@kyleholzinger Yes it's still open. The "closed" above your comment refers to the PR on the left which mentions this PR.

avegancafe · 2016-10-03T19:37:18Z

Right, was just wondering since this seems like an issue with cgo, not zeus @zubin

antifuchs · 2016-10-03T23:29:00Z

I've been researching this (since it's a blocker for our upgrade to macOS Sierra), and it seems something changed in Sierra to cause zeus's fsevents program to not work correctly anymore - the advice at facebook/react-native#9309 (comment) says to bump kernel file limits (they were very low in 10.11, and I wonder if they got bumped even lower in 10.12).

The other piece of advice is to upgrade to watchman 4.6 (we don't use watchman in Zeus, but bear with me). It includes a bunch of fsevents-related changes over 4.5, mostly in stream setup, and the handling of the UserDropped attribute. I'm installing a Sierra VM right now, to attempt to repro this too, so maybe I can dig up something more substantial.

avegancafe · 2016-10-03T23:31:33Z

@antifuchs it's not zeus' fsevents specifically, it's cgo-- this is the line that it's breaking on, which is an external call

But either way, a switch off of fsevents would fix the issue 😂

antifuchs · 2016-10-03T23:33:17Z

Oh, interesting, I thought zeus still ships its own objc fsevents wrapper! Disregard all I said, then (:

avegancafe · 2016-10-03T23:35:44Z

@antifuchs I'm not 100% positive but I think it uses the go one

dsounded · 2016-10-06T09:45:15Z

+1

pietdaniel · 2016-10-11T20:01:52Z

I was on sierra with 0.15.4 and was not experiencing this

2016-10-11 15:56 zeus-darwin-amd64[86241] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-22)

After upgrading to 0.15.10 the warning comes up.

avegancafe · 2016-10-11T20:05:52Z

Interesting, I wonder if they moved from native fsevents/self-packaged to cgo fsevents @pietdaniel

artforlife · 2016-10-30T21:44:43Z

Was anyone able to resolve this? I come from React-native side and following their steps didnt help to resolve it.

artforlife · 2016-10-30T22:33:04Z

I was able to resolve it as follows: facebook/react-native#9309 (comment)

mikz · 2016-10-31T08:46:08Z

Changing max files as suggested does not fix it for me.

pariser · 2016-10-31T23:53:35Z

@metcalf need a computer running Sierra to test? Wanna swing by one of these days?! I'm also experiencing this error and would be happy to help :-p

andrew-stripe · 2016-11-01T00:00:03Z

@pariser (or anyone else) any chance you have a Go environment set up and can try running make test under Sierra? I'm really hoping we can repro this in tests, add a test to https://github.com/fsnotify/fsevents that also repros the problem and fix it there.

andrew-stripe · 2016-11-01T00:00:42Z

(err, sorry @andrew-stripe == @metcalf, I'm signed in from my work account)

avegancafe · 2016-11-01T13:02:54Z

@andrew-stripe running now

avegancafe · 2016-11-01T15:35:57Z

@andrew-stripe @metcalf just got this:

Error: Package "/Users/Kyle/workspace/zeus" not a go package or not in GOPATH

I tried both adding it to $GOPATH as well as moving it inside of my go directory, neither worked

andrew-stripe · 2016-11-01T18:09:32Z

@kyleholzinger: Go likes a very particular directory structure. You need to set a GOPATH environment variable and place Zeus in $GOPATH/src/github.com/burke/zeus.

mcolyer · 2016-11-01T19:01:11Z

@andrew-stripe I just had a clean run on eb88f08 with go version go1.7.3 darwin/amd64 on 10.12.1.

cd rubygem && bin/rspec
.............................

Finished in 0.02885 seconds (files took 0.09957 seconds to load)
29 examples, 0 failures

andrew-stripe · 2016-11-01T20:33:59Z

That snippet of output only refers to the Ruby tests, I'd have expected to see failures (if any) in the Go tests. Those look like:

ZEUS_TEST_GEMPATH="/Users/andrew/code/go/src/github.com/burke/zeus/rubygem" GO15VENDOREXPERIMENT=1 govendor test +local
?       github.com/burke/zeus/go/clienthandler  [no test files]
?       github.com/burke/zeus/go/cmd/zeus   [no test files]
?       github.com/burke/zeus/go/config [no test files]
ok      github.com/burke/zeus/go/filemonitor    1.391s
?       github.com/burke/zeus/go/messages   [no test files]
?       github.com/burke/zeus/go/processtree    [no test files]
?       github.com/burke/zeus/go/shinylog   [no test files]
?       github.com/burke/zeus/go/statuschart    [no test files]
ok      github.com/burke/zeus/go/unixsocket 0.008s
?       github.com/burke/zeus/go/zerror [no test files]
?       github.com/burke/zeus/go/zeusclient [no test files]
ok      github.com/burke/zeus/go/zeusmaster 1.954s
?       github.com/burke/zeus/go/zeusversion    [no test files]

mcolyer · 2016-11-02T16:37:15Z

Sorry about that, here's the output I have:

ZEUS_TEST_GEMPATH="/Users/mcolyer/gopath/src/github.com/burke/zeus/rubygem" GO15VENDOREXPERIMENT=1 govendor test +local
?       github.com/burke/zeus/go/clienthandler  [no test files]
?       github.com/burke/zeus/go/cmd/zeus       [no test files]
?       github.com/burke/zeus/go/config [no test files]
ok      github.com/burke/zeus/go/filemonitor    1.256s
?       github.com/burke/zeus/go/messages       [no test files]
?       github.com/burke/zeus/go/processtree    [no test files]
?       github.com/burke/zeus/go/shinylog       [no test files]
?       github.com/burke/zeus/go/statuschart    [no test files]
ok      github.com/burke/zeus/go/unixsocket     0.008s
?       github.com/burke/zeus/go/zerror [no test files]
?       github.com/burke/zeus/go/zeusclient     [no test files]
ok      github.com/burke/zeus/go/zeusmaster     1.683s
?       github.com/burke/zeus/go/zeusversion    [no test files]

andrew-stripe · 2016-11-02T18:13:43Z

Hmm, doesn't fail tests, that makes life more difficult. That would be consistent with the explanation that it has to do with max file limits since the tests watch very few files. Recent versions of Watchman encourage you to watch an entire project directory which may help avoid this.

I'm not sure whether errors from the file watching code were surfaced in the same way in 0.15.4. It may have the same issue and is just swallowing the error message. Can anyone confirm whether they observe reloading on file changes on Sierra in 0.15.4 and not 0.15.10 (regardless of any error messages)?

pariser · 2016-11-05T05:04:11Z

I'm not seeing any reloading behavior in v0.15.4 or v0.15.10. (Nor am I seeing any error messages in v0.15.4).

andrew-stripe · 2016-11-07T02:08:33Z

Sounds like this was always broken and the old file monitor just swallowed the error messages. That's not surprising given how it worked.

My barely-informed guess is that this has to do with Zeus tracking each individual file instead of setting up a watch on the project directory. The test suite doesn't track enough files to hit any kind of limit and raising the maxfiles limit works for some projects. Assuming this is correct, the fix could be local to the filemonitor package but it wouldn't be trivial. It's also possible that something in https://github.com/fsnotify/fsevents/blob/master/fsevents.go is broken for Sierra.

We've decided to move away from Zeus in favor of our own code loader with Watchman for file monitoring. I'm happy to help review changes and answer questions but I don't expect to be investing much more in Zeus.

semenovDL · 2016-11-10T13:41:00Z

@andrew-stripe Do you have some examples of replacing zeus with watchman?

andrew-stripe · 2016-11-10T17:24:34Z

Unfortunately nothing off the shelf. We're building internal tooling that's
somewhat specific to our codebase. I'm hoping to open source the work
someday, but given that we're only about to go into internal beta that
could be quite awhile.

On Thu, Nov 10, 2016 at 5:41 AM, Alexey Kuznetsov notifications@github.com
wrote:

@andrew-stripe https://github.com/andrew-stripe Do you have some
examples of replacing zeus with watchman?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#589 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AWAzbFsp6MkIAjVlON2K0z-8lGP6KcCYks5q8x7ugaJpZM4J5EKO
.

leecunliffe · 2016-11-19T18:12:15Z

I've switched to using spring, seems it still works with the latest OSX sierra.

kenn · 2016-12-14T16:04:16Z

From what I read in facebook/react-native#9309 , it looks like it's a permission issue?

sideshowcoder · 2017-01-05T22:30:00Z

I wonder if a quick fix for that would be to monitor directory instead of files on osx, it seems to work everywhere except osx. I ran into it today as well but only on a larger rails app trying to figure out the limit now.

sideshowcoder · 2017-01-10T21:22:22Z

This seems to be related to the number of monitored files as I have 2 apps 1 causes the issue (the bigger) and a smaller which does not experience it, this is on the same machine with the same version of zeus.

To resolve this the way to go seems to be to monitor the directory instead of the files for changes and then filter in the filemonitor for events we are interested in (based on the file). I took a look at the code in filemonitor_darwin and it seems to be reasonable enough to change it here without the need to touch much else. First step so is to add a test as I wonder what the amount of files needed is. I'm gonna give this a shot when I have a Mac available (aka work).

It seems like there is a problem with fsevents when dealing with a large amount of files on OSX sierra this added test fails reliably on mac Macbook Pro, with the error ``` 2017-01-17 22:52 filemonitor.test[69205] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-22) ``` This more or less reporduces #589 #605 #607 on the basis of this being related to many files the fix would be to not monitor files individualy on OSX but rather monitor directories and filter client side, this should reduce the number of files / directories to be monitored.

siannopollo · 2017-01-19T22:06:21Z

It would be nice to see that commit by @sideshowcoder merged into master and a new gem released. Having to constantly restart zeus is kind of antithetical to its existence.

sideshowcoder · 2017-01-20T11:09:35Z

@siannopollo sadly the commit is not a fix yet, but just a failing test and a description of the fix I'm working towards, any help is greatly appricieated and I'm happy to merge as soon as it is ready 👍

sideshowcoder · 2017-01-22T16:28:54Z

@siannopollo if you fancy testing a local built, #620 now includes a potential fix for the problem, I have to do some more local testing before merging but it seems like it fixes the issue.

sideshowcoder · 2017-01-23T10:54:22Z

Please check version 0.15.13.pre and see if it fixes the problem it seems to work for my setup.

kenn · 2017-01-23T11:41:42Z

0.15.13.pre fixed the error for me! 👍 Thank you @sideshowcoder !

sideshowcoder · 2017-01-23T12:31:27Z

I'd consider this a fix then, seem to have the same success on other machines as I can see locally here 👍

siannopollo · 2017-01-23T21:13:53Z

0.15.13.pre also fixed this issue for me. Thanks @sideshowcoder

avegancafe · 2017-01-23T21:32:13Z

I feel like from now on if this fixed anyones issue just respond with a reaction to one of the other messages haha. Plz no more emails ❤️

siannopollo · 2017-02-01T18:20:44Z

@kyleholzinger Don't you find it ironic that in order to voice that opinion, you had to leave a comment that would send emails to everyone on this thread? I mean, Github provides a Subscribe/Unsubscribe button at the top of each issue if you don't want to be notified any more... ¯_(ツ)_/¯

siannopollo · 2017-04-06T15:00:04Z

@kyleholzinger But you have no control over the actions of others. So instead of trying to dictate how others should behave to suit your preferences, why don't you use the tools that Github provides to achieve the experience you desire?

siannopollo · 2017-04-06T15:21:40Z

@kyleholzinger Is classifying a seemingly civil discourse as "meaningless babel" also proper GitHub etiquette? Since this has devolved to name-calling, I'll unsubscribe from this thread now.

Iron-Ham · 2017-04-06T15:31:46Z

🎭

longkt90 mentioned this issue Sep 27, 2016

ERROR: f2d_register_rpc() => (null) (-22) on macSierra #599

Closed

semenovDL mentioned this issue Nov 17, 2016

Zeus crashing in Sierra #607

Closed

sideshowcoder mentioned this issue Jan 17, 2017

Many files problem on OSX Sierra #620

Merged

sideshowcoder closed this as completed Jan 23, 2017

Zeus doesn't reload code after change on macOS sierra #589

Zeus doesn't reload code after change on macOS sierra #589

Comments

pavel-d commented Sep 9, 2016 • edited Loading

Description of Problem

System details

Steps to Reproduce

Observed Behavior

Expected Behavior

metcalf commented Sep 9, 2016

pavel-d commented Sep 11, 2016

metcalf commented Sep 12, 2016

metcalf commented Sep 14, 2016

pavel-d commented Sep 14, 2016

salamagd commented Sep 16, 2016 • edited Loading

metcalf commented Sep 16, 2016 • edited Loading

metalelf0 commented Sep 22, 2016

avegancafe commented Oct 3, 2016

mix86 commented Oct 3, 2016 • edited Loading

zubin commented Oct 3, 2016

avegancafe commented Oct 3, 2016

antifuchs commented Oct 3, 2016

avegancafe commented Oct 3, 2016 • edited Loading

antifuchs commented Oct 3, 2016

avegancafe commented Oct 3, 2016

dsounded commented Oct 6, 2016

pietdaniel commented Oct 11, 2016

avegancafe commented Oct 11, 2016

artforlife commented Oct 30, 2016

artforlife commented Oct 30, 2016

mikz commented Oct 31, 2016

pariser commented Oct 31, 2016

andrew-stripe commented Nov 1, 2016

andrew-stripe commented Nov 1, 2016

avegancafe commented Nov 1, 2016

avegancafe commented Nov 1, 2016 • edited Loading

andrew-stripe commented Nov 1, 2016

mcolyer commented Nov 1, 2016

andrew-stripe commented Nov 1, 2016

mcolyer commented Nov 2, 2016

andrew-stripe commented Nov 2, 2016

pariser commented Nov 5, 2016

andrew-stripe commented Nov 7, 2016

semenovDL commented Nov 10, 2016

andrew-stripe commented Nov 10, 2016

leecunliffe commented Nov 19, 2016

kenn commented Dec 14, 2016

sideshowcoder commented Jan 5, 2017

sideshowcoder commented Jan 10, 2017

siannopollo commented Jan 19, 2017 • edited Loading

sideshowcoder commented Jan 20, 2017

sideshowcoder commented Jan 22, 2017

sideshowcoder commented Jan 23, 2017

kenn commented Jan 23, 2017

sideshowcoder commented Jan 23, 2017

siannopollo commented Jan 23, 2017

avegancafe commented Jan 23, 2017

siannopollo commented Feb 1, 2017

siannopollo commented Apr 6, 2017 • edited Loading

siannopollo commented Apr 6, 2017

Iron-Ham commented Apr 6, 2017

pavel-d commented Sep 9, 2016 •

edited

Loading

salamagd commented Sep 16, 2016 •

edited

Loading

metcalf commented Sep 16, 2016 •

edited

Loading

mix86 commented Oct 3, 2016 •

edited

Loading

avegancafe commented Oct 3, 2016 •

edited

Loading

avegancafe commented Nov 1, 2016 •

edited

Loading

siannopollo commented Jan 19, 2017 •

edited

Loading

siannopollo commented Apr 6, 2017 •

edited

Loading