Watch no longer working

[gshively11]

PM2 0.12.4

Node v0.12.0

[{
"name": "[scrubbed]",
"script": "./scripts/start.js",
"watch": true,
"exec_mode": "fork_mode",
"instances": 1,
"node_args": "--debug=5858",
"cwd": "/[scrubbed]/current"
}]

2015-03-02 09:58:11: [PM2][WORKER] Started with refreshing interval: 30000
2015-03-02 09:58:11: [[[[ PM2/God daemon launched ]]]]
2015-03-02 09:58:11: BUS system [READY] on port /[scrubbed]/.pm2/pub.sock
2015-03-02 09:58:11: RPC interface [READY] on port /[scrubbed]/.pm2/rpc.sock
2015-03-02 09:58:11: Starting execution sequence in -fork mode- for app name:node-inspector id:0
2015-03-02 09:58:11: App name:node-inspector id:0 online
2015-03-02 09:59:23: Starting execution sequence in -fork mode- for app name:[scrubbed] id:1
2015-03-02 09:59:23: App name:[scrubbed] id:1 online
2015-03-02 09:59:26: Starting execution sequence in -fork mode- for app name[scrubbed] id:2
2015-03-02 09:59:26: App name:[scrubbed] id:2 online
2015-03-02 11:07:07: Process with pid 7048 still not killed, retrying...
2015-03-02 11:07:07: App name:[scrubbed] id:1 exited
2015-03-02 11:07:07: Process with pid 7048 killed
2015-03-02 11:07:07: Starting execution sequence in -fork mode- for app name:[scrubbed] id:1
2015-03-02 11:07:07: App name:[scrubbed] id:1 online

[gshively11]

Verified it doesn't work on pm2 0.12.7 also

[soyuka]

Are you on osx?

Just tried and indeed there is an issue with fsevents, maybe related to node 0.12.

Temporary fix:

  "watch_options" : {
      "useFsEvents": false
  }

Or don't install the chokidar fsevents module in the pm2 directory :

rm -r node_modules/chokidar && npm i chokidar --no-optional

I'll try to dig deeper and submit an issue to chokidar or fsevents about this.

[gshively11]

Yes, on OSX. The temporary fix you mentioned does not seem to work.

[kryz81]

The same problem on ubuntu+vagrant, pm2 doesn't see file changes

pm2 0.12.7
node 0.12.0

[soyuka]

We tested this yesterday with ubuntu/debian and there was no issues. Did you test the workarounds from below?

[gshively11]

I should mention that I'm using vagrant as well. And I'm watching a synched folder, so it's a symlink.

[gshively11]

So I mispoke when I said I was on OSX. My host is OSX, but PM2 is running on a CentOS 6.5 VM. My bad.

[soyuka]

@gshively11 @ChrisPiechowicz Could you please run the watcher test and give me the output in a gist?

Tests are located under:

bash test/bash/watch.sh

in the pm2 module directory.

Thanks!

[gshively11]

Ran it as the user which runs our node apps:

$pm2 kill
[PM2] Stopping PM2...
[PM2] deleteProcessId process id 0
[PM2] deleteProcessId process id 1
[PM2] deleteProcessId process id 2
[PM2] All processes has been stopped and deleted
[PM2] PM2 stopped

# Determine wget / curl
which wget > /dev/null
if [ $? -eq 0 ]
then
    http_get="wget"
else
    echo -e "\033[31mYou need wget to run this test \033[0m";
    exit 1;
fi

function fail {
  echo -e "######## \033[31m  ✘ $1\033[0m"
  exit 1
}

function success {
  echo -e "\033[32m------------> ✔ $1\033[0m"
}

function spec {
  RET=$?
  sleep 0.3
  [ $RET -eq 0 ] || fail "$1"
  success "$1"
}

function ispec {
  RET=$?
  sleep 0.3
  [ $RET -ne 0 ] || fail "$1"
  success "$1"
}

function should {
    sleep 0.5
    OUT=`$pm2 prettylist | grep -o "$2" | wc -l`
    [ $OUT -eq $3 ] || fail "$1"
    success "$1"
}

cd $file_path

function waituntil {
  for (( i = 0; i <= $1; i++ )); do
      sleep 0.2
      echo -n "."
  done
  echo ""
}

echo -e "\033[1mRunning tests:\033[0m"
Running tests:

#####################
# Watch for changes #
#####################

>server-watch.js
test/bash/watch.sh: line 22: server-watch.js: Permission denied

$pm2 kill
[PM2] Spawning PM2 daemon
[PM2] Success
[PM2] Stopping PM2...
[PM2] [WARN] No process found
[PM2] All processes has been stopped and deleted
[PM2] PM2 stopped

cp server-watch.bak.js server-watch.js
cp: cannot create regular file `server-watch.js': Permission denied

$pm2 start server-watch.js --watch
[PM2] Spawning PM2 daemon
[PM2] Success
[PM2] Process server-watch.js launched
┌──────────────┬────┬──────┬──────┬────────┬───────────┬────────┬─────────────┬──────────┐
│ App name     │ id │ mode │ PID  │ status │ restarted │ uptime │      memory │ watching │
├──────────────┼────┼──────┼──────┼────────┼───────────┼────────┼─────────────┼──────────┤
│ server-watch │ 0  │ fork │ 7691 │ online │         0 │ 0s     │ 14.469 MB   │  enabled │
└──────────────┴────┴──────┴──────┴────────┴───────────┴────────┴─────────────┴──────────┘
 Use `pm2 info <id|name>` to get more details about an app

should 'process should be watched' 'watch: true' 1
$pm2 prettylist | grep -o "$2" | wc -l
------------> ✔ process should be watched

echo "console.log('test');" >> server-watch.js
test/bash/watch.sh: line 32: server-watch.js: Permission denied

sleep 1

cat server-watch.js
var http = require('http');

http.createServer(function(req, res) {
  res.writeHead(200);
  res.end('hey');
}).listen(8000);
$pm2 list
┌──────────────┬────┬──────┬──────┬────────┬───────────┬────────┬─────────────┬──────────┐
│ App name     │ id │ mode │ PID  │ status │ restarted │ uptime │      memory │ watching │
├──────────────┼────┼──────┼──────┼────────┼───────────┼────────┼─────────────┼──────────┤
│ server-watch │ 0  │ fork │ 7691 │ online │         0 │ 2s     │ 14.469 MB   │  enabled │
└──────────────┴────┴──────┴──────┴────────┴───────────┴────────┴─────────────┴──────────┘
 Use `pm2 info <id|name>` to get more details about an app

should 'process should have been restarted' 'restart_time: 1' 1
$pm2 prettylist | grep -o "$2" | wc -l
########   ✘ process should have been restarted

[kryz81]

@soyuka I can't find watch.sh in pm2 dir, I don't have "test" directory..

[akloeber]

Same here on Ubuntu 14.04.1 LTS (inside a Vagrant Box as well as on DigitalOcean droplet) running PM2 v0.12.5 on node v0.10.25.
In this case the watched folder is a symlink too.
Output of the test script, see https://gist.github.com/akloeber/d18147f903353a11178b.

Setting useFsEvents to false does not help.

[sheymann]

👍

[karan]

Can also confirm that watch isn't working for me.

Ubuntu 14.04.2
Node v0.12.0

[gshively11]

Any idea when this issue might be fixed?

[soyuka]

@gshively11 I can't reproduce anymore, and I haven't find the source of this issue.

It seems to be very random and I've got absolutely no clue of what's going on here... I had the same error on my computer but I cleaned up my nvm node versions, updated iojs and it worked as before :| with node 0.12/0.10...

When I had the issue I took pm2 back to the first version that had the watch feature but still had this issue! The only thing that had change between then and now is that I had installed iojs.

[akloeber]

@soyuka It still does not work in pm2@0.12.11 but I found out the following behavior while investigating:

The folder structure below my PM2's cwd is like that:

<PM2_CWD>
  bin
    app.js
    ...
  node_modules
    ...
  lib
    ...
  package.json

If I touch bin/app.js which is PM2's script entry point (or even another script inside the bin/ folder) PM2's watch mechanism works and the node is restarted. But if I touch a script in any other folder like lib/main.js there is no reaction in PM2.

Furthermore I was not able to bash test/bash/watch.sh as this script is no more present.

Environment:
pm2 0.12.11
node v0.10.25
Ubuntu 14.04.1 LTS

[soyuka]

Seems like it's working to me, please update your pm2/node versions if you still have issues.

[joemckie]

I'm having the same problems.

My versions are:

pm2 v0.14.3
node v0.10.39
Ubuntu 14.04.2 LTS (GNU/Linux 3.13.0-55-generic x86_64)

touch triggers the change event, but I need to be in the vagrant ssh session. Touching from my computer's terminal session does not work, nor does updating the file from a text editor. The test/bash/watch.sh isn't present for me to test.

FWIW, I'm running on the latest OSX, with vagrant v1.7.2.

[soyuka]

Is it possible that there are some symlinks and that followSymlinks option is not set properly?

[joemckie]

Possibly, I'm not 100% sure how vagrant sets up its synchronised folders. I've just tried setting the watch_options -> followSymlinks option to both true and false, but neither makes a difference.

[joemckie]

I just downloaded the development version of PM2 and ran the watch.sh test. It gave the same result as @gshively11 above.

[Bazze]

I'm having the exact same issue, cannot get the watch functionality to work reliably with Vagrant (1.7.2) and NFS shared folders on Ubuntu 14.04 (guest) and OSX 10.10.4 (host).

PM2: v0.14.3
NodeJS: v0.12.4

[soyuka]

@Bazze maybe you should open an issue on the chokidar repository about this.

[Bazze]

@soyuka: Thank you! Good idea, but I searched some amongst their issues instead and found this ticket (#242), seems to be pretty much related. I solve my issue for now by using the "usePolling": true watch option. Not optimal, but it works.

[soyuka]

@Bazze Oh interesting, I searched but never found this one.
Why are you thinking that usePolling is not optimal? If NFS doesn't manage file events it's the only solution.

#1313 related

[Bazze]

@soyuka: It's not optimal as in it's checking for changes at a fixed time interval, instead of "real-time". But for now I can live with waiting a few seconds for the changes to be discovered. :)

[soyuka]

100 ms (default polling option) is very low, consider increasing to save CPU usage.

@Bazze hope it'll do the job, no better solution :/

[Bazze]

It's 100ms by default? Takes 1-2 seconds at least for me before it notices any changes, weird. I'm gonna experiment a bit with the interval!

@soyuka: So far I can't see why it wouldn't (except that it could be a bit faster) ;)

[Bazze]

@soyuka: Regarding your commit 77bb8d0, shouldn't the option be usePolling and not polling? At least usePolling is what I use and what seems to be in the chokidar documentation.

[soyuka]

@Bazze thanks, friday evening.....

[Bazze]

@soyuka I know the feeling, let's call it a day! 🍻

[DenisGorbachev]

Similar, but somewhat different error: https://gist.github.com/DenisGorbachev/523f3a937faec86b31dd

starfall@nx:~/workspace/PM2$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04 LTS"

starfall@nx:~/workspace/PM2$ git show
commit 733d631301394630637b3cdbfa449ab1ac158f10
Author: Unitech <strzelewicz.alexandre@gmail.com>
Date:   Wed Jul 22 17:09:19 2015 +0200

    0.14.5

[MihaelIsaev]

Watch & reload doesn't work on Debian 9.1x64 on DigitalOcean droplet. I have a simple helloWorld project and trying to change a project file in a text editor or by pulling changes from git repo and nothing happens :(

[yass0016]

--watch is not working on Debian 9

[yPhil-gh]

I'm on Linux, and

"watch_options" : {
      "useFsEvents": false
}

Worked for me ; Thanks!

[catalsdevelop]

pm2 v2.9.3
macosx 10.11.6

"watch_options" : { "useFsEvents": false }

didn't work

[Mizumaki]

In my case, watch option not working when you start the pm2 which is once status: stopped.

I mean, if it is the first process, watch option works.
However, if you run pm2 stop ecosystem.config.js, and then pm2 start ecosystem.config.js, watch option won't work.