Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cluster mode suddenly start giving EADDRINUSE #3336

Closed
randomparanoia opened this issue Dec 7, 2017 · 6 comments
Closed

Cluster mode suddenly start giving EADDRINUSE #3336

randomparanoia opened this issue Dec 7, 2017 · 6 comments

Comments

@randomparanoia
Copy link

What's going wrong?

App only works on fork mode. Listens on port but is unable to spawn other instances. I'm out of ideas where the cause is... already tried via command line same issue when trying cluster mode.

Configuration is as follows:

{
  "apps" : [
    {
      "name"      : "app",
      "script"    : "npm",
      "args"      : "run serve:ssr",
      "watch"     : [
        "dist"
      ],
      "exec_mode" : "cluster",
      "instances" : "max",
      "env"       : {
        "PORT"     : 8085,
        "NODE_ENV" : "production"
      }
    }
  ]
}

Supporting information

/root/.pm2/pm2.log last 15 lines:
PM2        | events.js:136
PM2        |       throw er; // Unhandled 'error' event
PM2        |       ^
PM2        |
PM2        | Error: listen EADDRINUSE :::4000
PM2        |     at Object._errnoException (util.js:1031:13)
PM2        |     at _exceptionWithHostPort (util.js:1052:20)
PM2        |     at Server.setupListenHandle [as _listen2] (net.js:1367:14)
PM2        |     at listenInCluster (net.js:1408:12)
PM2        |     at Server.listen (net.js:1496:7)
PM2        |     at Function.listen (/opt/app/dist/server.js:228158:24)
PM2        |     at Object.<anonymous> (/opt/app/dist/server.js:222014:5)
PM2        |     at __webpack_require__ (/opt/app/dist/server.js:20:30)
PM2        |     at Object.defineProperty.value (/opt/app/dist/server.js:63:18)
PM2        |     at Object.<anonymous> (/opt/app/dist/server.js:66:10)

Please run the following command (available on PM2 >= 2.6)

$ pm2 report
===============================================================================
--- PM2 REPORT (Thu Dec 07 2017 01:20:39 GMT+0100 (CET)) ----------------------
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 2.8.0
node version         : 9.2.0
node path            : undefined
argv                 : /root/.nvm/versions/node/v9.2.0/bin/node,/root/.nvm/versions/node/v9.2.0/lib/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : root
uid                  : 0
gid                  : 0
uptime               : 103min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 2.8.0
node version         : 9.2.0
node path            : /root/.nvm/versions/node/v9.2.0/bin/pm2
argv                 : /root/.nvm/versions/node/v9.2.0/bin/node,/root/.nvm/versions/node/v9.2.0/bin/pm2,report
argv0                : node
user                 : root
uid                  : 0
gid                  : 0
===============================================================================
--- System info --------------------------------------------
arch                 : x64
platform             : linux
type                 : Linux
cpus                 : Intel Core Processor (Haswell, no TSX)
cpus nb              : 4
freemem              : 7147151360
totalmem             : 8171847680
home                 : /root
===============================================================================
--- PM2 list -----------------------------------------------
┌──────────────┬────┬──────┬───────┬────────┬─────────┬────────┬─────┬───────────┬──────┬──────────┐
│ App name     │ id │ mode │ pid   │ status │ restart │ uptime │ cpu │ mem       │ user │ watching │
├──────────────┼────┼──────┼───────┼────────┼─────────┼────────┼─────┼───────────┼──────┼──────────┤
│ app │ 0  │ fork │ 19266 │ online │ 0       │ 7m     │ 0%  │ 38.5 MB   │ root │ enabled  │
└──────────────┴────┴──────┴───────┴────────┴─────────┴────────┴─────┴───────────┴──────┴──────────┘
===============================================================================
--- Daemon logs --------------------------------------------
/root/.pm2/pm2.log last 20 lines:
PM2        | 2017-12-07 01:15:45: pid=23432 msg=failed to kill - retrying in 100ms
PM2        | 2017-12-07 01:15:45: pid=23443 msg=failed to kill - retrying in 100ms
PM2        | 2017-12-07 01:15:45: pid=23432 msg=failed to kill - retrying in 100ms
PM2        | 2017-12-07 01:15:45: pid=23443 msg=failed to kill - retrying in 100ms
PM2        | 2017-12-07 01:15:45: pid=23432 msg=failed to kill - retrying in 100ms
PM2        | 2017-12-07 01:15:45: pid=23443 msg=failed to kill - retrying in 100ms
PM2        | 2017-12-07 01:15:46: pid=23432 msg=failed to kill - retrying in 100ms
PM2        | 2017-12-07 01:15:46: pid=23443 msg=failed to kill - retrying in 100ms
PM2        | 2017-12-07 01:15:46: pid=23432 msg=failed to kill - retrying in 100ms
PM2        | 2017-12-07 01:15:46: pid=23443 msg=failed to kill - retrying in 100ms
PM2        | 2017-12-07 01:15:46: pid=23432 msg=failed to kill - retrying in 100ms
PM2        | 2017-12-07 01:15:46: pid=23443 msg=failed to kill - retrying in 100ms
PM2        | 2017-12-07 01:15:46: Process with pid 23432 still alive after 1600ms, sending it SIGKILL now...
PM2        | 2017-12-07 01:15:46: Process with pid 23443 still alive after 1600ms, sending it SIGKILL now...
PM2        | 2017-12-07 01:15:46: App [app] with id [9] and pid [23432], exited with code [0] via signal [SIGKILL]
PM2        | 2017-12-07 01:15:46: App name:app id:9 disconnected
PM2        | 2017-12-07 01:15:46: App name:app id:10 disconnected
PM2        | 2017-12-07 01:15:46: App [app] with id [10] and pid [23443], exited with code [0] via signal [SIGKILL]
PM2        | 2017-12-07 01:15:46: pid=23432 msg=process killed
PM2        | 2017-12-07 01:15:46: pid=23443 msg=process killed
@vmarchaud
Copy link
Contributor 

===============================================================================
--- PM2 list -----------------------------------------------
┌──────────────┬────┬──────┬───────┬────────┬─────────┬────────┬─────┬───────────┬──────┬──────────┐
│ App name     │ id │ mode │ pid   │ status │ restart │ uptime │ cpu │ mem       │ user │ watching │
├──────────────┼────┼──────┼───────┼────────┼─────────┼────────┼─────┼───────────┼──────┼──────────┤
│ app │ 0  │ fork │ 19266 │ online │ 0       │ 7m     │ 0%  │ 38.5 MB   │ root │ enabled  │
└──────────────┴────┴──────┴───────┴────────┴─────────┴────────┴─────┴───────────┴──────┴──────────┘

Your application is already started in fork, you need to delete it and then start it in cluster.

@randomparanoia
Copy link
Author

Actually that report was incorrect since I tested with fork and cluster by deleting current started application. I was able to run it only by command line. Init json/yml file was unable to run.

@randomparanoia
Copy link
Author

Figured out: If someone comes accross be aware of commands you pass on your script. thanks

@sophister
Copy link

sophister commented Jan 8, 2019 •
edited
Loading

pm2 version: 3.2.2

Same problem here. I use pm2 reload pm2.json qa to start/reload my app, it is ok first time, and pm2 list show this:

┌────────────────┬────┬─────────┬─────────┬───────┬────────┬─────────┬────────┬──────┬───────────┬────────┬──────────┐
│ App name       │ id │ version │ mode    │ pid   │ status │ restart │ uptime │ cpu  │ mem       │ user   │ watching │
├────────────────┼────┼─────────┼─────────┼───────┼────────┼─────────┼────────┼──────┼───────────┼────────┼──────────┤
│ NODE-RRD-PC-v1 │ 25 │ 1.0.0   │ cluster │ 21695 │ online │ 0       │ 22s    │ 1.5% │ 44.5 MB   │ tomcat │ disabled │
│ NODE-RRD-PC-v1 │ 26 │ 1.0.0   │ cluster │ 21702 │ online │ 0       │ 22s    │ 1.9% │ 44.4 MB   │ tomcat │ disabled │
│ NODE-RRD-PC-v1 │ 27 │ 1.0.0   │ cluster │ 21709 │ online │ 0       │ 22s    │ 1.7% │ 42.6 MB   │ tomcat │ disabled │
│ NODE-RRD-PC-v1 │ 28 │ 1.0.0   │ cluster │ 21720 │ online │ 0       │ 22s    │ 1.5% │ 42.7 MB   │ tomcat │ disabled │
│ NODE-RRD-PC-v1 │ 29 │ 1.0.0   │ cluster │ 21731 │ online │ 0       │ 22s    │ 3%   │ 42.7 MB   │ tomcat │ disabled │
└────────────────┴────┴─────────┴─────────┴───────┴────────┴─────────┴────────┴──────┴───────────┴────────┴──────────┘

then I use this same command pm2 reload pm2.json qa to reload app, and this time, only one process can start normally, other process will exit with message Error: listen EADDRINUSE :::9013. And pm2 list shows that, other processes are restarted due to this error:

┌────────────────┬────┬─────────┬─────────┬───────┬────────┬─────────┬────────┬──────┬───────────┬────────┬──────────┐
│ App name       │ id │ version │ mode    │ pid   │ status │ restart │ uptime │ cpu  │ mem       │ user   │ watching │
├────────────────┼────┼─────────┼─────────┼───────┼────────┼─────────┼────────┼──────┼───────────┼────────┼──────────┤
│ NODE-RRD-PC-v1 │ 25 │ 1.0.0   │ cluster │ 25414 │ online │ 83      │ 0s     │ 0%   │ 39.3 MB   │ tomcat │ disabled │
│ NODE-RRD-PC-v1 │ 26 │ 1.0.0   │ cluster │ 25407 │ online │ 83      │ 0s     │ 0%   │ 43.5 MB   │ tomcat │ disabled │
│ NODE-RRD-PC-v1 │ 27 │ 1.0.0   │ cluster │ 22179 │ online │ 7       │ 68s    │ 0.2% │ 43.5 MB   │ tomcat │ disabled │
│ NODE-RRD-PC-v1 │ 28 │ 1.0.0   │ cluster │ 25421 │ online │ 79      │ 0s     │ 0%   │ 34.4 MB   │ tomcat │ disabled │
│ NODE-RRD-PC-v1 │ 29 │ 1.0.0   │ cluster │ 25400 │ online │ 76      │ 0s     │ 0%   │ 47.5 MB   │ tomcat │ disabled │
└────────────────┴────┴─────────┴─────────┴───────┴────────┴─────────┴────────┴──────┴───────────┴────────┴──────────┘

and pm2 logs show this:

25|NODE-RRD-PC-v1  | 2019-01-08 16:07 +08:00: Error: listen EADDRINUSE :::9013
25|NODE-RRD-PC-v1  | 2019-01-08 16:07 +08:00:     at Server.setupListenHandle [as _listen2] (net.js:1286:14)
25|NODE-RRD-PC-v1  | 2019-01-08 16:07 +08:00:     at listenInCluster (net.js:1334:12)
25|NODE-RRD-PC-v1  | 2019-01-08 16:07 +08:00:     at Server.listen (net.js:1421:7)
25|NODE-RRD-PC-v1  | 2019-01-08 16:07 +08:00:     at LeekKoaApplication.listen (/opt/app/node-rrd-pc/node_modules/koa/lib/application.js:65:19)
25|NODE-RRD-PC-v1  | 2019-01-08 16:07 +08:00:     at LeekApp.run (/opt/app/node-rrd-pc/node_modules/leekjs/core/LeekApp.js:195:33)
25|NODE-RRD-PC-v1  | 2019-01-08 16:07 +08:00:     at process._tickCallback (internal/process/next_tick.js:68:7)
25|NODE-RRD-PC-v1  | 2019-01-08 16:07 +08:00:     at Function.Module.runMain (internal/modules/cjs/loader.js:744:11)
25|NODE-RRD-PC-v1  | 2019-01-08 16:07 +08:00:     at startup (internal/bootstrap/node.js:285:19)
25|NODE-RRD-PC-v1  | 2019-01-08 16:07 +08:00:     at bootstrapNodeJSCore (internal/bootstrap/node.js:739:3)
25|NODE-RRD-PC-v1  | 2019-01-08 16:07 +08:00: [uncaughtException] { Error: listen EADDRINUSE :::9013
25|NODE-RRD-PC-v1  | 2019-01-08 16:07 +08:00:     at Server.setupListenHandle [as _listen2] (net.js:1286:14)
25|NODE-RRD-PC-v1  | 2019-01-08 16:07 +08:00:     at listenInCluster (net.js:1334:12)
25|NODE-RRD-PC-v1  | 2019-01-08 16:07 +08:00:     at Server.listen (net.js:1421:7)
25|NODE-RRD-PC-v1  | 2019-01-08 16:07 +08:00:     at LeekKoaApplication.listen (/opt/app/node-rrd-pc/node_modules/koa/lib/application.js:65:19)
25|NODE-RRD-PC-v1  | 2019-01-08 16:07 +08:00:     at LeekApp.run (/opt/app/node-rrd-pc/node_modules/leekjs/core/LeekApp.js:195:33)
25|NODE-RRD-PC-v1  | 2019-01-08 16:07 +08:00:     at process._tickCallback (internal/process/next_tick.js:68:7)
25|NODE-RRD-PC-v1  | 2019-01-08 16:07 +08:00:     at Function.Module.runMain (internal/modules/cjs/loader.js:744:11)
25|NODE-RRD-PC-v1  | 2019-01-08 16:07 +08:00:     at startup (internal/bootstrap/node.js:285:19)
25|NODE-RRD-PC-v1  | 2019-01-08 16:07 +08:00:     at bootstrapNodeJSCore (internal/bootstrap/node.js:739:3)
25|NODE-RRD-PC-v1  | 2019-01-08 16:07 +08:00:   errno: 'EADDRINUSE',
25|NODE-RRD-PC-v1  | 2019-01-08 16:07 +08:00:   code: 'EADDRINUSE',
25|NODE-RRD-PC-v1  | 2019-01-08 16:07 +08:00:   syscall: 'listen',
25|NODE-RRD-PC-v1  | 2019-01-08 16:07 +08:00:   address: '::',
25|NODE-RRD-PC-v1  | 2019-01-08 16:07 +08:00:   port: 9013 }
PM2                | App [NODE-RRD-PC-v1:25] exited with code [1] via signal [SIGINT]
PM2                | App [NODE-RRD-PC-v1:25] starting in -fork mode-
PM2                | App [NODE-RRD-PC-v1:25] online

my pm2.json config file is:

{
    "apps" : [
        {
            "name"        : "NODE-RRD-PC-v1",
            "script"      : "index.js",
            "args"        : [],
            "node_args"   : "--harmony",
            "merge_logs"  : true,
            "error_file" : "pm2.error.log",
            "out_file" : "pm2.out.log",
            "log_date_format" : "YYYY-MM-DD HH:mm Z",
            "instances" : 5,
            "exec_mode" : "cluster",
            "cwd"         : ".",
            "max_memory_restart" : "200M",
            "env": {
                "NODE_ENV": "production"
            },
            "env_development": {
                "NODE_ENV": "development"
            },
            "env_production" : {
                "NODE_ENV": "production"
            },
            "env_qa" : {
                "NODE_ENV": "qa"
            },
            "env_stage" : {
                "NODE_ENV" : "stage"
            }
        }
    ]
}

@sophister
Copy link

It seems, when I run pm2 reload pm2.json qa first time, pm2 uses cluster mode to start my app, as I see logs from pm2 logs :

PM2      | App [NODE-RRD-PC-v1:0] starting in -cluster mode-
PM2      | App [NODE-RRD-PC-v1:0] online
PM2      | App [NODE-RRD-PC-v1:1] starting in -cluster mode-
PM2      | App [NODE-RRD-PC-v1:1] online
PM2      | App [NODE-RRD-PC-v1:2] starting in -cluster mode-
PM2      | App [NODE-RRD-PC-v1:2] online
PM2      | App [NODE-RRD-PC-v1:3] starting in -cluster mode-
PM2      | App [NODE-RRD-PC-v1:3] online
PM2      | App [NODE-RRD-PC-v1:4] starting in -cluster mode-
PM2      | App [NODE-RRD-PC-v1:4] online

But when I run pm2 reload pm2.json qa second time, pm2 logs show this:

PM2                    | App [NODE-RRD-PC-v1:1] exited with code [1] via signal [SIGINT]
PM2                    | App [NODE-RRD-PC-v1:1] starting in -fork mode-
PM2                    | App [NODE-RRD-PC-v1:1] online
PM2                    | App [NODE-RRD-PC-v1:2] exited with code [1] via signal [SIGINT]
PM2                    | App [NODE-RRD-PC-v1:2] starting in -fork mode-
PM2                    | App [NODE-RRD-PC-v1:2] online
PM2                    | App [NODE-RRD-PC-v1:3] exited with code [1] via signal [SIGINT]
PM2                    | App [NODE-RRD-PC-v1:3] starting in -fork mode-
PM2                    | App [NODE-RRD-PC-v1:3] online

@bam-tbf
Copy link

bam-tbf commented Jan 15, 2019

If anyone is still having this issue, please reference this ticket: #3946

You need to set exec_mode to "cluster_mode" not "cluster" if you want cluster mode.

@entrptaher entrptaher mentioned this issue Mar 31, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@sophister @vmarchaud @randomparanoia @bam-tbf