Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cluster mode and EADDRINUSE #4225

Closed
entrptaher opened this issue Mar 31, 2019 · 11 comments
Closed

Cluster mode and EADDRINUSE #4225

entrptaher opened this issue Mar 31, 2019 · 11 comments
Labels

Comments

@entrptaher
Copy link

This might be a duplicate issue however all linked issues are not able to resolve this issue I am having right now with the particular build version of https://github.com/vanila-io/spectrum.

What's going wrong?

  • Express API >> Program exits with EADDRINUSE, status is still online.
  • Bull Queue >> Program does not restart on error, status is still online.
  • Removing exec_mode does not work.
  • Changing cluster to cluster_mode does not work.

How could we reproduce this issue?

Unfortunately this happens on a production version, not on a development version. I have zero clues about reproduction right now.

Supporting information

--- PM2 report ----------------------------------------------------------------
Date                 : Sun Mar 31 2019 12:58:52 GMT+0000 (UTC)
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 3.2.9
node version         : 9.11.2
node path            : /root/.nvm/versions/node/v9.11.2/bin/pm2
argv                 : /root/.nvm/versions/node/v9.11.2/bin/node,/root/.nvm/versions/node/v9.11.2/lib/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : root
uid                  : 0
gid                  : 0
uptime               : 18min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 3.2.9
node version         : 9.11.2
node path            : /root/.nvm/versions/node/v9.11.2/bin/pm2
argv                 : /root/.nvm/versions/node/v9.11.2/bin/node,/root/.nvm/versions/node/v9.11.2/bin/pm2,report
argv0                : node
user                 : root
uid                  : 0
gid                  : 0
===============================================================================
--- System info --------------------------------------------
arch                 : x64
platform             : linux
type                 : Linux
cpus                 : AMD EPYC 7501 32-Core Processor
cpus nb              : 2
freemem              : 991858688
totalmem             : 4143312896
home                 : /root
===============================================================================
--- PM2 list -----------------------------------------------
┌──────────┬────┬─────────┬─────────┬──────┬────────┬─────────┬────────┬──────┬────────────┬──────┬──────────┐
│ App name │ id │ version │ mode    │ pid  │ status │ restart │ uptime │ cpu  │ mem        │ user │ watching │
├──────────┼────┼─────────┼─────────┼──────┼────────┼─────────┼────────┼──────┼────────────┼──────┼──────────┤
│ api      │ 0  │ 0.34.0  │ cluster │ 7642 │ online │ 0       │ 18m    │ 0.1% │ 108.2 MB   │ root │ disabled │
│ api      │ 3  │ 0.34.0  │ cluster │ 7655 │ online │ 0       │ 18m    │ 0.1% │ 109.0 MB   │ root │ disabled │
│ athena   │ 2  │ 0.34.0  │ cluster │ 7654 │ online │ 0       │ 18m    │ 0%   │ 109.2 MB   │ root │ disabled │
│ athena   │ 1  │ 0.34.0  │ cluster │ 7647 │ online │ 0       │ 18m    │ 0%   │ 108.5 MB   │ root │ disabled │
│ chronos  │ 4  │ 0.34.0  │ cluster │ 7671 │ online │ 0       │ 18m    │ 0%   │ 108.4 MB   │ root │ disabled │
│ chronos  │ 6  │ 0.34.0  │ cluster │ 7695 │ online │ 0       │ 18m    │ 0.1% │ 108.9 MB   │ root │ disabled │
│ hermes   │ 13 │ 0.34.0  │ cluster │ 7800 │ online │ 0       │ 18m    │ 0%   │ 107.9 MB   │ root │ disabled │
│ hermes   │ 12 │ 0.34.0  │ cluster │ 7785 │ online │ 0       │ 18m    │ 0.1% │ 108.9 MB   │ root │ disabled │
│ hyperion │ 5  │ 0.34.0  │ cluster │ 7676 │ online │ 0       │ 18m    │ 0.1% │ 108.2 MB   │ root │ disabled │
│ hyperion │ 7  │ 0.34.0  │ cluster │ 7713 │ online │ 0       │ 18m    │ 0.1% │ 109.0 MB   │ root │ disabled │
│ mercury  │ 8  │ 0.34.0  │ cluster │ 7727 │ online │ 0       │ 18m    │ 0.1% │ 108.2 MB   │ root │ disabled │
│ mercury  │ 10 │ 0.34.0  │ cluster │ 7755 │ online │ 0       │ 18m    │ 0.1% │ 109.2 MB   │ root │ disabled │
│ vulcan   │ 9  │ 0.34.0  │ cluster │ 7741 │ online │ 0       │ 18m    │ 0%   │ 109.2 MB   │ root │ disabled │
│ vulcan   │ 11 │ 0.34.0  │ cluster │ 7769 │ online │ 0       │ 18m    │ 0.1% │ 109.2 MB   │ root │ disabled │
└──────────┴────┴─────────┴─────────┴──────┴────────┴─────────┴────────┴──────┴────────────┴──────┴──────────┘
===============================================================================
--- Daemon logs --------------------------------------------
/root/.pm2/pm2.log last 20 lines:
PM2        |     at Function.Module.runMain (internal/modules/cjs/loader.js:697:11)
PM2        |     at startup (internal/bootstrap/node.js:201:19)
PM2        |     at bootstrapNodeJSCore (internal/bootstrap/node.js:516:3)
PM2        |   errno: 'EADDRINUSE',
PM2        |   code: 'EADDRINUSE',
PM2        |   syscall: 'listen',
PM2        |   address: '::',
PM2        |   port: 3006 }
PM2        | Error: listen EADDRINUSE :::3006
PM2        |     at Server.setupListenHandle [as _listen2] (net.js:1346:14)
PM2        |     at listenInCluster (net.js:1387:12)
PM2        |     at Server.listen (net.js:1475:7)
PM2        |     at Function.listen (/root/spectrum/node_modules/express/lib/application.js:618:24)
PM2        |     at module.exports._reactLoadable2.default.preloadAll.then (/root/spectrum/build-hyperion/webpack:/hyperion/index.js:208:7)
PM2        |     at <anonymous>
PM2        |     at process._tickCallback (internal/process/next_tick.js:182:7)
PM2        |     at Function.Module.runMain (internal/modules/cjs/loader.js:697:11)
PM2        |     at startup (internal/bootstrap/node.js:201:19)
PM2        |     at bootstrapNodeJSCore (internal/bootstrap/node.js:516:3)
PM2        | 2019-03-31T12:41:54: PM2 log: pm2 has been killed by signal, dumping process list before exit...

Here is the process.json file, I tested with cluster and cluster_mode stated on #3946 (comment) .

{
  "apps": [
    {
      "name": "api",
      "script": "yarn",
      "exec_mode":"cluster_mode",
      "instances":2,
      "args": "start:api"
    },
    {
      "name": "athena",
      "script": "yarn",
      "exec_mode":"cluster_mode",
      "instances":2,
      "args": "start:athena"
    },
    {
      "name": "chronos",
      "script": "yarn",
      "exec_mode":"cluster_mode",
      "instances":2,
      "args": "start:chronos"
    },
    {
      "name": "hyperion",
      "script": "yarn",
      "exec_mode":"cluster_mode",
      "instances":2,
      "args": "start"
    },
    {
      "name": "mercury",
      "script": "yarn",
      "exec_mode":"cluster_mode",
      "instances":2,
      "args": "start:mercury"
    },
    {
      "name": "vulcan",
      "script": "yarn",
      "exec_mode":"cluster_mode",
      "instances":2,
      "args": "start:vulcan"
    },
    {
      "name": "hermes",
      "script": "yarn",
      "exec_mode":"cluster_mode",
      "instances":2,
      "args": "start:hermes"
    }
  ]
}

In sum:

  • Removing exec_mode does not work.
  • Changing cluster to cluster_mode does not work.
  • Program exits with EADDRINUSE, status is still online.
  • Program does not restart on error, status is still online.

All related issues are either open or does not resolve this issue,

@shtefcs
Copy link

shtefcs commented Mar 31, 2019

Tahy, thanks for posting this issue here and hope we can solve it once for all. If anyone can help, give any points, would mean a lot to us.

@entrptaher
Copy link
Author

To add more to above post, it's really weird the error goes to /root/.pm2/pm2.log instead of the error log. Maybe the program checks for port and manually rejects with EADDRINUSE.

@entrptaher
Copy link
Author

entrptaher commented Apr 10, 2019

I tried

  • updating pm2 version latest 3.5.0.
  • downgrading pm2 version to 3.0.0, 2.5.0.

The process exits but does not restart either. Really fraustrating experience.

Also:

  • pm2 scale api 2 or any other scaling simply fails.

image

Without cross-env (and yarn/npm script together)

However, today I tried a different script as well, previously the start script had cross-env and it was failing (as mentioned above). But when I tried without cross-env, the program ran properly.

I used this code to run,

 NODE_ENV=production pm2 start build-api/main.js --name=api --instances=2

We need cross-env though to maintain env across multiple processes properly. But it's a good lead.

image

image

I am not sure how it will behave without cross-env, only can say after experiments.

@bam-tbf
Copy link

bam-tbf commented Apr 16, 2019

I was also having this issue and I was able to fix it by running pm2 delete on the apps that were having the EADDRINUSE issue and recreating them with pm2 start. I am not sure why this would fix it, but my only guess is that something saved in the pm2 env/config was causing an issue. For context, this issue started happening after I upgraded some packages on my ubuntu machine.

I confirmed that this permanently fixed the issue by running pm2 save and rebooting. The clustered app started and ran as expected without errors.

@zsolt-dev
Copy link

I have the same issue for very long time on all my projects.

my pm2.config.js:

module.exports = {
  /**
   * Application configuration section
   * http://pm2.keymetrics.io/docs/usage/application-declaration/
   */

  apps: [
    {
      name: 'media-app',
      script: 'npm',
      args: "run serve",
      exec_mode: 'cluster_mode',
      exec_interpreter: 'node',
      wait_ready: true,
      listen_timeout: 10000,
      instances: 2,
      max_memory_restart: '200M',
      max_restarts: 2, // if restarts this many times in min_uptime time, app is considered errored and will not start
      min_uptime: 5000,
      watch: false,
      log_date_format: 'YYYY-MM-DD HH:mm Z',
      env: {
        NODE_ENV: 'production',
      },
    },
  ],
};

in git hooks/post-receive I have this:

pm2 startOrReload pm2.config.js --update-env

It fails consistently after each reload.

What I do to fix it after each reaload:

pm2 delete media-app
pm2 startOrReload pm2.config.js --update-env

This issue deservers a fix after years of not functioning properly.

@munroe7
Copy link

munroe7 commented Aug 21, 2019

I was getting this same error when I tried to cluster my app that was already running with PM2. I deleted the running instance and started it again using the cluster param and no more errors!

@matharuajay
Copy link

matharuajay commented Sep 26, 2019

I am having the same problem. I even tried uninstalling and reinstalling pm2, deleting all the old files .pm2 folder but i still get EADDRINUSE error in cluster mode.
Here is my config file

{
"apps": [{
"script": "npm",
"args": "start",
"name": "backend",
"instances": "max",
"exec_mode": "cluster_mode",
"env": {
"NODE_ENV": "production"
},
"merge-logs": true,
"error_file": "/var/log/pm2/backend-err.log",
"out_file": "/var/log/pm2/backend-out.log"
}]
}

@stale
Copy link

stale bot commented May 23, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label May 23, 2020
@wdifruscio
Copy link

Is this fixed in newer versions? Experienced it on a production website after restarting pm2 with an ecosystem file and experienced a bunch of downtime

@stale stale bot removed the stale label May 27, 2020
@stale
Copy link

stale bot commented Jun 26, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Jun 26, 2020
@stale stale bot closed this as completed Jul 10, 2020
@t4t5
Copy link

t4t5 commented Nov 1, 2022

For those who are still experiencing this issue, I found that you can't run an npm script in the ecosystem.config.json file.

e.g. if you have a Next app that runs:

"script": "npm",
"args" : "start",

you should replace it with a call to the script directly:

"script": "./node_modules/.bin/next",
"args" : "start",

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

8 participants