Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reloading with a new environment and --update-env does not update the environment #3192

Closed
nitrocode opened this issue Oct 4, 2017 · 62 comments

Comments

@nitrocode
Copy link

nitrocode commented Oct 4, 2017

Edit: I didn't realize I had already posted this a few weeks ago #3164. @vmarchaud you gave the same solution but the solution doesn't work. :(

What's going wrong?

I have my staging and production environments setup in my ecosystem.config.js. When I run the following, my environment is still stuck on staging instead of production.

pm2 start ecosystem.config.js --env staging
pm2 reload all --env production --update-env

How could we reproduce this issue?

Edit: Here is a cloneable repo to reproduce the issue

  1. Set app to print the process.env.NODE_ENV variable

server.js

console.log(process.env.NODE_ENV);
  1. Setup 2 environments in ecosystem.config.js
module.exports = {
  apps: [
    // First application
    {
      name: 'app',
      script: 'server.js',
      instances: 'max',
      exec_mode: 'cluster',
      error_file: 'err.log',
      out_file: 'app.log',
      merge_logs: false,
      env: {
        NODE_ENV: 'development',
      },
      env_staging: {
        NODE_ENV: 'staging',
      },
      env_production: {
        NODE_ENV: 'production',
      },
    },
  ],
};
  1. Start app in environment 1
./node_modules/.bin/pm2 start ecosystem.config.js --env staging
  1. Reload app in environment 2
./node_modules/.bin/pm2 reload all --env production --update-env
  1. Cat one of the app files
$ cat app-1.log
staging
staging

The second line should say production because the environment was updated.

Workaround

Once the process is already started on env X, to get it to start on env Y

  1. Kill pm2 instances

    ./node_modules/.bin/pm2 stop all
    
  2. Kill pm2 god daemon

    pkill -9 PM2
    
  3. Start pm2 in the correct environment

    ./node_modules/.bin/pm2 reload all --env production --update-env
    

Supporting information

===============================================================================
--- PM2 REPORT (Wed Oct 04 2017 12:42:48 GMT-0400 (EDT)) ----------------------
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 2.7.0
node version         : 8.4.0
node path            : /Users/user/projects/app/tmp/../node_modules/.bin/pm2
argv                 : /Users/user/.nvm/versions/node/v8.4.0/bin/node,/Users/user/projects/app/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : user
uid                  : 692055800
gid                  : 1739585065
uptime               : 1min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 2.7.0
node version         : 8.4.0
node path            : /Users/user/.nvm/versions/node/v8.4.0/bin/pm2
argv                 : /Users/user/.nvm/versions/node/v8.4.0/bin/node,/Users/user/.nvm/versions/node/v8.4.0/bin/pm2,report
argv0                : node
user                 : user
uid                  : 692055800
gid                  : 1739585065
===============================================================================
--- System info --------------------------------------------
arch                 : x64
platform             : darwin
type                 : Darwin
cpus                 : Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz
cpus nb              : 8
freemem              : 70680576
totalmem             : 17179869184
home                 : /Users/user
===============================================================================
--- PM2 list -----------------------------------------------
┌──────────┬────┬─────────┬───────┬────────┬─────────┬────────┬─────┬───────────┬─────────┬──────────┐
│ App name │ id │ mode    │ pid   │ status │ restart │ uptime │ cpu │ mem       │ user    │ watching │
├──────────┼────┼─────────┼───────┼────────┼─────────┼────────┼─────┼───────────┼─────────┼──────────┤
│ app      │ 0  │ cluster │ 68381 │ online │ 1       │ 32s    │ 0%  │ 32.7 MB   │ user │ disabled │
│ app      │ 1  │ cluster │ 68382 │ online │ 1       │ 32s    │ 0%  │ 32.8 MB   │ user │ disabled │
│ app      │ 2  │ cluster │ 68418 │ online │ 1       │ 29s    │ 0%  │ 33.1 MB   │ user │ disabled │
│ app      │ 3  │ cluster │ 68419 │ online │ 1       │ 29s    │ 0%  │ 33.4 MB   │ user │ disabled │
│ app      │ 4  │ cluster │ 68455 │ online │ 1       │ 26s    │ 0%  │ 34.1 MB   │ user │ disabled │
│ app      │ 5  │ cluster │ 68456 │ online │ 1       │ 26s    │ 0%  │ 33.5 MB   │ user │ disabled │
│ app      │ 6  │ cluster │ 68492 │ online │ 1       │ 22s    │ 0%  │ 33.0 MB   │ user │ disabled │
│ app      │ 7  │ cluster │ 68493 │ online │ 1       │ 22s    │ 0%  │ 33.8 MB   │ user │ disabled │
└──────────┴────┴─────────┴───────┴────────┴─────────┴────────┴─────┴───────────┴─────────┴──────────┘
===============================================================================
--- Daemon logs --------------------------------------------
/Users/user/.pm2/pm2.log last 20 lines:
PM2        | 2017-10-04 12:42:25: Stopping app:app id:_old_4
PM2        | 2017-10-04 12:42:25: Stopping app:app id:_old_5
PM2        | 2017-10-04 12:42:25: App name:app id:_old_4 disconnected
PM2        | 2017-10-04 12:42:25: App [app] with id [_old_4] and pid [68012], exited with code [0] via signal [SIGINT]
PM2        | 2017-10-04 12:42:25: App name:app id:_old_5 disconnected
PM2        | 2017-10-04 12:42:25: App [app] with id [_old_5] and pid [68030], exited with code [0] via signal [SIGINT]
PM2        | 2017-10-04 12:42:25: pid=68012 msg=process killed
PM2        | 2017-10-04 12:42:25: Starting execution sequence in -cluster mode- for app name:app id:6
PM2        | 2017-10-04 12:42:25: pid=68030 msg=process killed
PM2        | 2017-10-04 12:42:25: Starting execution sequence in -cluster mode- for app name:app id:7
PM2        | 2017-10-04 12:42:26: App name:app id:6 online
PM2        | 2017-10-04 12:42:26: App name:app id:7 online
PM2        | 2017-10-04 12:42:29: Stopping app:app id:_old_6
PM2        | 2017-10-04 12:42:29: Stopping app:app id:_old_7
PM2        | 2017-10-04 12:42:29: App name:app id:_old_6 disconnected
PM2        | 2017-10-04 12:42:29: App [app] with id [_old_6] and pid [68048], exited with code [0] via signal [SIGINT]
PM2        | 2017-10-04 12:42:29: App name:app id:_old_7 disconnected
PM2        | 2017-10-04 12:42:29: App [app] with id [_old_7] and pid [68066], exited with code [0] via signal [SIGINT]
PM2        | 2017-10-04 12:42:29: pid=68048 msg=process killed
PM2        | 2017-10-04 12:42:29: pid=68066 msg=process killed

Please copy/paste the above report in your issue on https://github.com/Unitech/pm2/issues
@nitrocode
Copy link
Author

@vmarchaud is this a bug?

@BrandonCopley
Copy link

I am seeing lots of environment issues as well with this version of PM2. It appears to be ignoring anything in the "env_" fields.

@vmarchaud
Copy link
Contributor

Please read the docs here, its explained :

You want to restart your process and pick-up changes in your ecosystem.json ? Use pm2 reload ecosystem.json --update-env

@BrandonCopley
Copy link

BrandonCopley commented Oct 10, 2017 via email

@nitrocode
Copy link
Author

nitrocode commented Oct 11, 2017

@vmarchaud Im not sure if I understand. If I launch the app in env X and I want to reload the app in env Y, using --env Y --update-env, it does not seem to work. It will maintain env X unless I kill the PM2 god daemon and restart pm2 with the env Y.

Do my reproduction steps make sense?

@nitrocode
Copy link
Author

nitrocode commented Oct 17, 2017

@vmarchaud On step 4 in my original post, my command is equivalent to the one that you suggested from within the docs. Shouldn't this command reload pm2 in the correct environment with my above ecosystem config?

./node_modules/.bin/pm2 reload all --env production --update-env

The top comment on the PM2 - Process File page in Disqus has a thread of people with the same issue. They suggest this command instead.

./node_modules/.bin/pm2 startOrGracefulReload prod.json --env production --update-env

Edit: works fine now... how strange... Turns out that using reload all was the culprit. Using the below command worked fine.

./node_modules/.bin/pm2 reload ecosystem.config.js --env production --update-env

@cmal
Copy link

cmal commented Apr 3, 2018

It seems pm2 startOrReload ecosystem.config.js --update-env not working.

But pm2 restart ... works.

@nitrocode
Copy link
Author

@vmarchaud judging by comments and votes it seems to be an issue. Do you mind reopening this ticket?

@kerimdzhanov
Copy link

Any updates on this issue? --update-env still doesn't work.
Please re-open the issue.

@wallet77 wallet77 reopened this Apr 25, 2018
@wallet77
Copy link
Contributor

I will try to make some tests soon.
If I understand option --update-env works with restart command but when we use ecosystem.config.js it doesn't update env.
Am I right ?

@kerimdzhanov
Copy link

kerimdzhanov commented Apr 25, 2018

Hi! Thanks for a quick reply!

Unfortunately nope, --update-env doesn't work neither with restart nor with reload, nor gracefull..., startOr... and so on...

We're trying to make deployment on our server, the directory structure is something like:

v1/
v2/
v3/
current -> ./v3 (symlink)

In the beginning I thought that the problem is in a symlink, but then I've changed the directory to v2/ directly then run pm2 start ecosystem.config.js and when I'm cding to v3/ and trying to restart pm2 ecosystem, it still starts files from the v2/ directory. Changing any file's contents, changing version in package.json doesn't help.

@kerimdzhanov
Copy link

Yay! I found this article and make it work!

http://pm2.keymetrics.io/docs/tutorials/capistrano-like-deployments

The key difference is that I had to explicitly set cwd option in my ecosystem.config.js to /an/absolute/path/to/current (the symlink).

Thus,

pm2 startOrGracefulReload current/ecosystem.config.js --update-env

works gracefully as expected!

I'm sorry, it was my bad.

Thanks for your attention! You're doing a great job!

@cmal
Copy link

cmal commented Apr 27, 2018

we have cwd , but startOrReload not work

@delino12
Copy link

Try so many but this works pm2 restart 1 --update-env
or Better still pm2 reload 1 --update-env
Both allow you to pick up the changes on your env ASAP

@dendrochronology
Copy link

I can't get any combination of the above suggestions to make --update-env work, even with arbitrarily simple "hello world" examples.

The only way I can get pm2 to pick up on a config change is to delete a specific app process or kill the daemon entirely. I guess I'll just add that to my deploy scripts.

@nitrocode
Copy link
Author

@delino12 the restart didn't work for me. I wrote up a quick proof of concept repo here using pm2 3.0.3.

@delino12
Copy link

delino12 commented Jul 31, 2018

@nitrocode you have to kill the server. addition to that, app will pickup the new changes when ever a server is restarted. Please make sure you restart or reload both the .env and the application files changes.

@dendrochronology
Copy link

@delino12 by server, do you mean the pm2 daemon, or a managed node process?

I'm still trying to suss out whether this is a documentation bug, a widespread misunderstanding, or the --update-env flag just straight-up not working as advertised.

@nitrocode
Copy link
Author

nitrocode commented Aug 1, 2018

@delino12 I tried both of these commands and they did not start the app in the intended environment

pm2 reload all --env production --update-env
pm2 restart all --env production --update-env

I tried replacing all with 0 and that also did not work. Can you share the full command you used? Try the repo I posted above if you get a chance.

@delino12
Copy link

delino12 commented Aug 15, 2018

@nitrocode here is a quick step to handle env process
I will recommend you stop all app process first, make sure you stop the process from running, most application process is still running on the old env variables because the process is still running and server has cache the old env variables which will make changes not effective.

pm2 restart all --update-env
pm2 restart [appname] --update-env

http://pm2.keymetrics.io/docs/usage/environment

@nitrocode
Copy link
Author

I don't understand the difference in commands between the ones you suggested and the ones I tried. I tried the same commands you suggested with the repo above with the same results. Do you get different results?

@jcald1
Copy link

jcald1 commented Aug 21, 2018

Neither command works for me. I tried updating my pm2 installation, stopping all pm2 processes, killing all node process, then

pm2 reload ecosystem.config.js && pm2 restart all --update-env --env production && pm2 logs
and also tried:
pm2 reload ecosystem.config.js && pm2 reload all --update-env --env production && pm2 logs

It still starts with port 3011 instead of 3012. My code does read the 3011 port using process.env.PORT.

@viniwrubleski
Copy link

The only way I managed to make it work was the command:

pm2 stop [appname] && pm2 del [appname] && pm2 start ecosystem.config.js --env production

@fuss86
Copy link

fuss86 commented Oct 11, 2018

@viniwrubleski workaround works.

Pure reload / restart does not work as documented in docs.

@arihantdaga
Copy link

Its a bug.. Not working as explained in docs.
I just want to update an environment variable but its not updating.
I used pm2 reload process.json --update-env --only myappname

and in my configuration file (process.json) I updated a variable -

"env_test": {
"HOTEL_URL": "something"
}

to

"env_test": {
"HOTEL_URL": "somethingnew"
}

but even after reloading value of that environment variable inside my code is printing the same. as previous.

@baitkul
Copy link

baitkul commented Nov 13, 2018

I am getting this behavior only when I set exec_mode: 'cluster', but if I am not using cluster mode, switching environments works fine.

@IrosTheBeggar
Copy link

I'm still seeing this problem on my server. Is there any plan to fix the reload commands?

@appurist
Copy link

appurist commented Nov 19, 2018

I ran into this issue today. Updated the ecosystem file and ran a pm2 reload server --env production --update-env and no joy, old environment. Tried pm2 stop server, pm2 restart server, etc. I wasn't switching environments, but I had updated one of the environment variables with a critical config change. I could not get it to pick up the change from the ecosystem file, and I had limited time (I was about to be late for work) so no time to google and find this report. So I eventually did a pm2 stop server, updated the shell environment, and manually ran node server.js and I'm praying it's still up later tonight when I return.

I'll try some of the suggestions in this thread, but one thing that is unclear from this issue is whether the team has confirmed there is a bug with having environment changes picked up, and if not, what the correct command is to force an app to be reloaded with environment changes. Is there a correct way to do this already, or if there's a bug, what is the recommended workaround?

@vitalets
Copy link

Still having this issue.

pm2 -v
3.3.1

@knoxcard
Copy link

@vitalets - can you use the latest version and report your results?

@vitalets
Copy link

vitalets commented Mar 17, 2019

@vitalets - can you use the latest version and report your results?

Worked with pm2 3.4.0 👍
Env variable updated even without --update-env option =)

pm2 reload /srv/ecosystem.config.js --only my-app

@knoxcard
Copy link

knoxcard commented Mar 17, 2019

@vitalets - recently, I've noticed this behavior, but had not had the chance to confirm with multiple with/without tests. On behalf of myself and everyone else, Thank You!

@Djalmar
Copy link

Djalmar commented Apr 29, 2019

For me the only way to get this working(on version 3.5.0) is:

pm2 restart ecosystem.config.js --update-env --env production

@srosset81
Copy link

I'm on v3.5.1, with cluster mode, and the above command still doesn't work.
I have to kill and restart pm2 to make it work.

@Dartv
Copy link

Dartv commented Aug 14, 2019

Any update on this? As @srosset81 said, it doesn't work on 3.5.1

@fushihara
Copy link

fushihara commented Aug 24, 2019

To ensure that both the system environment variables and the environment variables defined in ecosystem.json are updated, you will need to run two commands. This is very inconvenient and inconsistent.
pm2 restart ecosyste.json --update-env && pm2 restart <pid> --update-env

it working fork mode.run at jar program.pm2 version 3.5.1

現状、システムの環境変数とjsonで定義された2種類の環境変数を確実に更新するには二回コマンドを実行する必要があると思いますが、これはとても不便です。

@InfinityCosmodrome
Copy link

After trying many of the above suggestions, not bringing some results, I found out that property mode: 'development' in my webpack.config.js file was that which defined the NODE_ENV variable, setting it always that value. That's why I couldn't update that variable via pm2, which used the bundle file of the built version (with Webpack) of my app.

@elrumordelaluz
Copy link

As already read on comments on this issue and related, seems that --update-env works well when updating a value, but doesn't remove the variable in case doesn't exists anymore. As commented here the only way to do it is pm2 delete <app> and pm2 start <app>. Hope this will be fixed in some way.

@nebular
Copy link

nebular commented Sep 11, 2020

Can anybody suggest a deamon like PM2 that does not try to reinvent ENV VARS, something that has been working flawlessly since 1970? (edit - later I understood how ecosystem works and it makes sense :) comment was the heat of the moment )

@DataTables
Copy link

So if you want to change an environment variable value, you need to use the ecosystem file, rather than just pm2 restart [appName]? I can't get pm2 restart [appName] --update-env to work at all. I've got a bunch of other stuff in the ecosystem file that I don't want to restart.

@vitalets
Copy link

Can anybody suggest a deamon like PM2 that does not try to reinvent ENV VARS, something that has been working flawlessly since 1970?

Use docker :)

@mannharleen
Copy link

@v4.4.0 pm2 this is still not fixed. The issue should be reopened.
I can confirm the following:

// while this works
pm2 reload /usr/src/app/ecosystem.config.js --update-env --only x-process

// using the pm2 API does not
pm2.reload('x-process',
    {
        updateEnv: true ///// this part here has no effect !
    },
    (err, proc) => {
        if (!err) {
            pm2.disconnect();
        } else {
            throw err
        }
    }
);

@MZanggl
Copy link

MZanggl commented Dec 21, 2020

To ensure that both the system environment variables and the environment variables defined in ecosystem.json are updated, you will need to run two commands. This is very inconvenient and inconsistent.
pm2 restart ecosystem.json --update-env && pm2 restart <pid> --update-env

This worked for me.

  • MY_VARIABLE=newvalue pm2 restart/reload ecosystem.config.json --update-env will only update the environment variables set in the config file. It won't set the new value to MY_VARIABLE as --update-env is being ignored...
  • MY_VARIABLE=newvalue pm2 restart/reload <name of app> --update-env will update MY_VARIABLE=newvalue, but the environment variables from the config file remain the way they were when starting the app initially.

Conclusion: Only use one of the two until pm2 restart ecosystem.config.json --update-env gets fixed.

@mannawar
Copy link

pm2 restart all worked for me as i gave sudo priviliges i.e sudo su

@kiriland
Copy link

I am currently using the latest PM2. I barely got test env vars running but after I added production vars it does not want to update them at all. I tried all the possible methods but this just does not work.
The strange part that it updates 2 out of 3 env vars.. I will be switching to docker as advised.

@SantiagoSchez
Copy link

For me the only way to get this working(on version 3.5.0) is:

pm2 restart ecosystem.config.js --update-env --env production

As of 2021 this is the only solution that worked for me; have to make it explicit the environment to run so that env vars get refreshed from file.

@ktvdev
Copy link

ktvdev commented Dec 13, 2021

Still having this issue as of v5.1.2

I use docker for everything that I can for a variety of reasons but I run a couple of services to do with my CI/CD pipeline that are easier to manage on bare metal due to OS interactions and the like. I like using .env files to manage my deployments (personal preference, standardised across most of my projects) and PM2 still doesn't seem to want to reload those vars with either restart/reload or using --update-env.

My current workaround to avoid having to worry about dynamically calling processes by name to perform operations on them is to just run pm2 delete <process> and then bring them back with pm2 resurrect.

This way I can both target services manually or batch process updates / testing by killing all the services that need to be updated then running the resurrect command to bring them all back with no additional input required.

@komanton
Copy link

komanton commented Aug 30, 2022

Just for the history.

Firstly, carefully read this documentation. https://pm2.io/docs/runtime/best-practices/environment-variables/

Secondly, it is not explicitly highlighted but you can not mixed approaches according to environment variable update and initial starting of pm2 command.

There are only two regimes:

  1. pm2 is started with env vars via command line, like $ ENV_VAR=somethingnew-1 pm2 reload app . And then the only one way to update it is run the command: $ ENV_VAR=somethingnew-2 pm2 reload app --update-env
  2. pm2 is started with env vars via Ecosystem file, like $ pm2 reload ecosystem.config.js . Where ecosystem.config.js contains ENV_VAR=somethingnew-1 (see format of Ecosystem file format in doc). And then the only one way to update it is run the command: $ pm2 reload ecosystem.config.js. Where ecosystem.config.js contains ENV_VAR=somethingnew-2

Other words, if you started pm2 with the command $ pm2 reload ecosystem.config.js. And, trying to update env var with command $ ENV_VAR=somethingnew-2 pm2 reload --update-env. So, env var will be not updated!!!

Note: the only one edge case when you can mix this two approaches is when you want to migrate between them gracefully. To do this, the only need to move all env vars from Ecosystem file to command like or vise versa from command line to Ecosystem file.

@Arctomachine
Copy link

What about variables that app itself reads from local .env files?
I use command pm2 startOrReload pm2.config.js --only xxx and the app keeps reading old values from .env file. Adding --update-env makes no difference, values from file are still outdated.

@vladaman
Copy link

I am also experiencing the same issue with 5.3.0

I loaded env variables from ecosystem.config.json; no matter what I try, it won't reload variables.

@s1davide
Copy link

I am also experiencing the same issue with 5.3.0

I loaded env variables from ecosystem.config.json; no matter what I try, it won't reload variables.

Use pm2 reload ./ecosystem.config.js

@thenbthoughts
Copy link

One reason may also be like.

suppose you have 2 repo.
/repo-backend-1
/repo-backend-2

In first repo, there is a env file for example.
S3_SECRET=efgh

In second repo, there is a env file for example.
S3_SECRET=abcd

And you are starting repo-1 by pm2.
So first it would take S3_SECRET=efgh.
Then you are starting the second repo-2 by pm2.
So then it would take S3_SECRET=abcd.

So in the first repo, the env variable will be abcd and the credentials are mismatched.

So make sure that both env variable are same.

@mkalygin
Copy link

mkalygin commented Sep 2, 2024

As of September 2024, this is still a problem. I found a bug with config extension, fixed it, and covered with tests. My use case was with dotenvx so I included an example usage of it too.

#5881

@melon
Copy link

melon commented Dec 31, 2024

After hours of trying, I found something interesting:

For v5.4.3

❌: not working
✅: working

  • pm2 restart ecosystem.config.js --update-env ❌
  • pm2 restart [id] --update-env ✅
  • pm2 reload ecosystem.config.js --update-env ❌
  • pm2 reload [id] --update-env ✅
  • pm2 startOrReload ecosystem.config.js --update-env ❌
  • pm2 startOrRestart ecosystem.config.js --update-env ❌
  • pm2 stop ecosystem.config.js && pm2 start ecosystem.config.js --update-env ❌
  • pm2 stop [id] && pm2 start [id] ❌
  • pm2 stop [id] && pm2 start [id] --update-env ✅

So let's wrap it up, for env to successfully change, you should:

  1. use "--update-env"
  2. use [id] instead of ecosystem.config.js

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests