enhancement: cli should distinguish between stdout and stderr

[exx8]

Motivation ("The Why")

Whenever npm prints error, it uses stdout. This is wrong. stdout should be used for standard output, for a regular run of the program.
The error should always be printed via stderr, and there many benefits of that, to name a few:

1. It easier for a programmed execution to detect errors (just see if something was printed in stderr, no string stuff) .
2. Many shells support a customized unique way to print error, we should use that instead of using the current universal style (print red ERR!) which might be incompatible for some systems.
3. for even a minute operations, sometimes the error just takes over because all the traceback, rendering the successful results effective invisible (especially in shells that are limited in the number of printed lines).

Example

whenever npm install doesn't find a package, npm should print to stderr.

How

using the standard error stream

Current Behaviour

prints red ERR in stdout.
the stderr remains empty through out the run of the program even if the command fails

Desired Behaviour

The error message will be printed to stderr.

References

For example the following message will be printed in stderr and not in stdout:
npm ERR! node-gd@1.4.0 install: node-gyp rebuild
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the node-gd@1.4.0 install script 'node-gyp rebuild'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the node-gd package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! node-gyp rebuild
npm ERR! You can get information on how to open an issue for this project with:
npm ERR! npm bugs node-gd
npm ERR! Or if that isn't available, you can get their info via:
npm ERR! npm owner ls node-gd
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR! /root/lospec/npm-debug.log

• n/a

[lukekarrys]

I have a few thoughts on this:

• There are a few open issues ([BUG] NPM 7.x broke the "--json" CLI parameter #2740) which specifically request errors to not be printed to stderr when in --json mode. I don't like the idea of stderr/stdout switching based on an seemingly unrelated flag like json

• All of our log messages are currently printed to stderr, anything else printed to stderr is not parseable unless done in --silent mode

• I think there is some context based on what types of errors we have that is missing from this discussion. For example,
  npm install rrreact --json should have a parseable output saying that the module was a 404. I think this should be on stdout regardless of the output mode.

• But unknown errors (maybe from a bug in npm or an error condition we dont properly handle) should probably go to stderr as you suggest. It feels not great to open every error to a discussion about what "type" of error it is, which I think would invite more RFCs instead of allowing us to settle on a decision.

[lukekarrys]

From the RFC meeting on 2/23 the next steps are to implement the following:

• all log output go to stderr (this is how it is currently)
• all non-recoverable error output go to stderr
• recoverable error output (such as registry 404, ERESOLVE, etc) should return on stdout in all output modes, but should be guaranteed to be parseable json in json mode

[kariharju]

I would also suggest removing the constant "shouting" of npm ERR!.

Only the lines that actually state some errors should contain text towards that. The original issue reference is a good example of this. It should be enough to highlight the single line npm ERR! Exit status 1, all the other lines are not errors.

This helps support side find the actual lines that matter a lot faster and the logs become.. well "less alarming" 😉

[lukekarrys]

I'm going to close this since since a lot of progress was made in standardizing and separating what goes on stderr vs stdout. There is still some work left to do, but followup items will be tracked in npm/statusboard#829