npm

[isaacs]

Continuation of this thread

I just want to try and make sure we fix it in a way that both you and the Homebrew maintainers can be happy with the result and to avoid our mutual users from getting confused or annoyed.

I believe that a huge source of difficulty, frustration, and wasted time here has been from attempts to implement solutions without fully understanding the problems. npm is not simple. Homebrew is not simple. Having one package manager installed by another, such that its API remains intact, is remarkably tricky -- in fact, it's likely impossible without a fairly deep understanding of both systems.

The original npm recipe was written without such an understanding of npm, and npm has changed significantly since then. My recipe was written without a deep understanding of Homebrew. We in the problem-solving industry tend pitch solutions, and then focus on what's good/bad about the solution, rather than on the problems, which is a great way to solve easy problems, but a terrible way to solve tricky ones. This problem will not be solved with me writing Homebrew ruby and you writing npm JavaScript, or else the issue would have been closed 6 months ago.

So let's talk about problems, with an eye on figuring out the landscape before we build on it. Eventually, I believe that the solution will be a matter of specifying an appropriate configuration, maybe making some code changes to npm, and running some make command in the npm code folder. That should be easy enough to implement in a recipe, if someone who actually knows ruby and Homebrew takes it on, and I'm happy to do my part in npm as necessary.

My goals in discussing these problems is not to pitch solutions, but only to label user expectations, several of which are currently not met.

1. the things npm installs

For Homebrew's purposes, npm installing things directly into $prefix is unacceptable. With the current recipe, it installs with the following settings:

• root (node modules and npm metadata): prefix/share/npm/lib
• binroot (executables): prefix/share/npm/bin

I'd like to propose that we also add:

• manroot (man pages): prefix/share/npm/man

An install caveat can tell the user what to do with these things, and npm will warn if the lib is not in the NODE_PATH, bin is not in PATH, and man is not in MANPATH.

2. npm updating npm

Quite often, a user will run into a situation with npm which is a bug that has been fixed in the latest release. Or, it may be a bug that I'm not familiar with, but the log output indicates that they're on a version of npm which is quite old, and I'd rather not waste time rolling back just to find out that it's already been handled properly.

When they come to me with that scenario, my response is generally, "Do npm i npm to get the latest version, and let me know if it's still broken."

However, with the current homebrew setup, this does not actually solve the problem, and even after installing npm with npm, they still have the older version of npm when they run npm.

This needs to work:

brew install npm
npm -v  --> old version
npm install npm
npm -v --> new version

3. brew uninstall npm

Doing brew uninstall npm should remove all the "cruft" that npm creates, as well as the npm code itself, but leave the programs installed by npm intact.

brew install npm
npm install express
brew uninstall npm
which npm --> not found
which express --> path/to/express

This means that the .npm folder (wherever that ends up living) needs to stick around (since that's where the module guts are, express in this example), but the cache and temp should be destroyed, along with the npm package, if it was self-installed subsequently. That is:

brew install npm
npm install express
npm install npm  #self-installation
brew uninstall npm
which npm --> not found
which express --> path/to/express

4. npm JavaScript interface

When npm is installed with homebrew, it should be available from within nodejs programs, provided of course that the npm module install target is added to the NODE_PATH.

brew install npm
export NODE_PATH=$(brew --prefix)/share/npm/lib # or whatever this ends up being
node
> semver = require("npm/utils/semver") // returns the npm semver util.

Furthermore, this should be updated when npm self-updates.

-=-=-=-=-=-

Is this list complete? What should be added to it? Are there items on this list that you believe are not actually problems?

[camillol]

I've never used npm, so correct me if I'm wrong.

You want npm-installed packages to stick around when npm is removed.
You want npm to be equivalent to an npm-installed package (so you can do npm i npm).
You want npm to be removed when you do brew uninstall npm.

This is a contradiction.

[isaacs]

This is a contradiction.

The third sentence would be an exception to the first. Currently, that's how make uninstall works -- it removes npm, and the cache and temp files, but not the other packages it's installed.

None of this is hard to implement. But implementing it before understanding the goals and constraints is how we got into this mess in the first place.

[camillol]

Sure, but under the current design, homebrew formulas cannot customize uninstallation. That's a constraint.

Now, for the goals: why exactly do people want to install npm using homebrew? Wouldn't it make more sense to blacklist npm and show a message saying "npm is its own package manager, use 'curl http://npmjs.org/install.sh | sh' to install"?

[isaacs]

Sure, but under the current design, homebrew formulas cannot customize uninstallation. That's a constraint.

I see. That does make this a more tricky problem, and that's good information.

why exactly do people want to install npm using homebrew?

I don't know. Probably because they install everything else with homebrew, and don't actually have much experience compiling or installing software on the command line by hand.

Wouldn't it make more sense to blacklist npm and show a message saying...

I am totally ok with that. In fact, that would be pretty awesome, and make my life a lot easier. (It'd be even better if it could check to make sure that curl and node are installed before printing the message!)

The downside, of course, is that it's basically just us throwing up our hands and saying it can't be solved. So that feels a bit like failure, and sucks.

But like I said, I'm not really comfortable pitching/debating solutions until at least Mike has weighed in on the problem space and we've all had at least a day or two to meditate on it.

[camillol]

I wouldn't call it failure. Not duplicating the job of other package managers is actually part of Homebrew's design.

curl comes with OS X, so it's always installed for us. I think checking whether node is installed, and changing the message to "run brew install node, then curl etc." if it isn't, shouldn't be a problem.

That said, let's wait for Mike.

[MikeMcQuaid]

People keep requesting npm. If we remove it, we'll just have them complaining. Trust me, this is a pain in the arse and I'd rather not support it but the users of both npm and Homebrew seem to want this.

I think all that needs to be fixed (as far as I can tell) for everyone to be happy are:

1. npm selfupdate needs to "work" (I think for it to work it should detect that Homebrew installed it and tell people to use that instead).
2. npm needs to not install into the Cellar or HOMEBREW_PREFIX but probably somewhere else on disk.
3. We tell users to add the necessary PATH/MANPATH/NODE_PATH to their shell.

Because of our versioned Cellar prefix, we can't really have npm upgrading itself. I would imagine this is our main point of contention.

[camillol]

Did you try having a blacklisted message in the past? Maybe they kept requesting it when they saw no npm formula because they thought it was an omission. With a message they should understand. Do you get a lot of complaints about mercurial or tex?

[isaacs]

So, resolved:

1. npm shall install stuff into someplace other than HOMEBREW_PREFIX or Cellar. Is prefix/share/npm/{man,bin,lib} ok with everyone?
2. This is frustrating for everyone, but our users want ponies that poop lollypops, and we have foolishly agreed to deliver.

I'm not super keen on sniffing for Homebrew and then refusing to install npm. There are a lot of ways that a package can end up being installed, and I foresee problems down that road. (For example, a few programs extend or use npm itself, and so they list it as a dependency.)

Also, there are use cases such as using a global npm to install packages into a nave virtualenv, or a globally managed npm that is used by multiple users of the same machine (both real use cases that I get paid to support.) So, "npm doesn't see that npm is installed, but can install itself, and doing so will do what users expect" is a very normal thing. I can't just sniff for "npm didn't install itself" and then throw errors about it.

Because of our versioned Cellar prefix, we can't really have npm upgrading itself. I would imagine this is our main point of contention.

Can you elaborate on that a bit? Is this a practical or philosophical constraint?

People keep requesting npm. If we remove it, we'll just have them complaining.

There are a few out-of-the-box solutions that don't involve us breeding lollypop-pooping ponies:

1. The npm recipe in mxcl/homebrew could instal node, do nothing, and then print out a caveat to run the npm one-liner. (This is actually easier for them than the caveats that we're talking about having them do.) If anyone complains, you could quite justifiably point out that there is no rubygem brew, and no pear brew, and no cpan brew, and that isaacs is a persnickety guy who has trouble playing nicely with others, and they should probably go bother him about it.
2. I could maintain an npm recipe in isaacs/homebrew that does some unauthorized stuff outside of the Cellar (ie, set configs and then run make), and the recipe on mxcl/homebrew could direct them there, along with a warning that this recipe is not authorized or supported by the main homebrew gang, and not philosophically sound.
3. The recipe could be set to have npm install itself, and then provide a pre-uninstallation hook that it could use to clean up its trash before it leaves. The downside here would be that the brew version would not accurately report the version(s) of npm installed, but I doubt most users would care overmuch about that. In that case, provided that the version in homebrew was the same as the version in the npm registry, doing brew install npm or npm install npm would always leave you in the same effective state with respect to the modules and executables on your system, which would be nice.
4. You could leave the recipe as it is, but replace the bug reporting information to post bugs here, rather than on the npm repository or mailing list. It wouldn't fix anything, but it would make me complain less.

[isaacs]

Do you get a lot of complaints about mercurial or tex?

Wow, yeah, mimicking brew install mercurial is a great solution in this case, I think. Isn't this basically the exact paradigm example of where a blacklist message is appropriate?

[MikeMcQuaid]

For 1) I think it's up to the users whether they want this, regardless of your intentions. I know it's irritating when people use your software in a way you don't intend. Once we start allowing e.g. multirepo support then people will just add this anyway as they kept requesting.

For 2) it's not just philosophy, it's general common sense. If npm is in /usr/local/Cellar/npm/0.2.11-5/ and that contains version 0.3 then it's rather confusing for us to manage upgrades and for our tools to list the npm version.

For 3) we aren't providing uninstallation hooks any time in the projected future.

For 4) that's not a terrible idea but I would suspect they will still come to you anyway.

[isaacs]

For 2) it's not just philosophy, it's general common sense. If npm is in /usr/local/Cellar/npm/0.2.11-5/ and that contains version 0.3 then it's rather confusing for us to manage upgrades and for our tools to list the npm version.

That's actually not what I'm suggesting. I'm suggesting that npm v0.2.11-5 is in /usr/local/Celar/npm/0.2.11-5, and npm 1.2.3 would be in /usr/local/share/npm/lib/.npm/npm/1.2.3. The main change would be that the /usr/local/bin/npm symlink would be removed, the npm manpages would not be installed to /usr/local/share/man, and so on. Basically, the recipe would do the following:

1. write /usr/local/etc/npmrc to have npm install packages to /usr/local/share/npm
2. make dev to install itself into that location.
3. print the necessary caveats about path/manpath/node_path.

So, of course, if you do npm i npm, you'll probably end up with a version of npm that is not what your brew tools say you have.

For 1) I think it's up to the users whether they want this

What is your reason for believing that users will object to a blacklist message telling them to install npm with curl?

Of course, if the recipe disappears or is simply missing, someone will send you a pull request to put it back in, thinking it's an omission. But if it's clearly a deliberate choice to not install npm with homebrew, and the message provides clear direction, I suspect that there will be little if any user annoyance.

Once we start allowing e.g. multirepo support then people will just add this anyway as they kept requesting.

I'm not asking you to clean up the whole world. Just mxcl/homebrew. If another fork becomes an equally painful thorn, I'll bring it up with them, but I don't think that's a real problem.

For 4) that's not a terrible idea but I would suspect they will still come to you anyway.

True that. But at least when they paste the log, I'd know right away it's a homebrew install, instead of having to find out halfway through the debugging process.

[MikeMcQuaid]

The npm symlink shouldn't be removed, that doesn't make much sense as it will then behave differently to every other application.

I know this is an npm design thing but I kinda object to have a .npm directory randomly in share. Can this not go in /somewhereelse or ~?

It would probably be good if npm's autoupdate could be disabled when using it in Homebrew. We're fairly good at just keeping it up to date or you could put a warning on startup.

There will still be user annoyance as there is so for the many other packages we blacklist and this will be worse as there's no reason to remove it really other than you not liking it. It seems to be working fine for a bunch of people in Homebrew.

For 4), can't we just append -homebrew to the version or something, if that's really a concern that's hard to debug?

[isaacs]

There will still be user annoyance as there is so for the many other packages we blacklist and this will be worse as there's no reason to remove it really other than you not liking it.

It's not just that I don't like it for some arbitrary esthetic reason or something. It's that it causes problems for users, and I'm sick of supporting it, and it breaks my heart that people are misled by your program into thinking that it's installed properly, when it isn't.

This is not the first time that you've mentioned "just because you don't like it" as an insufficient reason to do something. Great. Me liking it is not relevant. I get it. The repetition of this line leads me to infer that you believe this is some trivial subjective issue on my part, and I'm just being difficult. If that is indeed what you're attempting to insinuate, I find it dismissive, unprofessional, and insulting. If I've misunderstood you, then I apologize.

The issue isn't that I don't like Homebrew. I do in fact like Homebrew. The issue is that npm, when installed with the existing Homebrew recipe, doesn't actually work the way it says it works, I'm getting the credit for breaking it, and I don't have the ability to fix it.

You do. So, do you want it to work properly, or not? There are several options at your disposal here. I think I've made it clear that I will work with you within reason. I will answer any questions you have about how npm works, what it expects, or how people tend to use it. I will suggest and evaluate solutions, and explain problems. I've already accepted that I "won't like" whatever happens, but if it ends up with me not having to deal with confused and frustrated users sent my way by a broken installer, I'll like THAT enough to make up for any other problems I have with it.

That is my only goal here.

It would probably be good if npm's autoupdate could be disabled when using it in Homebrew.

npm does not have an "autoupdate" feature. npm is Just Another npm package. I'm not going to add code changes to prevent the installation of npm (or any packages that depend on npm) just for Homebrew.

If you wanna do that, go for it, it's not as if I can stop you. Start in lib/install.js and lib/update.js. It's just JavaScript.

We're fairly good at just keeping it up to date

I disagree. https://github.com/mxcl/homebrew/blob/master/Library/Formula/npm.rb#L4

I know this is an npm design thing but I kinda object to have a .npm directory randomly in share. Can this not go in /somewhereelse or ~?

It will go wherever the "root" config tells it to go. If you want npm to install modules into $HOME, then that can be arranged easily. Just change the config.

It seems to be working fine for a bunch of people in Homebrew.

It seems that way to you because you aren't dealing with the issues it causes.

For the "load the configs and self-install" approach, here's a recipe that works as I proposed: https://github.com/isaacs/homebrew/blob/master/Library/Formula/npm.rb

can't we just append -homebrew to the version or something, if that's really a concern that's hard to debug?

It's not just that I want to know that it's installed with Homebrew. It's that I want our mutual users to understand that the npm installed by Homebrew does not expose npm's entire api and is not supported by me, and that they shouldn't come to me with issues with it, since you've modified it. I want them to know, right away, when an error occurs, that I can't help them as long as they're using your program to install mine.

The alternative is that we can make the npm recipe work properly. If you're convinced that it already does, then I'm not sure why we're even bothering to discuss it.

[MikeMcQuaid]

If you have a problem with me or my approach, I'm sorry and please email me personally to talk about it. Let's leave the Homebrew bugtracker for discussion about how to solve these problems and stick to the technical discussion please.

It will go wherever the "root" config tells it to go. If you want npm to install modules into $HOME, then that can be arranged easily. Just change the config.
If I install it from your website, where does npm (and the packages it installs) get put?

If we can set this to a location outside HOMEBREW_PREFIX and/or the Cellar and stop npm from touching the files Homebrew has installed (e.g. selfupdate or whatever) then I think we have a workable solution.

[isaacs]

If you install npm manually, without changing anything beforehand, it'll default to these configs, assuming that node's execPath is /usr/local/bin/node:

• root: {execPath}/../../lib/node --> /usr/local/lib/node
• binroot: {execPath}/.. --> /usr/local/bin
• manroot: {execPath}/../share/man --> /usr/local/share/man

You can also set binroot or manroot to "false" or "null" to prevent them from being used at all. For manroot, that's not much of a big deal, but for binroot, that's a bigger issue, since a fair number of npm packages install an executable.

So, how about the recipe I pushed to my fork, but with these configurations, which are likely to work, even if the user doesn't change anything:

• binroot: $HOME/bin
• manroot: false (don't install manpages. oh well.)
• root: $HOME/.node_libraries

Is that acceptable?

[MikeMcQuaid]

This sounds pretty good. What about ~/.npm/bin, ~/.npm/man (or ~/.npm/share/man) and ~/.npm/lib?

Would this mean that npm will only modify/put files under ~/.npm?

A system-wide alternative could perhaps be the same but e.g. /usr/local/npm/{bin,man,lib}? I'd prefer the prior but we could suggest the latter if users needed it to be systemwide.

How does this sound? Would that work for you?

[isaacs]

I don't have any strong feelings about where npm installs things. I do have some weak feelings about it, though:

If npm were to install modules into ~/.node_libraries, then those files would be automatically picked up by require statements without editing any environment variables.

The .npm folder is created inside the npm root folder which is npm's "workspace". (A bit like the Cellar folder in Homebrew.) It might be confusing to see $HOME/.npm/lib/.npm/. What's the value of making it a dot folder? Is there precedent for that style in other Homebrew packages?

/usr/local/npm/{bin,man,lib} seems about the same to me as /usr/local/share/npm/{bin,man,lib}, which is how it works today. So, if there are indeed a bunch of people using npm successfully with Homebrew today, their installed modules would continue to function without any hiccup. Moving those modules to ~/.npm/{lib,bin} will probably break things.

It seems strange to me that one would install Homebrew in a global place, and then use Homebrew to install npm, and not expect npm to be installing things systemwide. If you cared about installing things system-wide, wouldn't you have most likely dropped Homebrew into ~/local instead of /usr/local? How does pip do it?

[MikeMcQuaid]

Ruby Gems and CPAN both install into a .directory in ~. I'm not 100% sure about pip, I think it installs in the Python directory and if it doesn't have permission (this is the bit I'm really unsure about) it installs into the user's ~.

I do see the appeal of .node_libraries but my only thought was that .npm means it's explicitly added to the environment (not that hard for users to do) and maintains a nice separation as a result for easy removal/debugging.

I agree about .npm/lib/.npm not being the most appealing. In that case you could perhaps have the root as ~ and the bin and man directories under that?

/usr/local/npm has the advantage of not being in directories controlled by Homebrew.

Many (perhaps even most) applications installed by Homebrew install themselves globally (e.g. /usr/local) and then put all data locally (i.e. ~)

I think we're getting pretty close to a solution here.

[isaacs]

Rubygems is an odd one there. It installs globally if you're root, but locally in ~/.gem if you aren't. I explored something similar initially with npm, and found it to be a bit of a nightmare. I am not opposed to exploring it again at some point in the future, but it's still young and under such heavy development that the additional complexity is a significant burden.

I have to support installing modules system-wide today, and it's easy enough to configure it into a directory under ~ or just use a build of node that lives in ~/local/bin or something.

I agree about .npm/lib/.npm not being the most appealing. In that case you could perhaps have the root as ~ and the bin and man directories under that?

The problem there is that npm writes to the root directory as well as to its .npm folder. So, after npm install connect you'll have this, for instance:

{root}/
  connect@0.5.0/
    {shim modules that load ../.npm/connect/0.5.0/package/lib/blah.js}
  connect @-> ./connect@0.5.0  {active version}
  .npm/
    connect/
      0.5.0/
        {package contents, dependency links, metadata, etc.}

So, if root is ~, then you'll be littering your home directory with node programs, and that's not really acceptable. Could drop the dot and have ~/npm, though. Then at least when I say "the .npm folder", no one will think I mean ~/.npm.

/usr/local/npm has the advantage of not being in directories controlled by Homebrew.

Good point. I see no problems with using prefix/npm.

I think we're getting pretty close to a solution here.

I agree. It certainly seems like we're painting bikesheds at this point.

Just so I'm clear, you're saying that you're in theory OK with this approach, and we're just figuring out where to put it? Or are there still other considerations to work through?

[MikeMcQuaid]

I think we're basically good now. /usr/local/npm works for me (or perhaps HOMEBREW_PREFIX/npm if that's ok with you) but I'll just check it over with either adamv or mxcl as this is a fairly big change to make.

Thanks for all your work with this.

[isaacs]

Yeah, I'm not fussy about the install paths, as long as they're consistent. The formula in my fork is signed-off by me. If you'd like it squashed into a single commit and sent in a pull req, I'd be happy to do that. LMK what adamv and mxcl think.

Thanks for all your work with this.

Thank you, as well.

[isaacs]

What's the status of this?

[MikeMcQuaid]

I'm waiting for mxcl or adamv to review this.

[mxcl]

Looks like there's good thinking in here. Can someone ping me at 8pm GMT and remind me to review this? At work currently. Sorry to ask for the ping but you know I'll forget otherwise.

[isaacs]

8pm GMT is midnight over here. I can send you a Google Calendar invite or something.

[isaacs]

Ping. Review this.

[mxcl]

Hi sorry about the delay here.

IMO npm should not be packaged. If you don't find this distasteful then we can remove the formula and replace it with something like what we do for brew install mercurial which says:

Install Mercurial with pip:

    brew install pip && pip install mercurial

Or easy_install:

    easy_install mercurial

As was stated, Homebrew doesn't package everything in the name of ensuring people install things in the proper way. This is for everyone's benefit. npm seems very well designed and can take care of itself and to this end should not have a formula, just a helper, to point people to the correct installation instructions.

Thanks, and let me know if this is no good, and sorry this wasn't escalated to me sooner and sorry I didn't deal with it sooner.

[isaacs]

Yeah, I totally grok that. I think I'd have the same reaction to packaging Homebrew in npm ;)

I don't think that the discussion/fistfight was useless, though. It definitely contributed some good ideas to the evolution of npm and the node module system.

I'm working on a huge refactor of npm that will make it a lot easier to work with other package manager systems. Maybe we can revisit this at some other time.

I'll send a separate pull request with the message and removal today.

[MikeMcQuaid]

Thanks isaacs. I didn't agree with removing this earlier but I do now. Sorry this took so long to sort out and thanks for working with us on this. I owe you a pint :)

[isaacs]

I owe you a pint :)

Next time you're in CA or I'm in UK, I'll hold you to that :D

[mxcl]

Cherry-picked: 6ae9864

[burningTyger]

this thread could be added to the non-formula brew as an explanation. I had to search the issues to find the reason for a blacklisted npm. Now I understand :)