[RRFC] Respect standard file locations

[probablykasper]

Motivation ("The Why")

npm currently stores files/directories in the user's home directory, which is non-standard (at least on macOS) and messy. It also interferes with apps that expect standard folder locations to be used (for example Apple Time Machine which expects cache to be stored in ~/Library/Caches).

This is a continuation of #389, which was closed due to concerns with backwards-compatibility and not valuable enough for enough users.

I explain below why I don't see any backwards compatibility issues. These points were mentioned in the original RRFC, and I've asked for thoughts about reopening it, but received no response after 5 months which is why I'm opening a new one.

On the point of it not being valuable enough for enough users, the original RRFC is in fact the 7th most upvoted. There are certainly also lots of people who are annoyed by the current behavior, but don't care enough to look for an RRFC to upvote.

For example, for Rust's cargo it's the 5th most upvoted issue with 143 upvotes (and 1 downvote).

How

Current Behaviour

npm currently stores files/directories in the user's home directory.

Desired Behaviour

To follow standard locations, the following paths would be affected:

• The userconfig path - ~/.npm/
  I can't see why there would be any backwards compatibility issues with changing the cache location.

• The config path - ~/.npmrc
  This should be perfectly backwards compatible by looking for .npmrc in the following order (similar to git's approach):

  • $XDG_CONFIG_HOME
  • OS-standard directory
    • macOS: ~/Library/Application Support/npm/.npmrc
    • Linux: Not sure, maybe ~/.config/npm/.npmrc or the way it is now if that's truly standard
    • Windows: %AppData%\npm\.npmrc (I think)
  • ~/.npmrc

References

#389

[ljharb]

Note that there's nothing "non-standard" about storing files in the home dir, because there isn't actually a standard. XDG is just an attempt to make one, but it is not "the" standard - and "messy" is subjective.

Re the cache location, see #389 (comment) - changing the cache location will likely break a ton of tools.

#389 (comment) gives you a perfectly reasonable workaround - one you can commit to version control if you want per-repo, or set in your userconfig, or both.

[probablykasper]

It definitely is standard, but you might be right about Apple/Microsoft/Linux not having official specifications for it. That's not what I'm referring to, sorry if I was unclear.

I should clarify that I'm a macOS user, and don't know 100% about Linux and Windows. Maybe this should be specific to macOS, where I know that it's standard to use the OS-provided folders?

"messy" is subjective.

It's messy because it goes against the structure already there. You have the Documents folder conjoined with application-specific cache folders and config files, and you end up with several different locations for the same kind of data (For example ~/Library/Preferences/myapp, ~/.myapp and ~/.config/myapp). It's unorganized and messy. The subjectivity is about whether you like that mess or not.

changing the cache location will likely break a ton of tools.

Thanks for sharing that, seems like at least .npmrc might need to be considered a breaking change. Does npm officially support interacting with .npmrc, or something that gives third party tools an expectation of the file location not being changed?

At least with the cache folder, I imagine that isn't the case.

[ljharb]

I don't find it messy; I find it a superior way to organize files. Again, "messy" is subjective.

npm absolutely officially supports interacting with the four possible .npmrc locations - nvm, for example, hardcodes these four paths and would break if this changed.

[WhyNotHugo]

npm currently stores files/directories in the user's home directory, which is non-standard (at least on macOS)

This is also true on Linux. I keep very few top level directories in my $HOME, and having .npm sitting there in the middle is very annoying.

---

It should also be noted that it's not even possible to override this annoying behaviour, since usage of npm_config_cache is broken, as reported in npm/cli#3350, which was apparently closed by accident and never again opened.

[RemiCardona]

The trend is slow but there's definitely growing support for the XDG spec, even for tools that predate the spec and its adoption by desktop environment. Here are some very subjective examples based on what I use regularly:

Pip (and a growing number of python tools, e.g. pylint) are supporting it. Pip supports the historic ~/.pip/pip.conf and the XDG ~/.config/pip/pip.conf. I have yet to see anyone formally deprecate the former. The pip cache is now in ~/.cache/pip/ by default, with no deprecation or automatic migration of the original location either.

Git also supports XDG, again with no plan for deprecating ~/.gitconfig that I'm aware of.

More surprisingly (at least to me): Firefox has started merging in XDG patches. It's not fully compliant and still uses its own ~/.mozilla/ but the work is on-going. Firefox does seem to handle the migration of files on its own. I was pleasantly surprised to find that ~/.cache/mozilla/firefox/$profile/ does indeed contain http cache data. (18yo meta tracker at https://bugzilla.mozilla.org/show_bug.cgi?id=259356)

My 2 cents: it'd be neat if npm (and nvm too, which recently solved me a lot of headaches) could support XDG. It's not essential, the lack of said support is not life-threatening. It'd be nice to have.

Cheers

[nlf]

i could absolutely see adding this support as a new feature. my biggest question would be what do we do if a config file is found at both ~/.npmrc and /.config/npm/npmrc? following suit with other places where we address this type of ambiguity we would choose and favor one of the files over the other, and log a warning that the other is being ignored. this could be perfectly appropriate here, but we need to be explicit and very clear about what config we're actually loading.

we would also need to be careful to specify that if you have a config file at ~/.config/npm/npmrc that it will only be usable in npm versions >= whatever version we add that support in. this is a potential for headaches, but i don't think an entirely unreasonable one to accept.

re: discoverability, npm@6 and above do support the npm config get command which allows you to return data about the configuration - including where it's located. i.e. npm config get userconfig will return the path to the user's npmrc file. is that something nvm could leverage to make this transition a bit smoother @ljharb (assuming we went this direction, that is)?

[ljharb]

@nlf not that i know of - nvm basically only manages node, it hardly interacts with npm at all.

I just don't think there's any inherent value in following XDG, and certainly nowhere near enough to outweigh the hazards of using the wrong config file (as you mentioned), or of the transition period. If we could use a time machine and start with XDG in the beginning, that'd be fine also, but the ship has simply sailed.

Note this has been thoroughly discussed in the past, eg in #389, and rejected for many reasons.

[nlf]

I just don't think there's any inherent value in following XDG

i think "value" is very subjective here. there are hazards to be sure, but as mentioned here in this thread there is precedent for making these moves set by other tools.

If we could use a time machine and start with XDG in the beginning, that'd be fine also, but the ship has simply sailed.

there are certainly some things where the decisions of npm-past are stuck and would be far too breaking to change. i don't personally feel that the location we store config files or ephemeral data like debug logs and the http cache is one of those things. i could very well be wrong, though!

Note this has been thoroughly discussed in the past, eg in #389, and rejected for many reasons.

i'm by no means saying that we will do this, but just because an idea was rejected in the past does not mean we should exclude it from discussion in the future. that issue was closed nearly 18 months ago, that's a very very long time in the world of software engineering.

[ljharb]

I agree "value" is subjective.

I think the location where we store config files, specifically (the rest i don't care about) is something that can't be changed. For example, a package could be accidentally published publicly when it was intended for a private registry - even one occurrence of that would imo be much too costly to be worth it, and i'm sure there's lots of other hazards i haven't come up with off the top of my head.

but just because an idea was rejected in the past does not mean we should exclude it from discussion in the future

while that's true, it would only be worth discussing further if there's new information to consider. Is there any?

[nlf]

I think the location where we store config files, specifically (the rest i don't care about) is something that can't be changed. For example, a package could be accidentally published publicly when it was intended for a private registry - even one occurrence of that would imo be much too costly to be worth it, and i'm sure there's lots of other hazards i haven't come up with off the top of my head.

i agree this risk is absolutely the biggest one we have. our users would have to be clear that the new config location only works in newer versions of npm, and communicating that can be a challenge for sure.

it's an interesting problem because there are more appropriate ways to solve for this concern, though. to speak to your example of someone accidentally publishing to the wrong registry, i would contend that if you know your package only belongs in one registry that should be defined in the publishConfig of your package.json or in a project level .npmrc, neither of which would be affected by this change at all. the real hazard here will lie with folks who work with multiple registries and define their additional registries in their user config file instead of in a project config or publishConfig. our lack of insight into how often this is the case does make any changes here a bit blind, though.

to alleviate that risk, it could be that we re-scope this issue to only speak to the location of the http cache, log files, and the npx storage which are far less risky to move.

while that's true, it would only be worth discussing further if there's new information to consider. Is there any?

there's relatively recent precedent being set by pip and firefox mentioned above, i'd classify that as new information.

[ljharb]

When was xdg support added to pip, git, and firefox?

[nlf]

When was xdg support added to pip, git, and firefox?

looks like pip and git started supporting it way back in 2014, so that was a misspeak on my part. the info was new to me, not actually new.

firefox's issue has been open for 19 years now, with the most recent activity ~3 months ago so that one's relatively new.

[nlf]

worth noting that the arch linux wiki has a big list of apps that support XDG, including data about when the support was added: https://wiki.archlinux.org/title/XDG_Base_Directory#Supported

[ljharb]

Interestingly that table already lists three lines of config that make npm support XDG.

[probablykasper]

I found Apple's documentation/guidelines that talk about about where data should be stored:

macOS Library Directory Details: https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/FileSystemProgrammingGuide/MacOSXDirectories/MacOSXDirectories.html

macOS File System Basics, section "The Library Directory Stores App-Specific Files": https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/FileSystemProgrammingGuide/FileSystemOverview/FileSystemOverview.html#//apple_ref/doc/uid/TP40010672-CH2-SW1

[probablykasper]

I think the location where we store config files, specifically (the rest i don't care about) is something that can't be changed. For example, a package could be accidentally published publicly when it was intended for a private registry - even one occurrence of that would imo be much too costly to be worth it, and i'm sure there's lots of other hazards i haven't come up with off the top of my head.

@ljharb How exactly could this problem occur? Would that be if the user decides to create a new config, overriding their old one?

[ljharb]

@probablykasper if the user uses an older version of npm that doesn’t know about the new location, it would silently run with the wrong config.

[RemiCardona]

For pip, I migrated our team's CI scripts to new locations when I was certain that old versions would not be an issue (eg, ditching python 3.4 support and its builtin pip 1.5 which doesn't handle xdg). I only recently migrated our documentation to the new xdg paths for git config as I had to rewrite large chunks it.

If npm decides to support xdg, it is certain that support for ~/.npmrc will be around for a long time. Just like git will probably support ~/.gitconfig indefinitely.

But at least, there's the option of using standard paths with predictable locations.

[ljharb]

There's already the option, using the 3 lines of config mentioned here: https://wiki.archlinux.org/title/XDG_Base_Directory#Supported

The ask here, I assume, would be to make these lines implicit when an XDG env var is present. Nobody's prevented from using XDG now explicitly.

[nlf]

Nobody's prevented from using XDG now explicitly.

I think this is where our focus should lie here. We have means of overriding these file locations available, we can solve with documentation how to use those configs to align with XDG specs.

It should also be noted that it's not even possible to override this annoying behaviour, since usage of npm_config_cache is broken, as reported in npm/cli#3350, which was apparently closed by accident and never again opened.

This is a blocker to solving this with documentation. We'll need to fix this bug and then we can document the workaround.

[sisrfeng]

Interestingly that table already lists ... lines of config that make npm support XDG.

I hope that someone with archlinux account can add export PATH=$PATH:$XDG_DATA_HOME/npm/bin to the notes