[RRFC] Respect platform specific directories for config, data, cache, etc.

[natestemen]

This is a follow on since npm/cli#654 was closed.

Motivation ("The Why")

When using npm, it should not leave files/directories in the users' home directory if known environment variables are set, e.g. XDG_CONFIG_HOME.

How

Current Behaviour

npm creates a ~/.npm directory.

Desired Behaviour

npm should create the needed config/cache files/directories with respect to https://wiki.archlinux.org/title/XDG_Base_Directory.

[lousyd]

To clarify, the XDG Base Directory Specification does not require that environment variables be set. For example, it says, "$XDG_CONFIG_HOME defines the base directory relative to which user-specific configuration files should be stored. If $XDG_CONFIG_HOME is either not set or empty, a default equal to $HOME/.config should be used."

While npm could certainly choose to use existence of the environment variable as an indication of whether any particular person wants npm to observe XDG or not, it would not be a reliable signal. There are going to be lots of cases of people who don't set the variables and yet still expect software to "play well with others" by putting files in the XDG-expected locations. The converse situation -- people who really don't want you to observe XDG -- is surely.... rare.

Just follow XDG. Don't require the environment variables be set.

[ljharb]

In my experience maintaining nvm, users who do not set the environment variables but have any concept or expectation of XDG in their minds don't exist. I think only caring about XDG when the env vars are set is a perfectly reasonable approach.

[samhh]

@ljharb I disagree with you.

I care about XDG but only explicitly set them when I either want to use irregular locations or when I need access to them for some other purpose (e.g. ironically handling programs that support specifying the path but are ignorant of XDG). I assume the defaults will be respected per the specification.

A better way of framing it may be:

• Users who care about XDG might or might not have set the variables. They'll expect XDG-compliance by default.
• Users who don't care about XDG won't have set the variables and probably won't care where their data lives.

[ljharb]

The vast majority of users don't care about XDG at all, though. I explicitly and actively want the way things are, and I personally don't like the XDG spec at all. Currently I do that by simply not setting the XDG variables, and thus "almost every tool in existence" does nothing, since virtually every tool also does not respect XDG even when set.

[samhh]

You're absolutely right that the vast majority of users don't care about XDG. Equally I don't think they'd care if it became the default. The most impassioned voices on this issue will I think naturally be those like myself who want support. Do you not think you're going to be in the minority here @ljharb? I may be mistaken.

A lot - not all - of major Linux applications support the spec. Notably the usual reason for lack of support is backwards-compatibility concerns. I think it's a much different story on macOS.

To be clear I'd ideally like support without the need to explicitly opt-in; see here how noisy my shell config has become trying to opt everything in.

[WhyNotHugo]

Prior discussion on the topic has happened here: npm/cli#1451

A few others have voiced support for the issue, and there's also a PR, although in an abandoned state, for anyone willing to pick that up.

FWIW, any user who "does not care" about XDG won't be negatively affected by this. I very much doubt there are many users (if any) that actively want random applications to dump cache data right into their home directories.

[ljharb]

A large part of the js ecosystem describes the locations of npm’s cache folder, as well as a user-level $HOME/.npmrc, and changing these defaults would never be worth the churn.

[samhh]

How many significant tools are there that make these assumptions? In theory if it's just a few then it wouldn't be too hard to update them to support checking two locations, first XDG and then legacy.

[ljharb]

@samhh ive never met a tool that publicized XDG support. I’ve only ever heard of XDG (before these npm issues) because like 3 people in 10 years have requested that nvm support it, which it does only when the env vars are set.

[samhh]

fish references it in its documentation for example.

Which tools are significant and directly access npm files?

[natestemen]

In my view supporting XDG isn't about being able to publicize/advertise the fact that you support XDG, but rather because it makes using a terminal ever so slightly better, and more understandable.

@ljharb I don't understand your considerable pushback here. Would you mind elucidating what it actually is about supporting XDG that you don't like? As of now it seems like you don't want to support it because:

1. you don't believe it's useful to those who don't have the env vars set (which I think we'll agree will be the majority of users)
2. it's not widely supported (as of now, but my understanding is that this is moving towards XDG adoption)
3. you believe very few people care about this

I think the last point is probably true, but that doesn't indicate whether it will be helpful for them or not. I think most people have never heard of XDG to begin with.

[WhyNotHugo]

I'm looking at my dotfiles and see 21 apps that support the basedir spec (including git, neovim and systemd).

Chromium also has support, albeit imperfect and it mixes runtime files with config files (this is more of a bug than refusal to support it).

The only exceptions I see are npm, go, yarn, Firefox (just historical reasons and slow moving) and tools that cannot support it, like ssh and zsh. This last group cannot support the spec, since they're the ones who define environment variables and that'd be a chicken and egg.

That's still 21 supporting apps vs 4 that don't -- none of which refuse to support the standard, just haven't gotten around to it yet.

Honestly, I see no reason to NOT support this standard and keep saving cache files into home directories. I'm still curious why someone would want cache files in their $HOME rather than in the directory specially designated for that. Do you want all cache files in $HOME, or just npm's?

[ljharb]

@natestemen i have no objection to supporting it when the env vars are defined. My pushback is against the following points:

1. Not supporting xdg is never a bug, it’s merely a choice, because xdg isn’t a universal or mandated thing
2. When the env vars are not defined, the current/default/original behavior of a tool should be retained.

I personally prefer, and find it much cleaner, to have my $HOME filled with things from every too., and actively don’t want them hidden away in an obscure directory. Those who disagree can certainly set their XDG vars and ask for tools to support that.

[samhh]

@ljharb Isn't it more reasonable - on the basis of active support for either approach - to suggest that those who want everything in their home directory to just set their XDG vars accordingly?

[WhyNotHugo]

I'm happy to debate this further, but I see no actual arguments in favour of not supporting the standard so far.

• Behaving as other apps, and following a general direction makes usability easier for end users.
• It's actually possible to easily backup all config files since they're all in one place.
• It's simple to free up disk space since cache data is all in one place (I can't image how you'd do it without following this spec to keep things separate).
• Littering users' home adds noise. For example, I've 8 directories in my $HOME. Having random apps create their own directories, my own stuff would be drowned in a sea of noise.

I understand that some people may want cache data in their $HOME, but there may also be people who want temporary files in /usr. Just because there's a random desire for it doesn't seem like a valid argument for that.

[ljharb]

@samhh when creating a new tool, sure. this isn't a new tool, and "the way it's always been done in this tool" is the default.

[samhh]

@ljharb That's the default until there's a compelling reason to change. The reason here is that there's a specification a lot of major software follows that a not insignificant minority of users want, and which doesn't preclude any other behaviour including stuffing everything in ~ as we do now.

[ljharb]

@samhh "a lot" and "not insignificant minority" both remain completely unestablished.

I completely agree that it's totally fine, when XDG env vars are set, for npm to start respecting them - that's what I made nvm do, when the quite insignificant minority of its users who even knew about XDG requested the feature. The only part I'm really pushing back on is changing the default behavior in the absence of those env vars.

[WhyNotHugo]

Those env vars are seldom defined. They're mostly there for users who want some unusual behaviour, or for unique environments (like Flatpak, for example).

The spec is all about sane default, implementing the spec except the defaults is pointless.

I've listed four good reasons for the improvement. I still have to hear one against it, or any criticism of those reasons.

[ljharb]

1. Most apps in the js ecosystem do not in any way follow XDG, and that - not the shell ecosystem - is the more important idiom to look to
2. it’s already possible to easily back up config files when you need them, but for npm, you generally shouldn’t have user-level config anyways except your credentials - which you shouldn’t be backing up
3. npm has its own command to clear its cache, you never should nor have to muck with paths
4. “noise” is subjective. For me, it makes them readily discoverable, and i want them there, by default.

[toastal]

Most apps in the js ecosystem do not in any way follow XDG

Data? My own counter anecdote, I've seen a growing number of projects unaware of the spec happily merge changes to the config location if asked (more so if someone opens a merge request), including Yarn and Berry or how about Netlify's CLI, etc.. I would then actually assume a vast majority of Node applications are deployed and running on Linux (because most servers run Linux). And while not scientific, the StackOverflow surveys show Linux support is quite high among developers. With Linux, you would need to start counting the number of distros that don't follow the spec because of its ubiquity. Their users are used to it and expect it...

and i want them there, by default

If it's just subjective, then personally don't use or set any XDG environment variables and don't punish environments that want applications to follow their specifications, especially when the fallback is what you want. This is the whole point people are trying to make.

[ljharb]

@toastal as i've stated many times, those who want different behavior and set the XDG vars should be supported. I'm only expressing a strong opinion that the defaults do not change.

[Flare576]

Ad-hoc data:

• From Github Search: "$HOME/.npmrc"
  • 33 Wikis reference $HOME/.npmrc
    • Most are small/personal projects, but of note is Microsoft/typescript
  • 9,344 code results
• From Github Search: "$XDG_CONFIG_HOME"
  • 491 Wikis
  • 291K Code results
• From Homebrew (sorry @ljharb, I know it's unsupported but it's the data I know how to get)

install: 49,534 (30 days), 160,736 (90 days), 533,418 (365 days)

If NPM were to adopt a new set of default directories (which seems to be the primary point of contention), it would absolutely be a breaking change in my opinion. Even nvm users, despite managing their npm installs and cache via a separate folder structure, would be impacted - adopting XDG standards would be a "all or nothing" venture, so the $XDG_CONFIG_HOME/.n[pv]mrc change would be felt.

[obedm503]

All it has to do is use the XDG environment variables if defined.

I'm with @Flare576. As much as I would like XDG defaults, it would be a breaking change. I would guess most users are not even aware XDG exists much less care for it.

It could be done if npm included some sort of migration code that moved files around when upgrading the version (like a database migration). Unlike a dabase migration, npm devs do not control where npm runs. It may be impossible to know if the migration went well (other than the ton of question on SO if it fails).

[samhh]

It depends how you define a breaking change. It potentially wouldn't be breaking for npm itself if we continued to check for and use the legacy location if it exists, and defaulted to XDG (present or not) otherwise.

[WhyNotHugo]

The XDG spec refers to several directories, and I think the breakage cause by moving each one varies.

I was pointed to this issue because I want npm's cache to be in the proper directory. It bugs me to no end that each time I use npm, it'll dump it's cache into my home, along with all my other important stuff.

I don't think using the XDG location for cached files would be an "unexpected breakage" for anyone. Quite the contrary, I image most people prefer cache directories out of the way, and not as first-level directories inside their $HOME, pretty much the same way you don't want temporary files in $HOME, but rather in $TMPDIR.

Config files are a bit different, since people (and tools) are expected to want to interact with them more and care about its location. git handles this in an very good backwards-compatible way: If $XDG_CONFIG_HOME/git/config exists, it will use that, but if it doesn't, it falls back to the non-XDG location: $HOME/.gitconfig. (note on this: for newer files, they only support XDG, like $XDG_CONFIG_HOME/git/ignore).

[ljharb]

Assertions about what most people prefer are impossible to back up in either direction. I personally actively expect everything in my home dir, and I find XDG to be a really weird way to solve what isn't a real problem.

The way git handles things (in the git config case) is exactly what I think npm should do here.

[toastal]

The way git handles things (in the git config case) is exactly what I think npm should do here.

Just to be clear, git-config looks for configuration files in this order:

1. $(prefix)/etc/gitconfig
2. $XDG_CONFIG_HOME/git/config
3. ~/.gitconfig
4. $GIT_DIR/config
5. $GIT_DIR/config.worktree

[probablykasper]

The good news is, you can make npm behave in an XDG-correct way relatively easily, by putting these lines in your ~/.bashrc or equivalent:

export npm_config_userconfig=$XDG_CONFIG_HOME/npm/config
export npm_config_cache=$XDG_CACHE_HOME/npm
# if you do this one, make sure to add $XDG_DATA_HOME/npm/bin to the $PATH,
# otherwise global installed executables won't be accessible on the cli.
export npm_config_prefix=$XDG_DATA_HOME/npm
export PATH=$PATH:$XDG_DATA_HOME/npm/bin

Was hoping to do this to remove some of the 18 items cluttering my home folder. Unfortunately it looks like setting npm_config_cache doesn't fully work due to npm/cli#3350

Can at least the cache be created out of the way, even if the rest of the spec is not implemented? I can't imagine there's any compelling reason to create a cache directory in $HOME.

I strongly agree with this, there should be no big migration issues from changing the cache location.

As for the config file, it would be backwards compatible if npm by default looks for .npmrc in the following order (similar approach to git, which @toastal mentioned). When creating the config file, the first one should be used:

• ~/Library/Application Support/npm/.npmrc
• ~/.npmrc

@isaacs Do you have any thoughts on this? Is it worthy of re-opening?

[thernstig]

@isaacs please re-visit the cache directory proposal to use XDG_CACHE_HOME (or falling back to the default ~/.cache/npm/ if XDG_CACHE_HOME is not set).

This is only breaking the cache and should not be considered backwards incompatible in any meaningful way.

There are literally thousands of Linux packages out there as an example, and if they all put their cache files in ~/.something we can all imagine how that would look.

I do not even consider this a matter of personal preference for what the npm project prefer, but it is more akin to good manners.

[isaacs]

@thernstig while I might be the one to blame, I'm no longer the one to convince about npm decisions, since I haven't worked on the project for a few years now.

[thernstig]

I wonder if we can open a new RFC regarding this? Or possibly ping some maintainer. Even bigger projects like git have started to follow the XDG spec in recent years.

[wesleytodd]

I believe the folks currently on the npm team are following most of these discussions, even if they are closed. There are no RFC calls anymore so I don't think opening a new issue will result in much action.

[thernstig]

I see, I would not have thought closed issues are given much thought, but if we are sure of this then I understand.

[danielpza]

Should this issue be reopened, then?

[ljharb]

@danielpza that would imply that the cli team is reconsidering it. I'm not sure why a few projects "started to follow the XDG spec in recent years" would change the previous reasoning.

[thernstig]

@ljharb there was no "previous reasoning" that was agreed on afaics. #389 (comment) is a good comment in that regard.

[ljharb]

@thernstig #389 (comment) is that, and matches previous discussions on RFC calls whenever it came up.

[thernstig]

@ljharb it has one upvote. git with arguably a factorial larger user base managed to do this change. I do not claim to know this is impossible, but even that referenced comments does not say it cannot be done.

[ljharb]

Whether it can be done isn’t being argued; of course it can. The question is whether it’s worth the churn and additional complexity in npm and a ton of userland tooling, which it’s been previously agreed by the CLI team multiple times it is not.

[codyrobbins]

I feel like I am in bizarro land reading the pushback against this change.

[Ra2-IFV]

NPM is shit, I have to say

[Ra2-IFV]

Polluting user's home directory is just very very bad, it's not for compatibility or what, it's just not being responsible for cross-platform support, it just doesn't respect standards for different platforms.
If you ask for reasons, then most tools that are still being maintained, at least in my cases, respects XDG standard, and more and more tools are going to support it.
If you ask for possibility, then there are commits that already implemented this, all you need to do is review it and merge it.
Now you're saying that it makes npm more complicated, I don't know why at all.

[isaacs]

@Ra2-IFV That is an extremely unhelpful and inappropriate response on an open source project.

npm team, I highly recommend locking this issue. No good can come of it. A decision has been made, any further discussion is only an invitation for abusive behavior.

[codyrobbins]

A decision has been made, any further discussion is only an invitation for abusive behavior.

This is also an extremely unhelpful and inappropriate response on an open source project.

[Ra2-IFV]

Yeah, sure, just go ahead, ignore all users requests "you" don't like, it's "your" project after all

[ljharb]

@Ra2-IFV yes, that is literally what the "as is" mean in an open source license, and literally how open source has always worked. Users can offer feedback but are entitled to precisely zero consideration, and any they get is a gift from maintainers.

[Ra2-IFV]

As people argued I don't think messing up user's directory is more reasonable than following any standards.
NPM is a big project, thousands people use it, developers still should be responsible for the project for ethical reasons or whatever. It's basic manners.

[Ra2-IFV]

Proper criticisms and suggestions from users make a project better, but here I can only see the NPM team is having such a negative attitude towards users.If creating a project that everybody hates it and nobody would use if they have choices, is all you want, I give up.
Now I won't go to criticize, or say offensive things any more, because it has lost its point, and I'm just going to let other people choose other package managers instead of npm.

[ljharb]

"messing up" is subjective; the XDG standard has so few "musts" that even a minimally compliant implementation won't make XDG proponents happy; most of the folks discussing here aren't the npm team, myself included; and yes, you've always been able to go choose other package managers, have fun.

[gitbugr]

@Ra2-IFV yes, that is literally what the "as is" mean in an open source license, and literally how open source has always worked. Users can offer feedback but are entitled to precisely zero consideration, and any they get is a gift from maintainers.

Just read through this as I've been searching for a solution to AWS lambda's not allowing you to write to $HOME (or any directory other than /tmp), and I'm shocked at the callousness here. I get that you're annoyed at this user for being less than polite, but to respond in such a way as to frame it as though npm is a thankless volunteer project that has no moral obligation to the people who use it is disingenuous at best. It's an industry-wide used piece of software under the Microsoft umbrella. It has countless contributors. It's nowhere near on-par with many of the other open source projects we depend on.

People have real issues they're experiencing and are reporting it, and giving feedback in the form of how they think is best for it to be solved. That's how open source lives and breathes. Users contribute their time too (some more than others.). And you're ultimately right, legally, there is no obligation. Don't think anyone would contend that, nor want it any other way. People don't appreciate the efforts of maintainers enough, and it's not right for every Tom, Dick and Harry to expect their feature be addressed above all else. But to be an ass and essentially say "go fuck yourself"? I understand it's directed at that user, but there are many here who have been polite and courteous. Have a little respect. Do you think people will be on your "side" with this? Is this the battle to have and drive people away from the project?

[Ra2-IFV]

NPM is shit, I have to say

I'm saying what people really think about this project in a very informal way
I'm irritated by the arrogance of the dev team towards users. A good open source project does not work in this way.

whatever you say

[toastal]

Don’t muddy discussions with Microsoft GitHub’s gamed social media like/star numbers. Star hacking is a problem, & this metric shows nothing of a project’s quality or leadership—nor should “a good open source project” feel inclined to care about chasing popularity contests on proprietary platforms.

[Ra2-IFV]

of cource it doen't mean anything, at least it shows the popularity in some ways.

nor should “a good open source project” feel inclined to care about chasing popularity contests on proprietary platforms.

I mean a good open source project listens for reasonable user feedbacks instead of saying "no we won't do this" to everything