Make Cargo obey platform-specific directory standards

[tbu-]

Strategy for backward-compatiblity:

When checking for the relevant Cargo directories, check in the following
order of preference:

1. Use the environment variable CARGO_HOME.
2. Use the platform-specific directories if they exist.
3. Use the legacy location (~/.cargo) if it exists.
4. Fall back to the platform-specific directories if everything else fails.

The new platform-specific directories are as follows:

On Windows, use AppData\Local\Temp\Cargo for cache files (obtained via
GetTempPath), AppData\Roaming\Cargo for the config files
(FOLDERID_RoamingAppData) and AppData\Local\Programs\Cargo for
installed binaries (FOLDERID_UserProgramFiles).

On Unixy systems, use the XDG spec: ~/.cache/cargo for cache files,
~/.config/cargo for config, ~/.local/bin for installed binaries.

http://standards.freedesktop.org/basedir-spec/basedir-spec-0.8.html
http://www.freedesktop.org/software/systemd/man/file-hierarchy.html

On OS X, use ~/Library/Caches/Cargo for cache files, ~/Library/Cargo
for config files and ~/Library/Cargo/bin for binary files.

Fixes #1734. Fixes #1976.

[rust-highfive]

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @huonw (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. The way Github handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

[tbu-]

Probably doesn't compile on Windows, because the xdg crate probably doesn't: whitequark/rust-xdg#5.

[tbu-]

Passes all tests now, ready to get merged!

[steveklabnik]

Since XDG is higher priority than the original one, isn't that going to cause issues? People who work on systems which follow XDG are gonna see their existing ~/.cargo dissapear... no?

[tbu-]

The XDG only has higher priority if it exists (~/.cache/cargo). This could help with deprecating ~/.cargo completely in the future.

EDIT: Systems which follow XDG might have ~/.cache, but they won't have ~/.cache/cargo.

[alexcrichton]

Thanks for the PR @tbu-! It looks like there was also quite a bit of legwork in getting everything to work across all platforms, so thanks for that as well! Some thoughts of mine:

• If we're going to knock this out, then I think it may be good to take care of Windows: Cargo should put files in ~/appdata, not directly in ~/ #1976 along the way as well.
• Is it standard to use XDG on systems like OSX and FreeBSD? I'm mostly worry about OSX, but this seems like a very Linux-specific concept. Although it works on OSX, I'm not sure if it's idiomatic.
• I'm worried about the fragmentation this causes across platforms. It's no longer easy given an arbitrary system to find out where Cargo has placed its cache directory or where you should place configuration.
• It looks like we'd be one of the first package managers to do something like this. None of bundler, npm, pypi, or rubygems seem to do anything related to this. Our target audience isn't always experienced linux folks who expect this behavior but often newbies who may not know where to probe to find Cargo-related files.
• It'd be much nicer to use if cfg!(windows) instead of #[cfg(windows)] and it looks like the xdg crate only has one unix-specific import.
• The xdg crate is a bit too eager to panic.

I wonder if it would be possible to have this still be opt-in, but perhaps have the opt-in to be very easy to set?

[tbu-]

Thanks for looking over the pull request!

If we're going to knock this out, then I think it may be good to take care of #1976 along the way as well.

I suppose, but I don't have a Windows machine right now.

Is it standard to use XDG on systems like OSX and FreeBSD? I'm mostly worry about OSX, but this seems like a very Linux-specific concept. Although it works on OSX, I'm not sure if it's idiomatic.

Yea, the points about OS X and *BSD are fair, I haven't researched the situation there yet.

I'm worried about the fragmentation this causes across platforms. It's no longer easy given an arbitrary system to find out where Cargo has placed its cache directory or where you should place configuration.

Fragmentation isn't really worse, you can find the configuration in ~/.config/cargo and the cache in ~/.cache/cargo, unless you set environment values, like before.

It looks like we'd be one of the first package managers to do something like this. None of bundler, npm, pypi, or rubygems seem to do anything related to this. Our target audience isn't always experienced linux folks who expect this behavior but often newbies who may not know where to probe to find Cargo-related files.

Update the tutorials to point to ~/.config/cargo instead of ~/.cargo/config (With a mention that the legacy location ~/.cargo/config is still supported).

I wonder if it would be possible to have this still be opt-in, but perhaps have the opt-in to be very easy to set?

I think making standard compliance an opt-in is basically not having standard compliance at all. The standards exist to give common behavior across many different programs, and if you have to configure each of them anyway, then the benefit of having that standard is lost.

The issues with the XDG crate (cfg!(windows) vs #[cfg(windows)] and the panic) can certainly be fixed.

[steveklabnik]

Yea, the points about OS X and *BSD are fair, I haven't researched the situation there yet.

I can't tell you about BSD, but for OSX: https://developer.apple.com/library/mac/documentation/FileManagement/Conceptual/FileSystemProgrammingGuide/FileSystemOverview/FileSystemOverview.html

TL;DR: very much not XDG.

[huonw]

r? @alexcrichton

[alexcrichton]

@tbu-

Ah yeah sorry to be clear I'm not worried about fragmentation within a platform, but rather across platforms. For example we can't really just update the tutorials to list different paths because now it's a different path for every platform. In that sense I think the fragmentation gets worse because as someone who works across OSX/Windows/Linux I've now got to always be conscious of where I'm at if I want to find these directories.

[bors]

☔ The latest upstream changes (presumably #2154) made this pull request unmergeable. Please resolve the merge conflicts.

[tbu-]

So, I researched a little and found out that the BSDs should probably get XDG handling, OS X caches go into ~/Library/Caches/Cargo and config into ~/Library/Cargo/. On Windows it'll be AppData\Roaming for the config and AppData\Local for the caches. What I'm still trying to figure out is where we should put the executables on OS X and Windows. Do you have ideas for that?

[alexcrichton]

I'm personally still worried about the fragmentation problem. It looks like this is basically going to end up with almost a different location for directories per OS rather than "windows" and "non windows", and at some point it seems like the benefit perhaps isn't that large?

This is why I was thinking that an opt-in may suffice for now in some form.

[tbu-]

@alexcrichton I guess if we want non-fragmentation, we should probably go for XDG on unixy systems (including Mac), and for AppData on Windows.

This would give us a pretty consistent story that interacts well with the OSs, as well as separating caches from configs.

[alexcrichton]

That seems like not a great situation to be in though, unfortunately. There's still fragmentation between Windows and non-Windows, and Cargo would be favoring Linux/Windows over all other OSes by saying they need to be slotted into one of those buckets. Overall it seems like it doesn't really reduce the net number of issues?

[tbu-]

Let's collect a list of advantages and disadvantages:

1. ~/.cargo/ everywhere:

+ Cross-platform consistency
- Very unexpected for Windows users

2. XDG on Unix, AppData on Windows

+ Cross-platform consistency among unixes
+ Clearly separates cache from config, helps backup tools
+ Uses the XDG standard on Unixes
+ Uses an expected directory on Windows
- Not the current solution -> needs code to maintain backward compatiblity for some time
- Not the preferred solution on Mac (but only on Mac, on BSDs, this is fine)

3. XDG on Unixes, Apple-specific on Mac, AppData on Windows

Essentially same as 2)
- Less cross-platform consistency among unixes, might be weird when ssh'ing from machine to machine
+ Uses the preferred solution on Mac

So by just looking at the advantages/disadvantage (tell me if I missed any!), I'd say that 2) and 3) are both favorable to 1). In fact, the only advantage of 1) over 2) seems to be that there is cross-platform consistency between Windows and unixy operating systems, which I'd argue is very much not expected for Windows users.

[bluss]

Thanks for working on this! Following XDG is a huge win and it would be much nicer if all applications did this. Primarily the clean separation of cache vs user data (config and data). Especially since cargo does not clean up its old caches, having them separated out for easy & confident removal is a win for the user. (The user will be safer to know it's only cache).

[alexcrichton]

I think that if we take this route we'd want to do the "best thing for the platform" in each case, so I wouldn't take the route of XDG everywhere on Unix (ignoring what OSX conventions are). To me any fragmentation is just as bad as lots of fragmentation, so I'm not worried about the degree of fragmentation along those lines.

I also think that it may not quite be as black-and-white as listing out pros/cons and seeing which bucket is bigger. The existence of fragmentation can have a multiplicative effect where all of Cargo's decisions affect downstream users and such. For example anyone wanting to write a tool against Cargo's directory layout or something like that would end up having to work with all this logic somehow as well to work across platforms.

I'd be curious on some other opinions from @rust-lang/tools as well, I have a feeling that @wycats in particular may have thoughts on this.

[whitequark]

For example anyone wanting to write a tool against Cargo's directory layout or something like that would end up having to work with all this logic somehow as well to work across platforms.

Unless Cargo's directory layout is considered a stable interface, wouldn't any such tool have to invoke Cargo or use it as a library anyway?

[tbu-]

@alexcrichton I didn't want to suggest to compare the number of pros and cons, as they can certainly be split up in different ways. I do however, still think that the benefits of 2) and 3) outweigh the solution for 1), especially 2) which is basically a pure improvement IMO (I think that 1) is consistency at the cost of Windows users, because there is virtually no program that stores its configuration in something like ~/.appname on Windows).

[Alexander--]

While we can't create symbolic links on Windows, we can create directory junctions, which as far as I can tell are just as good.

Yes, the exact type of link to use in this case seems irrelevant.

[bors]

☔ The latest upstream changes (presumably #2339) made this pull request unmergeable. Please resolve the merge conflicts.

[tbu-]

@Alexander-- Not part of the XDG spec, but the second link in the PR: http://www.freedesktop.org/software/systemd/man/file-hierarchy.html.

[alexcrichton]

This is an example of possible breakage from following these sorts of environment variables. It could be the case that someone invokes cargo with a new HOME variable expecting output to go in that location, but other environment variables (not specific to cargo) can end up thwarting that.

[tbu-]

Mh, fair enough, that's definitely a breakage from starting to follow XDG specifications where you previously did not. "The sooner the better?" Your link might even provide some more motivation, even git is doing it (which definitely isn't a desktop application).

[brson]

To me, being able to backup your cargo folder without copying cache files is the best reason to want this, and that could be achieved by changing the cache location to be platform-specific, possibly optionally. There's less reason to care where cargo stores its internal cache than its configuration.

It would be as insecure as adding . (e.g. the current directory) to PATH—too easy to screw up. Some well-meant app would cleverly put something there without my knowledge, and the next command I type leads to unpredictable results.

@Alexander-- I like this point, that encouraging $PATH to include somewhere user-writable weakens security (probably, I'm not sure). I assume that there are precedents in dynamic language package managers that default to non-global installation, but don't know that.

We're definitely are encouraging $CARGO_HOME/bin to be part of the search path. Even if it's not, cargo will run anything there that starts with cargo-, and multirust installs binaries to user-writable places. There are several opportunities to inject code into Rust if you have local access.

[andrewtj]

It'd be strange to use Cocoa conventions on OS X for something that isn't built with Cocoa and isn't pretending to be a native .app.

[Luthaf]

If no one's really sure of what happens on OSX, then we should probably continue doing what we're doing today there

It'd be strange to use Cocoa conventions on OS X for something that isn't built with Cocoa and isn't pretending to be a native .app.

Yeah, on OSX, ~Library is only used by graphical applications. Command-line applications usually use the same directory structure as on Linux. That is also what brew and brewed command-line tools do.

[MagaTailor]

I've just discovered this PR; on Linux I like the old behaviour better (being able to wipe or backup a single directory) so hopefully it all ends up on an opt-in/opt-out basis.

A use case on an SD based ARM system - operate entirely in ramdisk after decompressing a cargo tarball (.cargo contains everything incl. overrides) then just wipe the directory. Much faster and doesn't trash the card.

[tbu-]

@petevine It's opt-out, you can use the environment variable CARGO_HOME.

[tbu-]

@alexcrichton What's the status of this pull request? It seems that there's quite some controversy on changing the Linux behavior. Could we land the Windows version? The situation there is worse than the Linux one: We use the standards from a different platform (dot directories, which don't even interact correctly with the Windows Explorer, e.g. you can't name a file with a leading dot, see below). See also https://internals.rust-lang.org/t/beta-testing-rustup-rs/3316/21.

[retep998]

@tbu- Of course you can name a file with a leading period, you just have to trick explorer into doing it by naming it .foo. and it'll automatically convert it to .foo and work just fine.

[tbu-]

@retep998 Oh, nice trick. I did not know that one when I tried to create .gitignores. :)

[Screwtapello]

Some people in this thread have wondered whether Cargo would be the first package-manager to follow the XDG spec for per-user installation, and whether that would cause user confusion. For what it's worth, Python has looked for installed libraries in ~/.local since Python 2.6 (2013), and the Python package manager ("pip") has a ticket for installing there by default. Although that change hasn't landed in the official tree, apparently Ubuntu have already patched their build and Debian will have the same change soon. Even without that change, pip stores its config in ~/.config and caches downloaded packages in ~/.cache (both since the end of 2014).

As for ~/.local/bin not being on the default $PATH, apparently it is in Fedora, there's a ticket to make the same change in Debian.

[notriddle]

I think @wycats may be able to add more here, but my understanding is that it's just "one more layer" which may not be guaranteed to work. It's often difficult to imagine these sorts of situations, but there are so many embedded flavors of Linux around that any configuration is bound to exist somewhere, and there are bound to exist configurations where xdg doesn't work but just using the home directory does.

You expect people to be using Cargo, a program that exists to download source code and drive a compiler, on embedded controllers?

[tbu-]

@notriddle The standard uses a hardcoded directory .config unless an environment variable is set. It's unclear which of these steps shouldn't work on "embedded Linux".

[frewsxcv]

On OS X, use ~/Library/Caches/Cargo for cache files, ~/Library/Cargo
for config files and ~/Library/Cargo/bin for binary files.

I'm on OS X. How is this an improvement? I don't really care where my cache files end up, but what's the justification for moving my config files out of ~/?

[notriddle]

The standard uses a hardcoded directory .config unless an environment variable is set. It's unclear which of these steps shouldn't work on "embedded Linux".

I don't understand why people are concerned about cargo working on embedded Linux anyway. You need to be able to run the resulting binaries, but this won't affect them.

[tbu-]

@frewsxcv These paths would still be inside your home directory (note the ~, it stands for the home directory of the current user). But you should probably rather look at the RFC for discussion: rust-lang/rfcs#1615.

[frewsxcv]

These paths would still be inside your home directory

I meant directly in the home directory, not a subdirectory.

[jan-hudec]

@frewsxcv,

I meant directly in the home directory, not a subdirectory.

Things are never installed directly in the home directory. Currently they are installed in subdirectory called ~/.cargo. That is the old Unix ad-hoc approach that lead to zillions of .something files and directories in your home. The proposal is to instead put them under ~/Library, which is where MacOS native applications put things for sake of consistency (with other applications on each platform rather than between platforms, because platforms are not consistent with each other anyway). and to avoid adding to that zillion of .something items.

[tbu-]

@jan-hudec Please go to rust-lang/rfcs#1615, the discussion is happening there.

[alexcrichton]

Closing in favor of the RFC for now (clearing out the queue)