RFC: Templating CARGO_TARGET_DIR to make it the parent of all target directories

[poliorcetics]

Introduce a new templating option for CARGO_TARGET_DIR to tell it to move the crate/workspace's target
directory into a crate/workspace-specific subdirectory of the configured path, allowing for easy exclusion from save tools for example.

Rendered
FCP (previous FCP)

[ogoffart]

What I would like to see is a "global" target directory for all projects, that way, common dependency artifacts would be shared. (Not all projects would need to compile syn and other common dependencies over and over again)

I think that would solve the problems that this RFC tries to solve. But I realise this has other challenges (different features set of dependencies, when/how to clean that cache, ...)

[poliorcetics]

What I would like to see is a "global" target directory for all projects, that way, common dependency artifacts would be shared. (Not all projects would need to compile syn and other common dependencies over and over again)

I think that would solve the problems that this RFC tries to solve. But I realise this has other challenges (different features set of dependencies, when/how to clean that cache, ...)

Already possible by setting CARGO_TARGET_DIR globally, and it encounters the expected problems with features and cleaning

[poliorcetics]

If this is ever slated to merge I'll squash then, while the RFC is ongoing I'll make new commits to allow people to follow the evolution more easily :)

[poliorcetics]

Sorry it took me a long time to answer to comments

[epage]

@rfcbot resolve cargo-target-dir

[weihanglo]

This will be the very first templating supported in .cargo/config.toml. Is the syntax flexible enough to extend to other configuration like [build.rustflags], [env]. I can see it useful in other issues like

• Embed config-relative paths as a substring cargo#12193
• No way to pass environment information to compiler flags defined in Cargo configuration cargo#7352

[poliorcetics]

I see no reason for it to be limited.

{name} is not used anywhere currently IIRC and the most likely places it can crop up unexpectedly are paths, where this RFC already discuss how unlikely that is.

I suppose anything where cargo shells out to another program can also have it come up, but I'm not sure we can even design for every possible target.TTT.runner in eternity

[weihanglo]

Nice. Thanks for the reply. Maybe it is worth putting this in future posibilities in some way?

[poliorcetics]

Done in d549cb1 :) I tried noting possible future problems I could find in 10 minutes of thinking and why they're not limiting

[kornelski]

Cargo has these dilemmas about the location of the target dir, because Cargo has conflated two concepts:

1. intermediate build products (3rd party dependencies, object files, compiler-internal caches)
2. final build artifacts (linked build targets of the current workspace only)

Users want 1 in a centralized location to deduplicate/reuse intermediate products (files they never touch). And users want 2 in ./target to run or copy the files easily, from a convenient location that has no global naming clashes.

As long as Cargo will continue to dump both kinds of products into the same directory, it will get complaints about making build results difficult to reach, or even overwriting each other, while at the same time getting requests to unify the directory.

Adding a hash to the path is not solving either of these problems. The hash keeps common dependencies built redundantly and puts the final artifacts in an inconvenient location.

[sunshowers]

Cargo has these dilemmas about the location of the target dir, because Cargo has conflated two concepts:

1. intermediate build products (3rd party dependencies, object files, compiler-internal caches)
2. final build artifacts (linked build targets of the current workspace only)

Users want 1 in a centralized location to deduplicate/reuse intermediate products (files they never touch). And users want 2 in ./target to run or copy the files easily, from a convenient location that has no global naming clashes.

As long as Cargo will continue to dump both kinds of products into the same directory, it will get complaints about making build results difficult to reach, or even overwriting each other, while at the same time getting requests to unify the directory.

Adding a hash to the path is not solving either of these problems. The hash keeps common dependencies built redundantly and puts the final artifacts in an inconvenient location.

This isn't a full solution to this problem, but (having used targo full time for over a year) it solves several other problems that are roughly as important.

[poliorcetics]

Cargo has these dilemmas about the location of the target dir, because Cargo has conflated two concepts:

1. intermediate build products (3rd party dependencies, object files, compiler-internal caches)

2. final build artifacts (linked build targets of the current workspace only)

Users want 1 in a centralized location to deduplicate/reuse intermediate products (files they never touch). And users want 2 in ./target to run or copy the files easily, from a convenient location that has no global naming clashes.

As long as Cargo will continue to dump both kinds of products into the same directory, it will get complaints about making build results difficult to reach, or even overwriting each other, while at the same time getting requests to unify the directory.

I think you misunderstood the RFC.

Point 2. final build artifacts (linked build targets of the current workspace only) is already being worked by the out-dir/artifact-dir discussion, which is mentioned in the RFC, and out of scope for this one.

Users want 1 in a centralized location to deduplicate/reuse intermediate products (files they never touch).

That is already possible, always has been I think, just by setting a cargo-target-dir globally. The RFC discusses this and also why this can be an issue for some (e.g all the people that commented on the original issue, which is linked at the start of the RFC).

Adding a hash to the path is not solving either of these problems. The hash keeps common dependencies built redundantly and puts the final artifacts in an inconvenient location.

Because the problems you're describing are not what this RFC is proposing to solve.

What this RFC is proposing is to gather all target directories under a common "base" path while keeping them separate for each project, for the purposes of caching, cleaning local repositories, sharing build artifacts via NFS or any other usage that people will think of or have mentioned in the original issue.

This RFC proposes an opt-in option: it won't make life harder for people that don't use it, just like lots of people don't override their cargo-target-dir path and so don't have to deal with it.

Separating intermediate and final build artifacts or making them (either or) easy to access are not goals: those are either cargo issues or RFCs already.

[epage]

This RFC proposes an opt-in option: it won't make life harder for people that don't use it, just like lots of people don't override their cargo-target-dir path and so don't have to deal with it.

To add to this, I want to eventually change this but I feel its dependent on finalizing --out-dir / --artifact-dir which I suspect will eventually lead to separate locations for intermediate artifacts and final artifacts. The big question to deal with is "what is a final artifact?". For most people, its just bins and dylibs. But some also consider examples, tests, etc. However, thats a topic for --artifact-dir.

[kornelski]

What I'm trying to say that if Cargo redefined CARGO_TARGET_DIR to be only for artifacts, and not temp files or registry deps, then it would achieve the same goals motivating this RFC, but without the drawbacks of moving ./target.

To avoid hijacking this thread, I've posted: rust-lang/cargo#14125

[Ddystopia]

Just to clarify, would this RFC allow a use case when you compile the same project with, for example, different RUSTFLAGS alternatingly and it will not invalidate cache every time, but keep 2 of them and use appropriate?

[poliorcetics]

Just to clarify, would this RFC allow a use case when you compile the same project with, for example, different RUSTFLAGS alternatingly and it will not invalidate cache every time, but keep 2 of them and use appropriate?

Nop, only the location of the cache would be affected, not its workings

[epage]

@rfcbot cancel

[rfcbot]

@epage proposal cancelled.

[epage]

@rfcbot postpone

We discussed this, rust-lang/cargo#6790, and rust-lang/cargo#14125 in the Cargo team meeting today. In that discussion, we came up with an idea that merges these different ideas, see rust-lang/cargo#14125 (comment).

However, we recognize that we don't want an "ideal" to forever block people getting functionality now and we were thinking of time boxing that effort for 1 year after which we could re-evaluate whether we should go with a "short term solution" of this RFC.

@poliorcetics thank you for all of your work on this proposal. You've put in a lot of work on this and this RFC serves as the basis for this alternative idea we are exploring. If you'd like to continue with that as an extension of this work, we'd welcome your help! I hope you don't see this as "a rejection" or "this was thrownaway".

[rfcbot]

Team member @epage has proposed to postpone this. The next step is review by the rest of the tagged team members:

• @Eh2406
• @Muscraft
• @arlosi
• @ehuss
• @epage
• @joshtriplett
• @weihanglo

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[rfcbot]

The final comment period, with a disposition to postpone, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.

This is now postponed.