add track_path::path fn for usage in proc_macros

[drahnr]

Adds a way to declare a dependency on external files without including them, to either re-trigger the build of a file as well as covering the use case of including dependencies within the rustc invocation, such that tools like sccache/cachepot are able to handle references to external files which are not included.

Ref #73921

[rust-highfive]

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @matthewjasper (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

[drahnr]

Emits # file-dep:{} in the dependency file with the same escape logic as #71858 for the path.

I'd appreciate some bikeshedding on better naming in general, track_path could be misleading to be related to track_caller.

[petrochenkov]

r? @petrochenkov

[petrochenkov]

@drahnr
I understand the purpose of proc_macro::tracked_path::path, it is supposed to be used in proc macros to add dependencies on files.

But what is the purpose of the track_path! macro? Where is it supposed to be used?

• Not in build scripts, they already have cargo:rerun-if-changed=my_file (https://doc.rust-lang.org/cargo/reference/build-scripts.html)
• Not in proc macros, they already have proc_macro::tracked_path::path as mentioned above.
• Not in declarative macros, all ways to access files from declarative macros are already tracked.

[petrochenkov]

I think it's ok to add an interface like proc_macro::tracked_path::path that only adds a path to tracked files and does nothing else.

In #73921 (comment) I mentioned more high level interfaces for reading files as token streams (and simultaneously tracking them, of course).
However, the baseline tracking-only interface makes sense too, because it's always possible that your procedural macro will want to read files in some advanced ways (e.g. mmaping their parts into memory), which we cannot cover by higher level APIs like read_file_to_string or read_file_to_token_stream.

[drahnr]

Thanks for your feedback @petrochenkov !

I think it's ok to add an interface like proc_macro::tracked_path::path that only adds a path to tracked files and does nothing else.

In #73921 (comment) I mentioned more high level interfaces for reading files as token streams (and simultaneously tracking them, of course).
However, the baseline tracking-only interface makes sense too, because it's always possible that your procedural macro will want to read files in some advanced ways (e.g. mmaping their parts into memory), which we cannot cover by higher level APIs like read_file_to_string or read_file_to_token_stream.

Reasoning was mostly to get started with the rustc codebase and make sccache work for certain crates. In the sccache repo there was mention of another issue (I can't seem to find it right now) so I did not want to mix the concerns in one PR but rather have to distinct.

@drahnr
I understand the purpose of proc_macro::tracked_path::path, it is supposed to be used in proc macros to add dependencies on files.

But what is the purpose of the track_path! macro? Where is it supposed to be used?

* Not in build scripts, they already have `cargo:rerun-if-changed=my_file` (https://doc.rust-lang.org/cargo/reference/build-scripts.html)

* Not in proc macros, they already have `proc_macro::tracked_path::path` as mentioned above.

* Not in declarative macros, all ways to access files from declarative macros are already tracked.

I was taking the case into consideration where a shared crate is used, both in context of an regular application and a proc_macro. In the library context itself the shared code static libarary context, proc_macro::tracked_path::path would not be available iiuc, hence to allow this use case I suggest to add the builtin macro track_path.

Case example: https://github.com/djc/askama/blob/main/askama_shared/src/generator.rs#L796-L799

[drahnr]

@rustbot label -S-waiting-on-author +S-waiting-on-review

[drahnr]

@rustbot label -S-waiting-on-author +S-waiting-on-review

[Xanewok]

Diff in /checkout/compiler/rustc_interface/src/passes.rs at line 28:
 use rustc_plugin_impl as plugin;
 use rustc_query_impl::Queries as TcxQueries;
 use rustc_resolve::{Resolver, ResolverArenas};
+use rustc_serialize::json;
 use rustc_session::config::{CrateType, Input, OutputFilenames, OutputType, PpMode, PpSourceMode};
 use rustc_session::lint;
 use rustc_session::output::{filename_for_input, filename_for_metadata};
Diff in /checkout/compiler/rustc_interface/src/passes.rs at line 37:
 use rustc_span::FileName;
 use rustc_trait_selection::traits;
 use rustc_typeck as typeck;
-use tracing::{info, warn};
-use rustc_serialize::json;
 use tempfile::Builder as TempFileBuilder;
+use tracing::{info, warn};
 
 use std::any::Any;
 use std::cell::RefCell;
Diff in /checkout/compiler/rustc_interface/src/passes.rs at line 597:
         // Account for explicitly marked-to-track files
         // (e.g. accessed in proc macros).
         let file_depinfo = sess.parse_sess.file_depinfo.borrow();
-        let extra_tracked_files = file_depinfo
-            .iter()
-            .map(|path_sym| {
-                let path = PathBuf::from(&*path_sym.as_str());
-                let file = FileName::from(path);
-                escape_dep_filename(&file.prefer_local().to_string())
-            });
+        let extra_tracked_files = file_depinfo.iter().map(|path_sym| {
+            let path = PathBuf::from(&*path_sym.as_str());
+            let file = FileName::from(path);
+            escape_dep_filename(&file.prefer_local().to_string())
+        });
         files.extend(extra_tracked_files);
 
         if let Some(ref backend) = sess.opts.debugging_opts.codegen_backend {
Running `"/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/rustfmt" "--config-path" "/checkout" "--edition" "2018" "--unstable-features" "--skip-children" "--check" "/checkout/compiler/rustc_target/src/abi/call/x86_win64.rs" "/checkout/compiler/rustc_interface/src/proc_macro_decls.rs" "/checkout/compiler/rustc_target/src/abi/call/msp430.rs" "/checkout/compiler/rustc_interface/src/passes.rs" "/checkout/compiler/rustc_target/src/abi/call/powerpc.rs" "/checkout/compiler/rustc_target/src/abi/call/sparc64.rs"` failed.
If you're running `tidy`, try again with `--bless`. Or, if you just want to format code, run `./x.py fmt` instead.

Looks like it needs ./x.py fmt

[petrochenkov]

I guess the moral of the story is that you really have to test locally, otherwise landing a PR turns into an attrition warfare.

[drahnr]

While I ran the tidy before the last set of nits, it's on me, failing to follow the guide to re-run the test + tidy checks.
Sorry again, I'll be better.

[drahnr]

./x.py test as well as ./x.py test tidy --bless passed.

@rustbot label -S-waiting-on-author +S-waiting-on-review

[petrochenkov]

Thanks!
@bors r+

[bors]

📌 Commit 67e6a81 has been approved by petrochenkov