Tracking issue for RFC 2091: Implicit caller location

[aturon]

This is a tracking issue for the RFC "Implicit caller location" (rust-lang/rfcs#2091).

Steps:

• Implement the RFC (cc @rust-lang/compiler -- can anyone write up mentoring instructions?)
• Adjust documentation (see instructions on forge)
• Stabilization PR (see instructions on forge)

Unresolved questions:

• If we want to support adding #[track_caller] to trait methods, the redirection
  pass/query/whatever should be placed after monomorphization, not before. Currently the RFC
  simply prohibits applying #[track_caller] to trait methods as a future-proofing measure.

• Diverging functions should be supported.

• The closure foo::{{closure}} should inherit most attributes applied to the function foo, in
  particular #[inline], #[cold], #[naked] and also the ABI. Currently a procedural macro
  won't see any of these, nor would there be anyway to apply these attributes to a closure.
  Therefore, #[rustc_implicit_caller_location] currently will reject #[naked] and ABI, and
  leaving #[inline] and #[cold] mean no-op. There is no semantic reason why these cannot be
  used though.

[eddyb]

If we want to support adding #[track_caller] to trait methods,

This refers to impl of a trait, not the declarations within the trait, right?

IMO the easiest way to implement this involves no procedural macros or MIR passes.
We'd just pass extra arguments on direct calls, and require a shim to get function pointers of the function. That would also surely work with trait methods, since it'd be post-monomorphization.

To explain, the shim (which we may already have some variation of) would "just" do the direct call to the function, which would pass the location of the shim, the same as the original function.

[nikomatsakis]

This refers to impl of a trait, not the declarations within the trait, right?

I can't remember. =) But I suspect so. I don't think there's anything special about "trait methods" per se -- it's really about dynamic dispatch.

[nikomatsakis]

@kennytm had a prototype implementation:

https://github.com/kennytm/rust/tree/caller-info-4

Maybe @kennytm you can summarize the approach you took? Do you think you'll have time to rebase etc? I'd like ideally to get @eddyb to buy in to the overall approach. =)

[kennytm]

The prototype implementation works like this (note that #[track_caller] was called #[implicit_caller_location] there):

1. First there will be an AST transformation pass (implemented as an attribute proc-macro in src/libsyntax_ext/implicit_caller_location.rs) which expands:

   #[track_caller]
   #[...attributes...]
   fn foo(...) -> T {
       /* code... */
   }

   into

   #[rustc_track_caller]
   #[inline]
   #[...attributes...]
   fn foo(...) -> T {
       let __closure = |__location| { /* code... */ };
       FnOnce::call_once(__closure, intrinsics::caller_location())
   }

   The purpose of this pass is to conveniently make up a new DefId.

2. Create a new MIR inlining pass (main implementation in src/librustc_mir/transform/inline.rs). This pass does two things:

   1. Force-inline any functions calls foo() where the attributes of the callee foo contains #[rustc_track_caller].

   2. If intrinsics::caller_location() is called :-

      • If it is used from the __closure, replace it by the argument __location. This is to propagate the caller location.
      • Otherwise, replace the call with the caller's location span.

3. Define the caller_location() intrinsic.

   fn caller_location() -> core::panic::Location<'static>;

4. Because Location now needs to be known by the compiler, make the Location struct a lang-item.

5. Update the standard library to use track-caller features:

   1. Change panic! to use caller_location() (or its safe counterpart, Location::caller())
   2. Add #[track_caller] to unwrap() and expect() etc.

6. Implement the -Z location-detail flag (search for location_detail) so that the filename/line/column can be redacted.

[eddyb]

I think a syntactical transformations is unnecessary because we can instead change the "direct call ABI" (vs reifying to a fn pointer, including trait vtables via miri, which could go through a MIR shim).
MIR inlining should also not be needed.

[kennytm]

@eddyb So you are suggesting to change the extern "Rust" ABI to always pass an extra argument?

[eddyb]

@kennytm Only for functions declared with the attribute and called through static dispatch, everything else would use a shim when reifying (which I think we do already in some other cases).

[kennytm]

@eddyb Maybe you could write down some mentoring instructions i.e. which files to look at 😊

[nikomatsakis]

@eddyb do it! do it! I want this feature.

[Centril]

@eddyb How are the mentoring instructions coming along?

[eddyb]

Oh, I don't recall exactly what happened here.

For making static calls do something different, src/librustc_codegen_ssa/mir/block.rs is the place you need to change, while for shims, src/librustc_mir/shim.rs is where their MIR is computed.

But the rest of the pieces, I don't know off the top of my head where they happen.

#54183 added a virtual-call-only shim, which is close to what this needs, so that can be used for inspiration, but here we need both reification to fn pointers and virtual calls.

We could start by disallowing reification/virtualization of such functions, and only implement the static dispatch ABI changes.

[arielb1]

We could start by disallowing reification/virtualization of such functions, and only implement the static dispatch ABI changes.

I don't expect this attribute to be usable on "real" functions without us implementing reification.

But for implementing this, I think one good strategy would be:

Not supporting trait methods

For now, I don't think there is sufficient reason to support implicit caller location on trait methods, and supporting that is fairly complicated. So as @aturon said on the head PR, just don't do it.

Step 0: dealing with the attribute.

You should make sure that the #[blame_caller] attribute "works". This means that:

1. It emits a feature-gate error unless the correct feature-gate is enabled (see the feature guide for more information about feature gates).
2. It emits an error when used with odd syntax (e.g. #[blame_caller(42)]),
3. It emits an error when used on something that is not a free fn/inherent impl fn.
4. It does not emit an error when used "correctly'.
5. As the RFC says: using #[naked] or extern "ABI" together with #[rustc_implicit_caller_location] should raise an error.

I'm not particularly sure what's the exact way to do this, but you can maybe find some PR that implemented an attribute for that, or ask on Discord.

Make sure to add tests.

Step 1: have reified methods and direct calls go to different LLVM functions

At this stage, I won't even add a parameter, just have the call-sites go through different paths.

So the virtual-call-only shim at #54183 is a good model for how things need to be done. You'll need to add a ReifyShim that is similar to VirtualShim in behavior (see that PR). You'll probably need to split ty::Instance::resolve to ty::Instance::resolve_direct and ty::Instance::resolve_indirect (in addition to the already-existing ty::Instance::resolve_vtable) and adjust the call-sites. Have resolve_indirect return the shim if it is needed (i.e., when calling a #[blame_caller] function).

Step 2: Add a location parameter

Step 2.0: add a location parameter to the MIR of #[blame_caller]

MIR construction is in rustc_mir::build. There's already code there handling implicit arguments, which you could work with.

You'll need to fix ty::Instance::fn_sig to also contain the new type.

Step 2.1: add an "empty" location parameter to calls from the shim.

Change the shim you added in Step 1 to pass an "empty" location parameter - some "dummy" that represents a call that uses a shim. See the other shims for how to generate things in shims. Ask @eddyb if you are fighting const eval.

Step 2.2: add a value to the location parameter to direct calls.

Find all the calls to ty::Instance::resolve_direct you added previously, and make them pass the "correct" location parameter to the function. These callsites can poobably already modify parameters because of InstanceDef::Virtual, you should be able to just follow that.

Make sure that when #[track_caller] calls are nested, you pass the location of your call-site, rather than your location. Add a test for that. Also add a test that when one of the calls in the middle is not #[track_caller], the call-site is not passed.

You should probably add a function on ty::Instance that tells you whether you need to add the caller location.

Step 3: wire up intrinsic::caller_location

See some other intrinsic for how to type-check it. Make sure it can't be called from functions that aren't #[trace_caller]!

I think it might even be the best to do the wiring in MIR construction, like the move_val_init intrinsic - that way you wouldn't even have to convince MIR optimizations not to dead-code-eliminate your new parameter or something evil like that, and to make it not be affected by "plain" MIR inlining you'll only have to change the ty::Instance::resolve_direct callsite there (you did make MIR inlining use resolve_direct, did you?).

Step 4: You have it!

Now you only need to implement the library support.

[flavius]

Is anyone working on this?

[ayosec]

If nobody is working on this I would like to try to implement it.

Do you think that it is feasible as a first contribution to the compiler?

Are the steps in #47809 (comment) still valid?

[cramertj]

Yes, those steps still sound reasonable. This would be a difficult first contribution, but if you ask questions on https://rust-lang.zulipchat.com/# and ping @eddyb and @arielb1 I'm sure you'd be able to start making progress, and there are likely others who could help out.

[ayosec]

I'm sorry for the delay. I couldn't have time to work on this until now =\

I have completed the step 0 described in #47809 (comment) . The progress is available in my branch, though I guess that I will have to wait until the pull-request is open to know is everything is OK.

About error codes: if I understand correctly, every error emitted by the compiler has its unique code, so I have to register a new E0### for every error. I assume that the final code is assigned only when patch is ready to be merged into master, so I'm using dummy codes (E0900, E0901, ...) for the new errors.

[anp]

I'm interested in helping with this, are you still working on it @ayosec? If so, is there any way for me to contribute?

[ayosec]

are you still working on it @ayosec?

Yes. I'm sorry for the lack of updates. During August I had almost no time, and I'm trying to start to continue now. At this moment I'm working with the shim to add the location parameter.

If so, is there any way for me to contribute?

Right now I spend most of the time reading and understanding how everything works, so there is not so much code to write.

Something that would be very helpful is writing the documentation for the new errors (like this or this, at the moment), since I'm not very fluent in English.

[oli-obk]

Ping @ayosec do you have time to address this or is it OK with you if someone else takes over?

[ayosec]

I'm still working on it very slowly, but feel free to take it if you want. I can try to implement another issue once I get more time.

[anp]

Alright! I've revised the PR to the reference and have opened a PR to stabilize the attribute and wrapper: #72445.

[ivnsch]

Sorry, what is the name of the flag? location-detail and redact-caller-location are returning "unknown flag"

[anp]

The flag needs implementation still: #70580.

[anp]

The stabilization PR just landed! The PR to the reference should be ready to go as well, after which I think we can close this tracking issue (at least those are the only checkboxes remaining in the top message).

[nikomatsakis]

Amazing work, @anp!

[elichai]

This is awesome!
One problem I have is that it's not possible to put #[track_caller] on closures, this is really useful when thread_local is involved because then there's a lot of logic in closures that you might want to propagate
ie

thread_local! {static DONE: bool = false;}

#[track_caller]
fn assert_done() {
    DONE.with(
        #[track_caller]
        |b| assert!(b),
    );
}

fn main() {
    assert_done();
}

[anp]

The restriction on closures were in the original RFC, I think as a result of the implementation proposed then. Right now, I can't think of a reason the current implementation couldn't support this but I could easily be missing something. I opened #74042 to discuss/track.

[xd009642]

Apologies if this has been raised before, but I've been playing around with trying to track where allocations happen with something like so:

use libc_print::libc_println;
use std::alloc::{GlobalAlloc, Layout};
use std::panic::Location;

pub struct TracedAlloc<T: GlobalAlloc> {
    pub allocator: T,
}

unsafe impl<T> GlobalAlloc for TracedAlloc<T>
where
    T: GlobalAlloc,
{
    #[track_caller]
    unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
        libc_println!("Alloc {:?} at {:?}", layout, Location::caller());
        self.allocator.alloc(layout)
    }

    #[track_caller]
    unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
        libc_println!("Dealloc {:?} at {:?}", layout, Location::caller());
        self.allocator.dealloc(ptr, layout)
    }
}

However the caller location is always the line I've put #[global_allocator] which makes #[track_caller] useless in this context.

[KodrAus]

Are we happy to close this tracking issue now that #74042 has spun off work to support the attribute on closures?

[nikomatsakis]

Sure, I guess so. Thanks again @anp for seeing this through!

[schreter]

Hi. I know this is a bit off-topic and this particular issue is already closed, but I suppose, the right people are involved here :-), so someone might help.

I implemented for our project a version of Result which uses Try trait in such a way as to track the error handling locations, so we get a "call stack" of the error at the end. This works in conjunction with #[track_caller] very well and we see the locations the error handling took.

However, there is one major deficiency in Location - it only gives us the source name and line. Yes, with that, it's possible to manually look up the function where it happened, but it would be significantly simpler to evaluate bug reports by looking at the call trace with function names (we have practiced this in a large C++ project, where basically the dev support standardized on initially evaluating everything by function names call trace, ignoring line numbers). So it would come extremely handy if the Location could be extended with another &str containing the function name (ideally with generics). It can be also a mangled name, I don't care much, but the function name is important.

Before you should "backtrace!" - yes, but... We are using heavy asynchronous processing handling errors across awaits, where the backtrace has about zero value. Similar for tracing, we can't just always trace due to performance reasons. So the error handling "call stack" is a perfect compromise - cheap and sufficiently valuable (except that it's missing the function name).

Any suggestion where/how to address this issue (Location extension by file name)? According to my code study of the Rust compiler code, it should basically boil down to getting the function name from the current Span and adding it in addition to the file name to the generated const Location here:

rust/compiler/rustc_codegen_cranelift/src/common.rs

Line 351 in 012720f

let const_loc = self.tcx.const_caller_location((

Thanks & regards,

Ivan

[jplatte]

I remember asking about this quite a while ago and getting a response along the lines of "unlikely to happen for perf or code size reasons". Unfortunately I can't find an issue about it, so I guess it must have been on Zulip.

I think creating a separate issue would be more useful than continuing this conversation here, in any case.

[schreter]

I remember asking about this quite a while ago and getting a response along the lines of "unlikely to happen for perf or code size reasons". Unfortunately I can't find an issue about it, so I guess it must have been on Zulip.

Well, it's clear that this will increase generated constants segment by potentially quite a bit, because instead of a single string per file we'll need to store many strings per file. So it could be made optional at compilation time. But I personally don't think it'll be significantly slower (during compilation; runtime is obviously unaffected).

I think creating a separate issue would be more useful than continuing this conversation here, in any case.

Sure. Simply a new top-level issue w/o any special tags? The question is how to get it to the attention of relevant people who could do something about it.

[est31]

If you have the source code, it should be quite easy to build a tool that translates the Location to the name of the function. Location contains the file name, as well as line and column, so all you need to do is go to that specific line and column, and then parse the next ident. Done easily with syn where you can write a custom visitor with a visit_ident function, like this. Then you just compare the line/col number with the line/col number of the Location, done. Maybe if the source code is not public you can export the Location via e.g. json and import it via the tool.

[schreter]

If you have the source code, it should be quite easy to build a tool that translates the Location to the name of the function.

Of course it could be translated. But that's another step, which makes it quite cumbersome. The compiler already knows it at compile time and aside from costing more space in the generated executable (string section), there should be no adverse effects of having the function name in the Location. It would in general allow building various tools which use the location for debugging purposes, which in turn would help the community in general.

Let me move this to a new issue, for further discussion.