Expose span locations on stable

[dtolnay]

In the way suggested by #165 (comment).

[mitsuhiko]

Can confirm this works!

[dtolnay]

Tagging wip because this needs tests. Does not work when I tried it:

use proc_macro2::TokenTree;

fn main() {
    let sp = syn::parse_str::<TokenTree>("testing").unwrap().span();
    println!("start line={} column={}", sp.start().line, sp.start().column);
    println!("end line={} column={}", sp.end().line, sp.end().column);
}

start line=1 column=0
end line=1 column=0

I am not going to be able to get to this today. @mitsuhiko could you investigate?

[mitsuhiko]

Yes, will investigate. Information seems absent indeed.

[alexcrichton]

If we're going to offer this in a published stable version then I think we can probably go ahead and do this without a feature. The proc-macro2 crate can't have many major version bumps due to the degree of reliance on it through syn in the proc-macro ecosystem, so being stable is a big commitment. If we're willing to commit to this as a stable feature then I think it's worth providing by default.

I think that may also simplify the cfg story here (which I am always confused by every time I look at this!). That way outside of a proc-macro context you always have accurate information, but in a proc-macro context you'll only have accurate information if a nightly feature toggle is enabled.

[dtolnay]

I kept a feature because last time I checked there was a big performance hit for Span going from 0 bytes to 8 bytes. There are a lot of spans in the syntax tree...

[alexcrichton]

Ah ok that makes sense to me, and in that case seems fine to document as such as to why it's a feature

[mystor]

Hmm, I wonder if a span interning system could compress span sizes down even more, and if that would be helpful here.

[dtolnay]

I added a note in build.rs about the performance hit.

[dtolnay]

I think this works now.

I filed #167 to follow up on mitigating the performance impact.

[mitsuhiko]

Is this blocked on #167 or is it mergeable as such?

[dtolnay]

I don't think this needs to block on the performance work, because the increase in size of Span only happens if you opt in to the new feature. Thanks for the reminder!

[dtolnay]

Published in 0.4.27.

[chinedufn]

@dtolnay noob question here - is this performance hit increased memory, noticeably slower speed for working with a TokenStream.. or both.. or something else?

#167 suggests that it's increased memory but I'm not sure if that's also having a big impact on other things?

Trying to better understand whether or not to start taking advantage of this.

Thanks!!

[dtolnay]

@chinedufn it was noticeably longer compile time in debug mode for Serde's test suite the last time I tried it. You could enable the feature and check the compile time for some crate that uses your macro heavily.

[chinedufn]

Awesome advice, perfect, thanks!!

[chinedufn]

Hmm so I'm seeing some weird behavior for span locations

Take the following proc macro

#[test]
fn text_space_before_end_tag() {
    assert_eq!(
        &html! { <div>Before End Tag </div> }.to_string(),
        "<div>Before End Tag </div>"
    )
}

For the token Tag, the span is perfectly correct. Its span's end column is 36, which .. after counting characters it's definitely the 36th column.

However, the < in </div has a start column of 37.

I would expect it to be 38 here since there is a space between the end of the word Tag and the <.

Does that sound familiar at all? Held off on pushing up something easily reproducible in case there is an obvious reason here that you might already know but if not I can totally do that if that would help!

[alexcrichton]

@chinedufn that's likely actually a bug in rustc itself, want to see if you can reduce it to a standalone proc-macro and report it upstream?

[chinedufn]

Will do, thanks!

[chinedufn]

For posterity! -> rust-lang/rust#58958