span.start() and span.end() is not working in proc_macro context even in nightly version.

[abishekatp]

Hi, @dtolnay Thank for you great work. I am trying to use proc_macro2 to parse the css content in the project I am working on. You can find it here.

The Problem

• Based on this documentation span.start() and span.end() should work on both inside proc_macro context and outside proc_macro context in the nightly version of the rust. This is is working as expected in the outside proc_macro context.
• The problem is the values are always 0 inside the proc_macro context.

This is the piece of code I am trying to use in both scenarios.

pub(crate) fn add_spaces(
      source: &mut String,
      span: proc_macro2::Span,
      pre_line: &mut usize,
      pre_col: &mut usize,
 ) {
      let start = span.start();
      let end = span.end();
      let cur_col = start.column;
      let cur_line = start.line;
      if *pre_line == cur_line && cur_col > *pre_col {
          source.push(' ');
      }
      *pre_col = end.column;
      *pre_line = end.line;
  } 

[tonyg]

I have this problem too. I'm bisecting now. (I'm afraid I have to use nightly rather than stable for other reasons, so the bisection will be in terms of nightlies.)

• broken: rustc 1.74.0-nightly (5ae769f06 2023-09-26), proc_macro2 1.0.67
• working: rustc 1.71.0-nightly (7f94b314c 2023-04-23), proc_macro2 1.0.56

Version 1.0.56 doesn't build with 1.74.0, but luckily 1.0.67 builds with 1.71.0, and yields the broken results, so I'll stick with that and see where I get.

It looks like the change from 1.0.56 to 1.0.57 is where the breakage appears, at least for me!

Oh. Sure enough, from 1.0.57 and on to the current main revision, we have in wrapper.rs

    #[cfg(span_locations)]
    pub fn start(&self) -> LineColumn {
        match self {
            Span::Compiler(_) => LineColumn { line: 0, column: 0 },
            Span::Fallback(s) => s.start(),
        }
    }

Presumably this is because LineColumn no longer exists in proc_macro! This will have changed somewhere between 1.71.0-nightly and the current nightlies.

However, there are separate line() and column() functions. Patching proc_macro2 with the following implementations seems to work fine for my little tests so far:

    #[cfg(span_locations)]
    pub fn start(&self) -> LineColumn {
        match self {
            #[cfg(proc_macro_span)]
            Span::Compiler(s) => LineColumn {
                line: s.line(),
                column: s.column(),
            },
            #[cfg(not(proc_macro_span))]                                                       
            Span::Compiler(_) => LineColumn { line: 0, column: 0 },
            Span::Fallback(s) => s.start(),
        }
    }

    #[cfg(span_locations)]
    pub fn end(&self) -> LineColumn {
        match self {
            #[cfg(proc_macro_span)]
            Span::Compiler(s) => LineColumn {
                line: s.end().line(),
                column: s.end().column(),
            },
            #[cfg(not(proc_macro_span))]                                                       
            Span::Compiler(_) => LineColumn { line: 0, column: 0 },
            Span::Fallback(s) => s.end(),
        }

Now, how to fix this properly? Easiest would be to alter proc_macro2 to include the implementations just above. However then proc_macro2 wouldn't be usable with older versions of the toolchain. Is that a problem (@dtolnay?)?

An alternative would be to expose line() and column() as new methods on Spans (with some kind of end().line() and end().column()), following the proc_macro approach. @dtolnay, your thoughts here would be very welcome.

[tonyg]

span_start_bug.zip

^ Here's the tiny crate I've been using to explore the problem.

Buggy rm -rf target; cargo build --example exercise output:

   Compiling proc-macro2 v1.0.67
   Compiling unicode-ident v1.0.12
   Compiling macro-provider v0.0.0 (/home/tonyg/src/span_start_bug)
SPAN START: LineColumn { line: 0, column: 0 } END: LineColumn { line: 0, column: 0 }
SPAN START: LineColumn { line: 0, column: 0 } END: LineColumn { line: 0, column: 0 }
SPAN START: LineColumn { line: 0, column: 0 } END: LineColumn { line: 0, column: 0 }
SPAN START: LineColumn { line: 0, column: 0 } END: LineColumn { line: 0, column: 0 }
SPAN START: LineColumn { line: 0, column: 0 } END: LineColumn { line: 0, column: 0 }
    Finished dev [unoptimized + debuginfo] target(s) in 1.18s

Correct output:

   Compiling proc-macro2 v1.0.67 (/home/tonyg/src/span_start_bug/scratch/proc-macro2)
   Compiling unicode-ident v1.0.12
   Compiling macro-provider v0.0.0 (/home/tonyg/src/span_start_bug)
SPAN START: LineColumn { line: 2, column: 40 } END: LineColumn { line: 2, column: 41 }
SPAN START: LineColumn { line: 2, column: 42 } END: LineColumn { line: 2, column: 43 }
SPAN START: LineColumn { line: 2, column: 44 } END: LineColumn { line: 2, column: 45 }
SPAN START: LineColumn { line: 2, column: 46 } END: LineColumn { line: 2, column: 47 }
SPAN START: LineColumn { line: 2, column: 48 } END: LineColumn { line: 2, column: 49 }
    Finished dev [unoptimized + debuginfo] target(s) in 1.28s

[tonyg]

Unfortunately I wonder if this is going to be hard to write a test case for since it involves running in proc_macro context? There are existing test cases that check column/line values, but since they pass, presumably they use the Fallback.

[tonyg]

(Here's the commit where the code for retrieving the LineColumn from proc_macro was removed: 5f9d3fe)

[abishekatp]

Hi, @dtolnay Thank for you great work. I am trying to use proc_macro2 to parse the css content in the project I am working on. You can find it here.

The Problem

• Based on this documentation span.start() and span.end() should work on both inside proc_macro context and outside proc_macro context in the nightly version of the rust. This is is working as expected in the outside proc_macro context.
• The problem is the values are always 0 inside the proc_macro context.

This is the piece of code I am trying to use in both scenarios.

pub(crate) fn add_spaces(
      source: &mut String,
      span: proc_macro2::Span,
      pre_line: &mut usize,
      pre_col: &mut usize,
 ) {
      let start = span.start();
      let end = span.end();
      let cur_col = start.column;
      let cur_line = start.line;
      if *pre_line == cur_line && cur_col > *pre_col {
          source.push(' ');
      }
      *pre_col = end.column;
      *pre_line = end.line;
  } 

Current Solution:

To resolve this problem I temporarily used following approach. The problem was the proc_macro2::span.start().column was not working inside proc_macro scope but it was working fine for normal(outside proc_macro scope) use cases. Also I can't use proc_macro::span.unwrap().start().column() outside proc_macro scope. So I conditionally switched between these two cases based on the function input attribute is_proc_macro: bool. This resolved the problem for me. Now I can use this utility both inside and outside proc_macro scope. You can find the crate here.

pub(crate) fn add_spaces(
    source: &mut String,
    span: proc_macro2::Span,
    pre_line: &mut usize,
    pre_col: &mut usize,
    is_proc_macro: bool,
) {
    let mut start_col = span.start().column;
    let mut start_line = span.start().line;
    let mut end_col = span.end().column;
    let mut end_line = span.end().line;
    if is_proc_macro {
        start_col = span.unwrap().start().column();
        start_line = span.unwrap().start().line();
        end_col = span.unwrap().end().column();
        end_line = span.unwrap().end().line();
    }
    let cur_col = start_col;
    let cur_line = start_line;
    if *pre_line == cur_line && cur_col > *pre_col {
        source.push(' ');
    }
    *pre_col = end_col;
    *pre_line = end_line;
}

note: you have to add #![feature(proc_macro_span)] in the top you lib.rs file to use span.unwrap()

[douglas-raillard-arm]

I'd love to see some progress on this one. The use case is emitting some C code out of a "rust function" that contains the C code as a single string literal for the function body. Having access to the precise span of that string literal would allow me to add a #line preprocessor directive in the generated C code, so that any C compilation error would report the location in the Rust source.

Currently, this can only be done by emitting a line!() call in the token stream, which kind of works but is off by some amount.