Threading custom spans through lexer/parser

[viperML]

After having written a lexer and a parser for my lang, I wonder how should implement my spans, having in the back of my mind consuming them with ariadne.

So the first question is: should the spans of the second step (parser) refer to the original input text, or would it be "better" (for any meaning of it) to refer to spans of the previous step (lexer)?

I was trying to have my lexer output spanned tokens, and my parser input spanned tokens, but I don't really know if I should use any type in the library or pass around tuples.

[zesterer]

Spans should always refer to the original text (and so be preserved throughout the various compiler passes, where possible) because their only purpose is for use in diagnostics (I guess with the exception of some sort of macro system, but that's a whole other kettle of fish).

Several of the examples in the repository handle spans through both lexer and parser (and beyond), you might find them useful. For example, nano_rust.rs.

[viperML]

Looking at nano_rust, I see that expr_parser uses a typle (Span, Token), and then you use .map_extra(|..., extra| ... extra.span()

My guess is that the parser automatically uses the Span portion of the input tuple as the span of extra.span() ?

Ninja edit: nevermind, it was a Spanned output..

[zesterer]

Yes, chumsky will automatically generate appropriate spans for you that cover whatever input a parser consumed. You can even use a custom span type by implementing the chumsky::Span trait (if you have an existing type already). Otherwise, chumsky provides the SimpleSpan type.

[viperML]

The piece of information I was probably missing was this type: https://github.com/zesterer/chumsky/blob/24514d6bd1924c6b3511ed412d52f3e92ea2775e/examples/nano_rust.rs#L186C85-L186C85