Implement template strings

[dylwil3]

This PR implements template strings (t-strings) in the parser and formatter for Ruff.

I have attempted to arrange the commits for easy review commit-by-commit. They are labeled by what piece of the code they touch, and I have written extended commit messages for some of them to indicate the important bits.

Please let me know if there's anything that can be made clearer or if I can help make the review process smoother!

[github-actions]

mypy_primer results

No ecosystem changes detected ✅

[github-actions]

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Formatter (stable)

✅ ecosystem check detected no format changes.

Formatter (preview)

✅ ecosystem check detected no format changes.

[carljm]

So cool!

I'm assuming the ty bits here are just the minimum to not panic, and we can handle the ty comments I'd have (e.g. some tests, maybe some code sharing between infer_fstring_expression and the f-string portion of infer_tstring_expression) in a separate ty-focused PR later on. Going to leave review on this one to people who know Ruff.

[MichaReiser]

Thanks for working on this. Exciting to seeing this come together.

I've only reviewed the parser, ast and parts of the linter at this point. Overall, my impression is that we went too far with copy pasting. We should consider more carefully where it is important to have different representations for t- and f-strings and where it is fine to share the same structs or parametrizing functions if they're identical except for e.g. some token kinds. Sharing the same structures should help with reusing more code as well.

[dylwil3]

Overall, my impression is that we went too far with copy pasting

That's fair - I will see what can be done about merging more logic! Thank you!

[AlexWaygood]

(I'll also leave this mostly to Micha/Dhruv, but please feel free to ping me if there's something specific you want my opinion on!)

[dhruvmanila]

Looking at Micha's review, I'll wait for the follow-up changes to de-duplicate and then review but I agree that we should parameterize the methods and do minimal changes where f-strings and t-strings are different.

[dylwil3]

What's the motivation for having different nodes (not questioning whether we should have different FString and TString root nodes but do all nodes have to be different?)

Just to respond in general to the code duplication (I will still work on merging logic):

My main motivation for keeping so many things separate (e.g. the AST nodes) was to mimic to some extent what is being done in the CPython implementation. Granted, much of their parser is generated, so it's possible they are more comfortable with repetition than we ought to be, but it seemed like, down the line, if changes are introduced such that the behavior of t-strings and f-strings diverges further, this would be easier to maintain. However, I understand if this is too cautious/"YAGNI"-thinking. Anyway, just wanted to put on the record that I wasn't just being lazy for laziness sake 😄

[dylwil3]

Two quick questions as I continue to merge various things:

1. Is there a standard pattern in Rust to get around the fact that we don't have dependent types? There are several places where merging functions would be possible except that the return types are different. So like, we have a function of signature (data) -> TStringThing and a function of signature (data) -> FStringThing. I want instead to do (data, enum variant?) -> ?? // FStringThing/TStringThing depending on enum variant? I guess am I supposed to do (data) -> T and parameterize by T instead where T is allowed to either be FStringThing or TStringThing? Hopefully this question makes sense...
2. When I made both FStringElement and TStringElement share a node type (FTStringLiteralElement) (I know you hate the name!), this broke the code generation. Is there some other way I'm supposed to do this in ast.toml or should I modify the script? (Maybe a question for @dcreager ?)

[MichaReiser]

For 1. Hard to say without seeing a concrete example. The alternative is to use a trait with an associated type:

trait InterpolatedString {
	type Element;
}

fn with_string<S>(string: S) -> S::Element where S: InterpolatedString {}

My main motivation for keeping so many things separate (e.g. the AST nodes) was to mimic to some extent what is being done in the CPython implementation.

I'm fine not sharing some AST nodes, e.g., if they're handled differently by most visitors anyway. I mainly wanted to make sure it was an intentional decision (maybe it was?).

[dhruvmanila]

Thank you again for working on this!

This seems pretty close to landing! I've a few comments but they're mostly around avoiding duplication so I'll leave it up to you.

[MichaReiser]

Awesome. Thank you for addressign all my feedback.

My only concern left is that we use different terminology for the same thing in different places. Sometimes we use FT, sometimes we use InterpolatedString. We should make sure to only use one of the two.

[dylwil3]

Thank you so much @MichaReiser and @dhruvmanila !!

[AlexWaygood]

🥳🥳 Congrats @dylwil3!!