This repository has been archived by the owner on Jun 3, 2021. It is now read-only.
Separate macro replacement from macro parsing #440
Closed
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This allows you to replace any iterator of
Token
without having to have the original&str
.The core idea is to separate the preprocessor into 3 separate, nested iterators:
FileProcessor
) deals with multiple files/lexers. If one included file runs out of tokens, it seamlessly goes on to the next one.MacroReplacer
) performs macro replacement.PreProcessor
) deals with preprocessing directives. Additionally, because of the weird way I lumped the lexer and preprocessor together, it recognizes keywords and turnsToken::Id
intoToken::Keyword
as necessary.The
PreProcessor
sometimes does not want to replace its tokens (e.g. for#if defined(a)
). In this case, it reaches through the replacer into theFileProcessor
to drag out those tokens. Because of this, theMacroReplacer
cannot have any idea of pending tokens, or itspending
will conflict with thePreProcessor
'spending
.I want to support inputs other than
FileProcessor
(this is the whole point of the PR after all), but at the same time I need to be able to peek ahead in the input to see whether a token defined as a function macro is followed by a(
. To support both these use cases, I add a new trait calledPeekable
; the intent is that users ofrcc
can use arbitrary iterators overToken
by callingiter.peekable()
.I'm not super happy with how tightly coupled the
PreProcessor
is to theFileProcessor
, but that's a project for another day.This also implements
Debug
forInternedStr
to actually be useful, since I was looking at the debug output a lot.Needed for rust-lang/rust-bindgen#1782. Closes #436.
r? @pythondude325