Skip to content
This repository has been archived by the owner on Jun 3, 2021. It is now read-only.

Separate macro replacement from macro parsing #440

Closed
wants to merge 5 commits into from
Closed

Conversation

jyn514
Copy link
Owner

@jyn514 jyn514 commented May 20, 2020

This allows you to replace any iterator of Token without having to have the original &str.

The core idea is to separate the preprocessor into 3 separate, nested iterators:

  1. The innermost iterator (FileProcessor) deals with multiple files/lexers. If one included file runs out of tokens, it seamlessly goes on to the next one.
  2. The middle iterator (MacroReplacer) performs macro replacement.
  3. The outermost iterator (PreProcessor) deals with preprocessing directives. Additionally, because of the weird way I lumped the lexer and preprocessor together, it recognizes keywords and turns Token::Id into Token::Keyword as necessary.

The PreProcessor sometimes does not want to replace its tokens (e.g. for #if defined(a)). In this case, it reaches through the replacer into the FileProcessor to drag out those tokens. Because of this, the MacroReplacer cannot have any idea of pending tokens, or its pending will conflict with the PreProcessor's pending.

I want to support inputs other than FileProcessor (this is the whole point of the PR after all), but at the same time I need to be able to peek ahead in the input to see whether a token defined as a function macro is followed by a (. To support both these use cases, I add a new trait called Peekable; the intent is that users of rcc can use arbitrary iterators over Token by calling iter.peekable().

I'm not super happy with how tightly coupled the PreProcessor is to the FileProcessor, but that's a project for another day.

This also implements Debug for InternedStr to actually be useful, since I was looking at the debug output a lot.

Needed for rust-lang/rust-bindgen#1782. Closes #436.

r? @pythondude325

@jyn514
Copy link
Owner Author

jyn514 commented May 20, 2020

I realized as I wrote up the PR that most of the PR description should be doc-comments, adding those now.

- Return Vec of replaced tokens
- Add Peekable trait to allow using any `std::iter::Peekable` iterator
- Split up file handling and preprocessing
- Make `impl Debug for InternedStr` actually useful

I was staring at a lot of debugged identifiers while working on this;
I figured they should at least be useful.

- Only perform replacement _after_ processing `defined`
- Document edge cases
- Export `replace`
- Add doc-comments about architecture
Copy link
Collaborator

@hdamron17 hdamron17 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me and works on Mac.

@jyn514
Copy link
Owner Author

jyn514 commented May 20, 2020

I don't expect much of the frontend work to be target-dependent (other than sizeof(int) and that sort of thing). It's changes to the backend that are much more likely to break on different platforms, especially linking.

@jyn514
Copy link
Owner Author

jyn514 commented May 20, 2020

Closing in favor of #441

@jyn514 jyn514 closed this May 20, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Separate macro replacement and macro parsing
2 participants