Adds a `ProcMacro` form of syntax extension #36154

nrc · 2016-08-31T00:04:45Z

This commit adds syntax extension forms matching the types for procedural macros 2.0 (RFC #1566), these still require the usual syntax extension boiler plate, but this is a first step towards proper implementation and should be useful for macros 1.1 stuff too.

Supports both attribute-like and function-like macros.

Note that RFC #1566 has not been accepted yet, but I think there is consensus that we want to head in vaguely that direction and so this PR will be useful in any case. It is also fairly easy to undo and does not break any existing programs.

This is related to #35957 in that I hope it can be used in the implementation of macros 1.1, however, there is no direct overlap and is more of a complement than a competing proposal. There is still a fair bit of work to do before the two can be combined.

r? @jseyfried

cc @alexcrichton, @cgswords, @eddyb, @aturon

jseyfried · 2016-08-31T00:21:02Z

src/libsyntax/ext/base.rs

+
+struct TokResult<'a> {
+    parser: Parser<'a>,
+    span: Span,


This is the call site span of the macro invocation that expanded into the TokResult, right?

bors · 2016-09-03T10:21:12Z

☔ The latest upstream changes (presumably #35957) made this pull request unmergeable. Please resolve the merge conflicts.

This commit adds syntax extension forms matching the types for procedural macros 2.0 (RFC rust-lang#1566), these still require the usual syntax extension boiler plate, but this is a first step towards proper implementation and should be useful for macros 1.1 stuff too. Supports both attribute-like and function-like macros.

nrc · 2016-09-21T22:38:54Z

@jseyfried r? I think I addressed all your earlier comments

jseyfried · 2016-09-21T23:19:28Z

src/libsyntax/ext/expand.rs

+                let parser = self.cx.new_parser_from_tts(&tok_result.to_tts());
+                let result = Box::new(TokResult { parser: parser, span: attr.span });
+
+                let items: Vec<Annotatable> = match item {


This match could be the trailing expression statement if each arm had type Expansion.

jseyfried · 2016-09-21T23:19:46Z

src/libsyntax/ext/expand.rs

+                        match result.make_items() {
+                            Some(items) => items.into_iter()
+                                                .map(|i| Annotatable::Item(i))
+                                                .collect(),


e.g. this could be Some(items) => Expansion::Item(items),

jseyfried · 2016-09-21T23:31:49Z

src/libsyntax/parse/lexer/mod.rs

+            Some(t) => self.pos > t,
+            None => false,
+        }
+


nit: extra newline

jseyfried · 2016-09-21T23:33:26Z

src/libsyntax/parse/lexer/mod.rs

+    pub fn from_span<'b>(span_diagnostic: &'b Handler,
+                         span: Span,
+                         codemap: &CodeMap)
+                         -> StringReader<'b> {


I believe this method is unused.

jseyfried · 2016-09-21T23:34:07Z

src/libsyntax/ext/base.rs

@@ -60,6 +62,14 @@ impl HasAttrs for Annotatable {
 }

 impl Annotatable {
+    pub fn span(&self) -> Span {


I believe this is unused.

jseyfried · 2016-09-21T23:40:49Z

src/librustc_plugin/registry.rs

@@ -111,6 +111,8 @@ impl<'a> Registry<'a> {
            }
            MultiDecorator(ext) => MultiDecorator(ext),
            MultiModifier(ext) => MultiModifier(ext),
+            SyntaxExtension::ProcMacro(ext) => SyntaxExtension::ProcMacro(ext),
+            SyntaxExtension::AttrProcMacro(ext) => SyntaxExtension::AttrProcMacro(ext),


These four lines could just be _ => extension, (unless the explicitness is intentional, that is).

jseyfried · 2016-09-21T23:50:17Z

src/libsyntax/ext/base.rs

+                    let new_items = changer.fold_impl_item(item);
+                    items.push_all(new_items);
+
+                    return Some(items);


Since this arm always returns, macros will only be able to expand into one impl item (same with trait items) -- not sure if this is intentional.

jseyfried · 2016-09-22T00:04:36Z

src/libsyntax/ext/base.rs

+                    items.push_all(new_items);
+                }
+                Ok(None) => {
+                    return Some(items);


parse_item() will return None if the next token does not begin an item. For example, if a procedural macro returned + in an item position, it would silently expand into nothing here.

ext::tt::macro_rules::ParserAnyMacro solves this problem with ensure_complete_parse.
In general, I think it would be a good idea to follow the semantics of ParserAnyMacro more closely or even merge ParserAnyMacro and TokResult (moving the ChangeSpan fold into expand.rs), at least until we clean up the semantics of the parser.parse_* methods used here.

jseyfried · 2016-09-22T00:07:03Z

src/libsyntax/ext/base.rs

+
+    fn make_expr(mut self: Box<Self>) -> Option<P<ast::Expr>> {
+        match self.parser.parse_expr() {
+            Ok(e) => Some(e),


Something like ensure_complete_parse is also needed here so that valid[expression] } followed by arbitrary tokens is not allowed (c.f. my comment about ParserAnyMacro).

jseyfried · 2016-09-22T00:12:39Z

src/libsyntax/ext/expand.rs

+                                                  TokenStream::from_tts(marked_tts));
+                let parser = self.cx.new_parser_from_tts(&tok_result.to_tts());
+                let result = Box::new(TokResult { parser: parser, span: span });
+                kind.make_from(result)


It might be a good idea to change the spans here instead of in each method of TokResult -- e.g. kind.make_from(result).fold_with(&mut ChangeSpan { span: span }).

nrc · 2016-09-22T08:13:55Z

@jseyfried changes made

jseyfried · 2016-09-22T08:42:30Z

src/libsyntax/ext/expand.rs

+                let result = Box::new(TokResult { parser: parser, span: attr.span });
+
+                match item {
+                    Annotatable::Item(_) => {


One last simplification that I didn't notice earlier -- I believe this whole match can be replaced with:

kind.make_from(result).unwrap_or_else(|| { let msg = format!("macro could not be expanded into {} position", kind.name()); self.cx.span_err(&msg); kind.dummy(attr.span) })

ah nice, I was hoping there was a way to get rid of this whole match

jseyfried · 2016-09-22T08:48:14Z

@nrc r=me with that last simplification.

nrc · 2016-09-22T19:20:21Z

@bors: r=@jseyfried

bors · 2016-09-22T19:20:22Z

📌 Commit 3863834 has been approved by @jseyfried

bors · 2016-09-22T23:33:42Z

⌛ Testing commit 3863834 with merge 3a5d975...

@jseyfried

Adds a `ProcMacro` form of syntax extension This commit adds syntax extension forms matching the types for procedural macros 2.0 (RFC #1566), these still require the usual syntax extension boiler plate, but this is a first step towards proper implementation and should be useful for macros 1.1 stuff too. Supports both attribute-like and function-like macros. Note that RFC #1566 has not been accepted yet, but I think there is consensus that we want to head in vaguely that direction and so this PR will be useful in any case. It is also fairly easy to undo and does not break any existing programs. This is related to #35957 in that I hope it can be used in the implementation of macros 1.1, however, there is no direct overlap and is more of a complement than a competing proposal. There is still a fair bit of work to do before the two can be combined. r? @jseyfried cc @alexcrichton, @cgswords, @eddyb, @aturon

bors · 2016-09-23T02:54:54Z

jseyfried reviewed Aug 31, 2016
View reviewed changes

nrc force-pushed the proc-macro-init branch from 56405da to 973eae2 Compare September 21, 2016 22:38

jseyfried reviewed Sep 22, 2016

View reviewed changes

nrc force-pushed the proc-macro-init branch from 973eae2 to d956d7e Compare September 22, 2016 08:13

jseyfried reviewed Sep 22, 2016

View reviewed changes

reviewer comments and rebasing

3863834

nrc force-pushed the proc-macro-init branch from d956d7e to 3863834 Compare September 22, 2016 19:19

bors merged commit 3863834 into rust-lang:master Sep 23, 2016

This was referenced Sep 23, 2016

Load macros from #[macro_use] extern crates in resolve #36616

Merged

Clarify the workings of StringReader in the lexer #36470

Closed

jseyfried mentioned this pull request Apr 10, 2017

proc_macro: implement TokenTree, TokenKind, hygienic quote!, and other API #40939

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds a `ProcMacro` form of syntax extension #36154

Adds a `ProcMacro` form of syntax extension #36154

nrc commented Aug 31, 2016

jseyfried Aug 31, 2016

nrc Sep 1, 2016

bors commented Sep 3, 2016

nrc commented Sep 21, 2016

jseyfried Sep 21, 2016

jseyfried Sep 21, 2016 •

edited

Loading

jseyfried Sep 21, 2016

jseyfried Sep 21, 2016 •

edited

Loading

jseyfried Sep 21, 2016

jseyfried Sep 21, 2016

jseyfried Sep 21, 2016

jseyfried Sep 22, 2016 •

edited

Loading

jseyfried Sep 22, 2016

jseyfried Sep 22, 2016 •

edited

Loading

nrc commented Sep 22, 2016

jseyfried Sep 22, 2016 •

edited

Loading

nrc Sep 22, 2016

jseyfried commented Sep 22, 2016

nrc commented Sep 22, 2016

bors commented Sep 22, 2016

bors commented Sep 22, 2016

bors commented Sep 23, 2016

Adds a ProcMacro form of syntax extension #36154

Adds a ProcMacro form of syntax extension #36154

Conversation

nrc commented Aug 31, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bors commented Sep 3, 2016

nrc commented Sep 21, 2016

Choose a reason for hiding this comment

jseyfried Sep 21, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jseyfried Sep 21, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jseyfried Sep 22, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jseyfried Sep 22, 2016 • edited Loading

Choose a reason for hiding this comment

nrc commented Sep 22, 2016

jseyfried Sep 22, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jseyfried commented Sep 22, 2016

nrc commented Sep 22, 2016

bors commented Sep 22, 2016

bors commented Sep 22, 2016

bors commented Sep 23, 2016

Adds a `ProcMacro` form of syntax extension #36154

Adds a `ProcMacro` form of syntax extension #36154

jseyfried Sep 21, 2016 •

edited

Loading

jseyfried Sep 21, 2016 •

edited

Loading

jseyfried Sep 22, 2016 •

edited

Loading

jseyfried Sep 22, 2016 •

edited

Loading

jseyfried Sep 22, 2016 •

edited

Loading