Error using quote_tokens: expected identifier, found keyword self

[hannobraun]

Since upgrading to the latest nightly, I'm getting a compiler error for the following piece of code:

let field_set = quote_tokens!(context,
    $var_name: self.$collection.pop(id),
);

This code is available in the Rustecs repository: https://github.com/hannobraun/rustecs/blob/cd844f90b3073626e4252ff46781bed6ab0b337e/rustecs_macros/src/generate/intermediate.rs#L73

The error:

/home/hanno/Projects/rustecs/rustecs_macros/src/generate/intermediate.rs:73:15: 73:19 error: expected identifier, found keyword `self`
/home/hanno/Projects/rustecs/rustecs_macros/src/generate/intermediate.rs:73             $var_name: self.$collection.pop(id),
                                                                                                   ^~~~
error: aborting due to previous error
Could not compile `rustecs_macros`.

I was able to create this reduced example:

// lib.rs
#![feature(quote)]

fn quote() {
    quote_tokens!(context,
        $var_name: self
    );
}

While there are many reasons this code should not compile, it fails with the same error message. If I

• change self to something else or
• remove the : or
• Remove the $ in $var_name

the error goes away and the compile fails with a different error (as expected).

I can reproduce the issue with the latest nightly. I'm not sure about the exact version, since rustc --version only produces rustc 0.13.0-nightly. I don't know which version broke it, but it must have been recent. The last successful Travis CI build was using 0.13.0-nightly (45cbdec41 2014-11-07 00:02:18 +0000). See https://travis-ci.org/hannobraun/rustecs/builds/40199720

More info about my system:

$ rustc --version=verbose
rustc 0.13.0-nightly
binary: rustc
commit-hash: unknown
commit-date: unknown
host: x86_64-unknown-linux-gnu
release: 0.13.0-nightly

[yuriks]

I also hit what I believe to be the same issue. The following code: (http://is.gd/Z0B3a3)

#![feature(macro_rules)]

struct Foo { x: u32, y: u32 }

macro_rules! make_copy(
    ($Self:ident { $($field:ident),+ }) => (
        fn copy(&self) -> $Self {
            //let me = self;
            $Self {
                $($field: self.$field),+
            }
        }
    )
)

impl Foo {
    make_copy!(Foo {x, y})
}

fn main() {
    let x = Foo { x: 0, y: 2 };
    let _y = x.copy();
}

Gives this error:

test.rs:10:27: 10:31 error: expected identifier, found keyword `self`
test.rs:10                 $($field: self.$field),+

Enabling the commented out line and changing self to me in the expansion makes it build correctly. This also broke after upgrading to the current nightly from one from about two days ago.

[hannobraun]

Please reopen this issue. The original error still occurs on current master.

Error:

/home/hanno/Projects/rustecs/rustecs_macros/src/generate/intermediate.rs:73:15: 73:19 error: expected identifier, found keyword `self`
/home/hanno/Projects/rustecs/rustecs_macros/src/generate/intermediate.rs:73             $var_name: self.$collection.pop(id),
                                                                                                   ^~~~
error: aborting due to previous error

Rust version:

s$ rustc --version=verbose
rustc 0.13.0-dev (6f7081fad 2014-11-14 05:37:17 +0000)
binary: rustc
commit-hash: 6f7081fad5889c7b41460103fc8abc98f3285c60
commit-date: 2014-11-14 05:37:17 +0000
host: x86_64-unknown-linux-gnu
release: 0.13.0-dev

@yuriks Is your code working now?

[yuriks]

@hannobraun The test case I gave now works, but my original program, when changed back to not use the workaround, still doesn't. I'm going to have to create a new testcase. rustc 0.13.0-nightly (5745e4195 2014-11-12 22:57:16 +0000)

[jakerr]

A complete working reproduction of this regression is below:

#![feature(quote)]

extern crate syntax;

use syntax::print::pprust;
use syntax::codemap::DUMMY_SP;
use syntax::ast;
use syntax::ext::base::ExtCtxt;
use syntax::parse::token;

#[allow(unused_imports)]
fn main() {
    let var_name = ast::Ident::new(
            token::intern("my_var")
    );
    let collection = ast::Ident::new(
            token::intern("my_collection")
    );
    with_fake_extctxt(|cx| {
        let field_set = quote_tokens!(cx,
            $var_name: self.$collection.pop(id),
        );
        println!("{}", pprust::tts_to_string(field_set.deref()));
    });
}

fn with_fake_extctxt<T>(f: |&syntax::ext::base::ExtCtxt| -> T) -> T {
    let ps = syntax::parse::new_parse_sess();

    let mut cx = syntax::ext::base::ExtCtxt::new(&ps, Vec::new(), syntax::ext::expand::ExpansionConfig {
        deriving_hash_type_parameter: false,
        crate_name: from_str("test").unwrap(),
        enable_quotes: true,
        recursion_limit: 100
    });

    cx.bt_push(syntax::codemap::ExpnInfo{
        call_site: DUMMY_SP,
        callee: syntax::codemap::NameAndSpan {
            name: "test".to_string(),
            format: syntax::codemap::MacroBang,
            span: None,
        }
    });

    f(&cx)
}

This currently works on http://play.rust-lang.org/ probably because the rustc version there is behind. It reproduces this bug in rustc 0.13.0-nightly (336349c93 2014-11-17 20:37:19 +0000)

[hannobraun]

Thanks, @jakerr, for taking care of this. I meant to re-check and update my reduced example but didn't get to it so far. Things have been a bit crazy those last few weeks

[jakerr]

@hannobraun no worries!

I got as far as finding what is wrong, but not as far as figuring out how to fix it.

In parser.rs L2664-L2677

...
                } else {
                    // A nonterminal that matches or not
                    let namep = match p.token { token::Ident(_, p) => p, _ => token::Plain };
                    let name = p.parse_ident();
                    if p.token == token::Colon && p.look_ahead(1, |t| t.is_ident()) {
                        p.bump();
                        let kindp = match p.token { token::Ident(_, p) => p, _ => token::Plain };
                        let nt_kind = p.parse_ident();
                        let m = TtToken(sp, MatchNt(name, nt_kind, namep, kindp));
                        m
                    } else {
                        TtToken(sp, SubstNt(name, namep))
                    }
                }

There's code that looks for $ : ident and then decides that it is a MatchNt while this is ambiguous because it may be a SubstNt : ident. I thought maybe the parser should notice that it is currently parsing an invocation of macro_rules and only look for MatchNt in that case but not sure if that's the cleanest solution.

I think I'll leave it to the experts. @pczarn do you mind taking a look?

[pczarn]

The error appears when this code is looking ahead after a colon and finds an ident, but that ident is also a keyword. A solution could use check_strict_keywords and check_reserved_keywords.

[jakerr]

True, but if you only eliminate the error for keywords you will get errors when trying to treat the ident after the colon as a matcher kind later right?

For example

quote_tokens!($foo: bar.baz)

Would tokenize to MatchNt with name foo and kind bar which would eventually lead to a parse error in macro_parser.rs

p.fatal(format!("unsupported builtin nonterminal parser: {}",  name).as_slice())

Isn't it?

Sorry for lack of good links, I'm mobile.

[pczarn]

That'd be good intuition, but the code in macro_parser.rs only applies to the macro_rules extension, not syntax extensions such as quote_tokens! from quote.rs. Sorry, I've completely missed the point of this issue last time. True, it will tokenize to MatchNt, and it would have had surprising behavior in quote_tokens! which I've fixed.

It seems I'll have to add quote_matcher! without interpolation in quoted MatchNt.