Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Hygiene issues when reparsing token trees #16987

Closed
uorbe001 opened this issue Sep 4, 2014 · 1 comment · Fixed by #23085
Closed

Hygiene issues when reparsing token trees #16987

uorbe001 opened this issue Sep 4, 2014 · 1 comment · Fixed by #23085
Labels
A-syntaxext Area: Syntax extensions

Comments

@uorbe001
Copy link

uorbe001 commented Sep 4, 2014

I get errors when I call to_tokens on a parsed block and parse that again using procedural macros. It seems to be related to hygiene being lost when you do that, but I might be wrong, here's a gist that shows the problem: https://gist.github.com/uorbe001/1118f22c378d2e382d29

If I try to compile that gist, I get the following error on the assert! line:

<quote expansion>:4:50: 4:53 error: unresolved name `one`.
<quote expansion>:4                          name_66,ctxt_2!(1i == one);

I thought this was related to #8063, but apparently this is a separate issue.
@uorbe001
Copy link
Author

I forgot to mention, this seems to have been introduced with #16477

@huonw huonw added the A-syntaxext Area: Syntax extensions label Sep 10, 2014
goffrie added a commit to goffrie/rust that referenced this issue Mar 5, 2015
@goffrie
Interpolate AST nodes in quasiquote.
239775c 
This changes the `ToTokens` implementations for expressions, statements,
etc. with almost-trivial ones that produce `Interpolated(*Nt(...))`
pseudo-tokens. In this way, quasiquote now works the same way as macros
do: already-parsed AST fragments are used as-is, not reparsed.

The `ToSource` trait is removed. Quasiquote no longer involves
pretty-printing at all, which removes the need for the
`encode_with_hygiene` hack. All associated machinery is removed.

Two new `Nonterminal`s are added: NtArm and NtMethod, which the parser
now interpolates. These are just for quasiquote. They aren't used by
macros (although they could be in the future).

`ToTokens` is no longer implemented for `Arg` (although this could be
added again) and `Generics` (which I don't think makes sense).

This breaks any compiler extensions that relied on the ability of
`ToTokens` to turn AST fragments back into inspectable token trees. For
this reason, this closes rust-lang#16987.

As such, this is a [breaking-change].

Fixes rust-lang#16472.
Fixes rust-lang#15962.
Fixes rust-lang#17397.
Fixes rust-lang#16617.
@goffrie goffrie mentioned this issue Mar 5, 2015
Merged
goffrie added a commit to goffrie/rust that referenced this issue Mar 10, 2015
@goffrie
Interpolate AST nodes in quasiquote.
ce0a295 
This changes the `ToTokens` implementations for expressions, statements,
etc. with almost-trivial ones that produce `Interpolated(*Nt(...))`
pseudo-tokens. In this way, quasiquote now works the same way as macros
do: already-parsed AST fragments are used as-is, not reparsed.

The `ToSource` trait is removed. Quasiquote no longer involves
pretty-printing at all, which removes the need for the
`encode_with_hygiene` hack. All associated machinery is removed.

Two new `Nonterminal`s are added: NtArm and NtMethod, which the parser
now interpolates. These are just for quasiquote. They aren't used by
macros (although they could be in the future).

`ToTokens` is no longer implemented for `Arg` (although this could be
added again) and `Generics` (which I don't think makes sense).

This breaks any compiler extensions that relied on the ability of
`ToTokens` to turn AST fragments back into inspectable token trees. For
this reason, this closes rust-lang#16987.

As such, this is a [breaking-change].

Fixes rust-lang#16472.
Fixes rust-lang#15962.
Fixes rust-lang#17397.
Fixes rust-lang#16617.
goffrie added a commit to goffrie/rust that referenced this issue Mar 13, 2015
@goffrie
Interpolate AST nodes in quasiquote.
5b8312d 
This changes the `ToTokens` implementations for expressions, statements,
etc. with almost-trivial ones that produce `Interpolated(*Nt(...))`
pseudo-tokens. In this way, quasiquote now works the same way as macros
do: already-parsed AST fragments are used as-is, not reparsed.

The `ToSource` trait is removed. Quasiquote no longer involves
pretty-printing at all, which removes the need for the
`encode_with_hygiene` hack. All associated machinery is removed.

Two new `Nonterminal`s are added: NtArm and NtMethod, which the parser
now interpolates. These are just for quasiquote. They aren't used by
macros (although they could be in the future).

`ToTokens` is no longer implemented for `Arg` (although this could be
added again) and `Generics` (which I don't think makes sense).

This breaks any compiler extensions that relied on the ability of
`ToTokens` to turn AST fragments back into inspectable token trees. For
this reason, this closes rust-lang#16987.

As such, this is a [breaking-change].

Fixes rust-lang#16472.
Fixes rust-lang#15962.
Fixes rust-lang#17397.
Fixes rust-lang#16617.
goffrie added a commit to goffrie/rust that referenced this issue Mar 16, 2015
@goffrie
Interpolate AST nodes in quasiquote.
2a7c548 
This changes the `ToTokens` implementations for expressions, statements,
etc. with almost-trivial ones that produce `Interpolated(*Nt(...))`
pseudo-tokens. In this way, quasiquote now works the same way as macros
do: already-parsed AST fragments are used as-is, not reparsed.

The `ToSource` trait is removed. Quasiquote no longer involves
pretty-printing at all, which removes the need for the
`encode_with_hygiene` hack. All associated machinery is removed.

A new `Nonterminal` is added, NtArm, which the parser now interpolates.
This is just for quasiquote, not macros (although it could be in the
future).

`ToTokens` is no longer implemented for `Arg` (although this could be
added again) and `Generics` (which I don't think makes sense).

This breaks any compiler extensions that relied on the ability of
`ToTokens` to turn AST fragments back into inspectable token trees. For
this reason, this closes rust-lang#16987.

As such, this is a [breaking-change].

Fixes rust-lang#16472.
Fixes rust-lang#15962.
Fixes rust-lang#17397.
Fixes rust-lang#16617.
goffrie added a commit to goffrie/rust that referenced this issue Apr 9, 2015
@goffrie
Interpolate AST nodes in quasiquote.
ec6db72 
This changes the `ToTokens` implementations for expressions, statements,
etc. with almost-trivial ones that produce `Interpolated(*Nt(...))`
pseudo-tokens. In this way, quasiquote now works the same way as macros
do: already-parsed AST fragments are used as-is, not reparsed.

The `ToSource` trait is removed. Quasiquote no longer involves
pretty-printing at all, which removes the need for the
`encode_with_hygiene` hack. All associated machinery is removed.

A new `Nonterminal` is added, NtArm, which the parser now interpolates.
This is just for quasiquote, not macros (although it could be in the
future).

`ToTokens` is no longer implemented for `Arg` (although this could be
added again) and `Generics` (which I don't think makes sense).

This breaks any compiler extensions that relied on the ability of
`ToTokens` to turn AST fragments back into inspectable token trees. For
this reason, this closes rust-lang#16987.

As such, this is a [breaking-change].

Fixes rust-lang#16472.
Fixes rust-lang#15962.
Fixes rust-lang#17397.
Fixes rust-lang#16617.
goffrie added a commit to goffrie/rust that referenced this issue Apr 19, 2015
@goffrie
Interpolate AST nodes in quasiquote.
ec1b84c 
This changes the `ToTokens` implementations for expressions, statements,
etc. with almost-trivial ones that produce `Interpolated(*Nt(...))`
pseudo-tokens. In this way, quasiquote now works the same way as macros
do: already-parsed AST fragments are used as-is, not reparsed.

The `ToSource` trait is removed. Quasiquote no longer involves
pretty-printing at all, which removes the need for the
`encode_with_hygiene` hack. All associated machinery is removed.

A new `Nonterminal` is added, NtArm, which the parser now interpolates.
This is just for quasiquote, not macros (although it could be in the
future).

`ToTokens` is no longer implemented for `Arg` (although this could be
added again) and `Generics` (which I don't think makes sense).

This breaks any compiler extensions that relied on the ability of
`ToTokens` to turn AST fragments back into inspectable token trees. For
this reason, this closes rust-lang#16987.

As such, this is a [breaking-change].

Fixes rust-lang#16472.
Fixes rust-lang#15962.
Fixes rust-lang#17397.
Fixes rust-lang#16617.
goffrie added a commit to goffrie/rust that referenced this issue Apr 23, 2015
@goffrie
Interpolate AST nodes in quasiquote.
d653477 
This changes the `ToTokens` implementations for expressions, statements,
etc. with almost-trivial ones that produce `Interpolated(*Nt(...))`
pseudo-tokens. In this way, quasiquote now works the same way as macros
do: already-parsed AST fragments are used as-is, not reparsed.

The `ToSource` trait is removed. Quasiquote no longer involves
pretty-printing at all, which removes the need for the
`encode_with_hygiene` hack. All associated machinery is removed.

A new `Nonterminal` is added, NtArm, which the parser now interpolates.
This is just for quasiquote, not macros (although it could be in the
future).

`ToTokens` is no longer implemented for `Arg` (although this could be
added again) and `Generics` (which I don't think makes sense).

This breaks any compiler extensions that relied on the ability of
`ToTokens` to turn AST fragments back into inspectable token trees. For
this reason, this closes rust-lang#16987.

As such, this is a [breaking-change].

Fixes rust-lang#16472.
Fixes rust-lang#15962.
Fixes rust-lang#17397.
Fixes rust-lang#16617.
goffrie added a commit to goffrie/rust that referenced this issue Apr 23, 2015
@goffrie
Interpolate AST nodes in quasiquote.
7691bce 
This changes the `ToTokens` implementations for expressions, statements,
etc. with almost-trivial ones that produce `Interpolated(*Nt(...))`
pseudo-tokens. In this way, quasiquote now works the same way as macros
do: already-parsed AST fragments are used as-is, not reparsed.

The `ToSource` trait is removed. Quasiquote no longer involves
pretty-printing at all, which removes the need for the
`encode_with_hygiene` hack. All associated machinery is removed.

A new `Nonterminal` is added, NtArm, which the parser now interpolates.
This is just for quasiquote, not macros (although it could be in the
future).

`ToTokens` is no longer implemented for `Arg` (although this could be
added again) and `Generics` (which I don't think makes sense).

This breaks any compiler extensions that relied on the ability of
`ToTokens` to turn AST fragments back into inspectable token trees. For
this reason, this closes rust-lang#16987.

As such, this is a [breaking-change].

Fixes rust-lang#16472.
Fixes rust-lang#15962.
Fixes rust-lang#17397.
Fixes rust-lang#16617.
goffrie added a commit to goffrie/rust that referenced this issue Apr 24, 2015
@goffrie
Interpolate AST nodes in quasiquote.
dac0f82 
This changes the `ToTokens` implementations for expressions, statements,
etc. with almost-trivial ones that produce `Interpolated(*Nt(...))`
pseudo-tokens. In this way, quasiquote now works the same way as macros
do: already-parsed AST fragments are used as-is, not reparsed.

The `ToSource` trait is removed. Quasiquote no longer involves
pretty-printing at all, which removes the need for the
`encode_with_hygiene` hack. All associated machinery is removed.

A new `Nonterminal` is added, NtArm, which the parser now interpolates.
This is just for quasiquote, not macros (although it could be in the
future).

`ToTokens` is no longer implemented for `Arg` (although this could be
added again) and `Generics` (which I don't think makes sense).

This breaks any compiler extensions that relied on the ability of
`ToTokens` to turn AST fragments back into inspectable token trees. For
this reason, this closes rust-lang#16987.

As such, this is a [breaking-change].

Fixes rust-lang#16472.
Fixes rust-lang#15962.
Fixes rust-lang#17397.
Fixes rust-lang#16617.
goffrie added a commit to goffrie/rust that referenced this issue Apr 26, 2015
@goffrie
Interpolate AST nodes in quasiquote.
2d9831d 
This changes the `ToTokens` implementations for expressions, statements,
etc. with almost-trivial ones that produce `Interpolated(*Nt(...))`
pseudo-tokens. In this way, quasiquote now works the same way as macros
do: already-parsed AST fragments are used as-is, not reparsed.

The `ToSource` trait is removed. Quasiquote no longer involves
pretty-printing at all, which removes the need for the
`encode_with_hygiene` hack. All associated machinery is removed.

A new `Nonterminal` is added, NtArm, which the parser now interpolates.
This is just for quasiquote, not macros (although it could be in the
future).

`ToTokens` is no longer implemented for `Arg` (although this could be
added again) and `Generics` (which I don't think makes sense).

This breaks any compiler extensions that relied on the ability of
`ToTokens` to turn AST fragments back into inspectable token trees. For
this reason, this closes rust-lang#16987.

As such, this is a [breaking-change].

Fixes rust-lang#16472.
Fixes rust-lang#15962.
Fixes rust-lang#17397.
Fixes rust-lang#16617.
bors added a commit that referenced this issue Apr 26, 2015
@bors
Auto merge of #23085 - goffrie:interpolating-quote, r=huonw
6365080 
This changes the `ToTokens` implementations for expressions, statements, etc. with almost-trivial ones that produce `Interpolated(*Nt(...))` pseudo-tokens. In this way, quasiquote now works the same way as macros do: already-parsed AST fragments are used as-is, not reparsed.

The `ToSource` trait is removed. Quasiquote no longer involves pretty-printing at all, which removes the need for the `encode_with_hygiene` hack. All associated machinery is removed.

New `Nonterminal`s are added: NtArm, NtImplItem, and NtTraitItem. These are just for quasiquote, not macros.

`ToTokens` is no longer implemented for `Arg` (although this could be added again) and `Generics` (which I don't think makes sense).

This breaks any compiler extensions that relied on the ability of `ToTokens` to turn AST fragments back into inspectable token trees. For this reason, this closes #16987.

As such, this is a [breaking-change].

Fixes #16472.
Fixes #15962.
Fixes #17397.
Fixes #16617.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-syntaxext Area: Syntax extensions
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Interpolate AST nodes in quasiquote. goffrie/rust
2 participants
@uorbe001 @huonw