-
Notifications
You must be signed in to change notification settings - Fork 20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: Don't consume the dot in decimal literals if not followed by digit since 0.5.0 #891
Merged
Merged
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
7426418
Add more tests with identifier following decimal expression
Xanewok 788f76d
fix: After 0.5.0 we can't consume the dot if not followed by digit
Xanewok 2ae750d
Add a changeset file
Xanewok 3c9f225
refactor: Use disjoint token definitions for DecimalLiteral
Xanewok 84fd161
Update crates/solidity/inputs/language/src/definition.rs
Xanewok File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| --- | ||
| "@nomicfoundation/slang": patch | ||
| --- | ||
|
|
||
| Fix parsing `<NUMBER>.member` member access expression |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
16 changes: 16 additions & 0 deletions
16
crates/solidity/outputs/cargo/slang_solidity/src/generated/language.rs
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
26 changes: 26 additions & 0 deletions
26
crates/solidity/outputs/cargo/tests/src/cst_output/generated/decimal_number_expression.rs
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
15 changes: 15 additions & 0 deletions
15
crates/solidity/outputs/cargo/tests/src/cst_output/generated/expression.rs
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
2 changes: 1 addition & 1 deletion
2
crates/solidity/outputs/spec/generated/public/05-expressions/04-numbers.md
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
19 changes: 19 additions & 0 deletions
19
.../cst_output/DecimalNumberExpression/float_ident_after_period/generated/0.4.11-failure.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| # This file is generated automatically by infrastructure scripts. Please don't edit by hand. | ||
|
|
||
| Source: > | ||
| 1 │ 1.2.a │ 0..5 | ||
|
|
||
| Errors: # 1 total | ||
| - > | ||
| Error: Expected DaysKeyword or EtherKeyword or FinneyKeyword or HoursKeyword or MinutesKeyword or SecondsKeyword or SzaboKeyword or WeeksKeyword or WeiKeyword or YearsKeyword. | ||
| ╭─[crates/solidity/testing/snapshots/cst_output/DecimalNumberExpression/float_ident_after_period/input.sol:1:4] | ||
| │ | ||
| 1 │ 1.2.a | ||
| │ ─┬─ | ||
| │ ╰─── Error occurred here. | ||
| ───╯ | ||
|
|
||
| Tree: | ||
| - (DecimalNumberExpression): # "1.2.a\n" (0..6) | ||
| - (literal꞉ DecimalLiteral): "1.2" # (0..3) | ||
| - (SKIPPED): ".a\n" # (3..6) |
19 changes: 19 additions & 0 deletions
19
...s/cst_output/DecimalNumberExpression/float_ident_after_period/generated/0.5.0-failure.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| # This file is generated automatically by infrastructure scripts. Please don't edit by hand. | ||
|
|
||
| Source: > | ||
| 1 │ 1.2.a │ 0..5 | ||
|
|
||
| Errors: # 1 total | ||
| - > | ||
| Error: Expected DaysKeyword or EtherKeyword or FinneyKeyword or HoursKeyword or MinutesKeyword or SecondsKeyword or SzaboKeyword or WeeksKeyword or WeiKeyword. | ||
| ╭─[crates/solidity/testing/snapshots/cst_output/DecimalNumberExpression/float_ident_after_period/input.sol:1:4] | ||
| │ | ||
| 1 │ 1.2.a | ||
| │ ─┬─ | ||
| │ ╰─── Error occurred here. | ||
| ───╯ | ||
|
|
||
| Tree: | ||
| - (DecimalNumberExpression): # "1.2.a\n" (0..6) | ||
| - (literal꞉ DecimalLiteral): "1.2" # (0..3) | ||
| - (SKIPPED): ".a\n" # (3..6) |
19 changes: 19 additions & 0 deletions
19
.../cst_output/DecimalNumberExpression/float_ident_after_period/generated/0.6.11-failure.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| # This file is generated automatically by infrastructure scripts. Please don't edit by hand. | ||
|
|
||
| Source: > | ||
| 1 │ 1.2.a │ 0..5 | ||
|
|
||
| Errors: # 1 total | ||
| - > | ||
| Error: Expected DaysKeyword or EtherKeyword or FinneyKeyword or GweiKeyword or HoursKeyword or MinutesKeyword or SecondsKeyword or SzaboKeyword or WeeksKeyword or WeiKeyword. | ||
| ╭─[crates/solidity/testing/snapshots/cst_output/DecimalNumberExpression/float_ident_after_period/input.sol:1:4] | ||
| │ | ||
| 1 │ 1.2.a | ||
| │ ─┬─ | ||
| │ ╰─── Error occurred here. | ||
| ───╯ | ||
|
|
||
| Tree: | ||
| - (DecimalNumberExpression): # "1.2.a\n" (0..6) | ||
| - (literal꞉ DecimalLiteral): "1.2" # (0..3) | ||
| - (SKIPPED): ".a\n" # (3..6) |
19 changes: 19 additions & 0 deletions
19
...s/cst_output/DecimalNumberExpression/float_ident_after_period/generated/0.7.0-failure.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| # This file is generated automatically by infrastructure scripts. Please don't edit by hand. | ||
|
|
||
| Source: > | ||
| 1 │ 1.2.a │ 0..5 | ||
|
|
||
| Errors: # 1 total | ||
| - > | ||
| Error: Expected DaysKeyword or EtherKeyword or GweiKeyword or HoursKeyword or MinutesKeyword or SecondsKeyword or WeeksKeyword or WeiKeyword. | ||
| ╭─[crates/solidity/testing/snapshots/cst_output/DecimalNumberExpression/float_ident_after_period/input.sol:1:4] | ||
| │ | ||
| 1 │ 1.2.a | ||
| │ ─┬─ | ||
| │ ╰─── Error occurred here. | ||
| ───╯ | ||
|
|
||
| Tree: | ||
| - (DecimalNumberExpression): # "1.2.a\n" (0..6) | ||
| - (literal꞉ DecimalLiteral): "1.2" # (0..3) | ||
| - (SKIPPED): ".a\n" # (3..6) |
1 change: 1 addition & 0 deletions
1
...y/testing/snapshots/cst_output/DecimalNumberExpression/float_ident_after_period/input.sol
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| 1.2.a |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
19 changes: 19 additions & 0 deletions
19
...apshots/cst_output/DecimalNumberExpression/float_no_fraction/generated/0.6.11-failure.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| # This file is generated automatically by infrastructure scripts. Please don't edit by hand. | ||
|
|
||
| Source: > | ||
| 1 │ 1. │ 0..2 | ||
|
|
||
| Errors: # 1 total | ||
| - > | ||
| Error: Expected DaysKeyword or EtherKeyword or FinneyKeyword or GweiKeyword or HoursKeyword or MinutesKeyword or SecondsKeyword or SzaboKeyword or WeeksKeyword or WeiKeyword. | ||
| ╭─[crates/solidity/testing/snapshots/cst_output/DecimalNumberExpression/float_no_fraction/input.sol:1:2] | ||
| │ | ||
| 1 │ 1. | ||
| │ ┬ | ||
| │ ╰── Error occurred here. | ||
| ───╯ | ||
|
|
||
| Tree: | ||
| - (DecimalNumberExpression): # "1." (0..2) | ||
| - (literal꞉ DecimalLiteral): "1" # (0..1) | ||
| - (SKIPPED): "." # (1..2) |
19 changes: 19 additions & 0 deletions
19
...napshots/cst_output/DecimalNumberExpression/float_no_fraction/generated/0.7.0-failure.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| # This file is generated automatically by infrastructure scripts. Please don't edit by hand. | ||
|
|
||
| Source: > | ||
| 1 │ 1. │ 0..2 | ||
|
|
||
| Errors: # 1 total | ||
| - > | ||
| Error: Expected DaysKeyword or EtherKeyword or GweiKeyword or HoursKeyword or MinutesKeyword or SecondsKeyword or WeeksKeyword or WeiKeyword. | ||
| ╭─[crates/solidity/testing/snapshots/cst_output/DecimalNumberExpression/float_no_fraction/input.sol:1:2] | ||
| │ | ||
| 1 │ 1. | ||
| │ ┬ | ||
| │ ╰── Error occurred here. | ||
| ───╯ | ||
|
|
||
| Tree: | ||
| - (DecimalNumberExpression): # "1." (0..2) | ||
| - (literal꞉ DecimalLiteral): "1" # (0..1) | ||
| - (SKIPPED): "." # (1..2) |
17 changes: 17 additions & 0 deletions
17
...st_output/DecimalNumberExpression/integer_ident_after_period/generated/0.4.11-failure.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| # This file is generated automatically by infrastructure scripts. Please don't edit by hand. | ||
|
|
||
| Source: > | ||
| 1 │ 1.a │ 0..3 | ||
|
|
||
| Errors: # 1 total | ||
| - > | ||
| Error: Expected DecimalLiteral. | ||
| ╭─[crates/solidity/testing/snapshots/cst_output/DecimalNumberExpression/integer_ident_after_period/input.sol:1:1] | ||
| │ | ||
| 1 │ 1.a | ||
| │ ──┬─ | ||
| │ ╰─── Error occurred here. | ||
| ───╯ | ||
|
|
||
| Tree: | ||
| - (SKIPPED): "1.a\n" # (0..4) |
19 changes: 19 additions & 0 deletions
19
...cst_output/DecimalNumberExpression/integer_ident_after_period/generated/0.5.0-failure.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| # This file is generated automatically by infrastructure scripts. Please don't edit by hand. | ||
|
|
||
| Source: > | ||
| 1 │ 1.a │ 0..3 | ||
|
|
||
| Errors: # 1 total | ||
| - > | ||
| Error: Expected DaysKeyword or EtherKeyword or FinneyKeyword or HoursKeyword or MinutesKeyword or SecondsKeyword or SzaboKeyword or WeeksKeyword or WeiKeyword. | ||
| ╭─[crates/solidity/testing/snapshots/cst_output/DecimalNumberExpression/integer_ident_after_period/input.sol:1:2] | ||
| │ | ||
| 1 │ 1.a | ||
| │ ─┬─ | ||
| │ ╰─── Error occurred here. | ||
| ───╯ | ||
|
|
||
| Tree: | ||
| - (DecimalNumberExpression): # "1.a\n" (0..4) | ||
| - (literal꞉ DecimalLiteral): "1" # (0..1) | ||
| - (SKIPPED): ".a\n" # (1..4) |
19 changes: 19 additions & 0 deletions
19
...st_output/DecimalNumberExpression/integer_ident_after_period/generated/0.6.11-failure.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| # This file is generated automatically by infrastructure scripts. Please don't edit by hand. | ||
|
|
||
| Source: > | ||
| 1 │ 1.a │ 0..3 | ||
|
|
||
| Errors: # 1 total | ||
| - > | ||
| Error: Expected DaysKeyword or EtherKeyword or FinneyKeyword or GweiKeyword or HoursKeyword or MinutesKeyword or SecondsKeyword or SzaboKeyword or WeeksKeyword or WeiKeyword. | ||
| ╭─[crates/solidity/testing/snapshots/cst_output/DecimalNumberExpression/integer_ident_after_period/input.sol:1:2] | ||
| │ | ||
| 1 │ 1.a | ||
| │ ─┬─ | ||
| │ ╰─── Error occurred here. | ||
| ───╯ | ||
|
|
||
| Tree: | ||
| - (DecimalNumberExpression): # "1.a\n" (0..4) | ||
| - (literal꞉ DecimalLiteral): "1" # (0..1) | ||
| - (SKIPPED): ".a\n" # (1..4) |
19 changes: 19 additions & 0 deletions
19
...cst_output/DecimalNumberExpression/integer_ident_after_period/generated/0.7.0-failure.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| # This file is generated automatically by infrastructure scripts. Please don't edit by hand. | ||
|
|
||
| Source: > | ||
| 1 │ 1.a │ 0..3 | ||
|
|
||
| Errors: # 1 total | ||
| - > | ||
| Error: Expected DaysKeyword or EtherKeyword or GweiKeyword or HoursKeyword or MinutesKeyword or SecondsKeyword or WeeksKeyword or WeiKeyword. | ||
| ╭─[crates/solidity/testing/snapshots/cst_output/DecimalNumberExpression/integer_ident_after_period/input.sol:1:2] | ||
| │ | ||
| 1 │ 1.a | ||
| │ ─┬─ | ||
| │ ╰─── Error occurred here. | ||
| ───╯ | ||
|
|
||
| Tree: | ||
| - (DecimalNumberExpression): # "1.a\n" (0..4) | ||
| - (literal꞉ DecimalLiteral): "1" # (0..1) | ||
| - (SKIPPED): ".a\n" # (1..4) |
1 change: 1 addition & 0 deletions
1
...testing/snapshots/cst_output/DecimalNumberExpression/integer_ident_after_period/input.sol
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| 1.a |
17 changes: 17 additions & 0 deletions
17
...t/DecimalNumberExpression/leading_period_ident_after_decimal/generated/0.4.11-failure.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| # This file is generated automatically by infrastructure scripts. Please don't edit by hand. | ||
|
|
||
| Source: > | ||
| 1 │ .1a │ 0..3 | ||
|
|
||
| Errors: # 1 total | ||
| - > | ||
| Error: Expected DecimalLiteral. | ||
| ╭─[crates/solidity/testing/snapshots/cst_output/DecimalNumberExpression/leading_period_ident_after_decimal/input.sol:1:1] | ||
| │ | ||
| 1 │ .1a | ||
| │ ──┬─ | ||
| │ ╰─── Error occurred here. | ||
| ───╯ | ||
|
|
||
| Tree: | ||
| - (SKIPPED): ".1a\n" # (0..4) |
1 change: 1 addition & 0 deletions
1
...snapshots/cst_output/DecimalNumberExpression/leading_period_ident_after_decimal/input.sol
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| .1a |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure I understand the fix here:
5, it will always be matched by the first definition.5., it will be matched by the second definition till version0.5.0, and rejected afterwards.5.6, it will always be matched by the last definition.Inserting this new definition into the 3rd position just overrides the last one, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's important to think in terms of accepted lexemes by the scanner here. From 0.5.0 upwards, if we have
5.member(), the old definitions did not allow us to lex5as a standalone literal:.[0-9], so the old 3rd definition fails[0-9].[0-9], so it also fails.This rule matches the logic introduced in the Solidity's lexer for 0.5.0, which is to stop and return a Token::Number if there's a period ahead that's not followed by a digit: https://github.com/ethereum/solidity/blob/cc79c91b93d322cd38beec08a1ce575d3622c7e4/liblangutil/Scanner.cpp#L972. Now, since we have
Optional(Sequence([Atom("."),Fragment(DecimalDigits)])), we enforce that if there's a dot, it must have a decimal ahead; we must either consume the dot with digits after or nothing, following the first digit.This allows us to lex
5and then be able to parse.member()as the postfix member access operator, since otherwise consuming the dot and lexing5.as a decimal literal would've made that impossible.We can't modify the last rule that works across all versions, because <0.5.0
5.member()did not parse correctly exactly because the solc lexer consumed the dot, as per above, so we can't enable that behaviour prior to 0.5.0.Does this make sense? Do you have a better idea how to express that behaviour more clearly?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see. Thanks for explaining!
I think my concern here is that the new definition basically supersedes other definitions. In this case, what do you think of the following:
An integer (without a dot or a fraction) is enabled in all versionsAn integer and a dot (without a fraction) is enabled till "0.5.0"fractionhere and make it optionalAn integer, a dot, and a fraction is enabled from "0.5.0"A dot and a fraction (without an integer) is enabled in all versionsAn integer, a dot, and a fraction is enabled in all versionsThis has the additional benefit of keeping the specific scanners mutually exclusive, which is going to allow building a state machine for the scanner. Most of the
TokenDefinitions in the DSL are built for this reason.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That sounds good in theory and I've tried it, but we still need to be able to optionally skip the
(. DECIMAL_DIGITS). That is because the first rule has to have a negative lookahead for.; otherwise we end up accepting5.member()as valid in < 0.5.0.I've pushed a version that is hopefully more acceptable, since a bit more disjoint now. Whenever we move to compiling these down to a state machine, I'm happy to revisit the definitions 🤞