fix: Don't consume the dot in decimal literals if not followed by digit since 0.5.0

[Xanewok]

See ethereum/solidity#4172.

This unblocks parsing member access expressions where the primary
expression can't consume the dot in order to parse the .member postfix
expression.

[changeset-bot]

🦋 Changeset detected

Latest commit: 84fd161

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@nomicfoundation/slang	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[OmarTawfik]

Not sure I understand the fix here:

• For input 5, it will always be matched by the first definition.
• For input 5., it will be matched by the second definition till version 0.5.0, and rejected afterwards.
• For input 5.6, it will always be matched by the last definition.

Inserting this new definition into the 3rd position just overrides the last one, right?

[Xanewok]

It's important to think in terms of accepted lexemes by the scanner here. From 0.5.0 upwards, if we have 5.member(), the old definitions did not allow us to lex 5 as a standalone literal:

• it's followed by a dot, so 1st definition fails
• it's >=0.5.0, so the second rule is disabled
• it's not just .[0-9], so the old 3rd definition fails
• it's not [0-9].[0-9], so it also fails.

This rule matches the logic introduced in the Solidity's lexer for 0.5.0, which is to stop and return a Token::Number if there's a period ahead that's not followed by a digit: https://github.com/ethereum/solidity/blob/cc79c91b93d322cd38beec08a1ce575d3622c7e4/liblangutil/Scanner.cpp#L972. Now, since we have Optional(Sequence([Atom("."),Fragment(DecimalDigits)])), we enforce that if there's a dot, it must have a decimal ahead; we must either consume the dot with digits after or nothing, following the first digit.

This allows us to lex 5 and then be able to parse .member() as the postfix member access operator, since otherwise consuming the dot and lexing 5. as a decimal literal would've made that impossible.

We can't modify the last rule that works across all versions, because <0.5.0 5.member() did not parse correctly exactly because the solc lexer consumed the dot, as per above, so we can't enable that behaviour prior to 0.5.0.

Does this make sense? Do you have a better idea how to express that behaviour more clearly?

[OmarTawfik]

I see. Thanks for explaining!
I think my concern here is that the new definition basically supersedes other definitions. In this case, what do you think of the following:

• An integer (without a dot or a fraction) is enabled in all versions
  • stays the same
• An integer and a dot (without a fraction) is enabled till "0.5.0"
  • add the fraction here and make it optional
• An integer, a dot, and a fraction is enabled from "0.5.0"
  • add this new one (all three are required)
• A dot and a fraction (without an integer) is enabled in all versions
  • stays the same
• An integer, a dot, and a fraction is enabled in all versions
  • removed, as it is replaced by the second definition.

This has the additional benefit of keeping the specific scanners mutually exclusive, which is going to allow building a state machine for the scanner. Most of the TokenDefinitions in the DSL are built for this reason.

[Xanewok]

That sounds good in theory and I've tried it, but we still need to be able to optionally skip the (. DECIMAL_DIGITS). That is because the first rule has to have a negative lookahead for .; otherwise we end up accepting 5.member() as valid in < 0.5.0.

I've pushed a version that is hopefully more acceptable, since a bit more disjoint now. Whenever we move to compiling these down to a state machine, I'm happy to revisit the definitions 🤞

[OmarTawfik]

Left one question above. Thanks!