fix(compiler)!: Apply correct rules for parsing Unicode whitespace #1554
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ospencer
commented
Dec 17, 2022
| @@ -113,7 +116,9 @@ let dec_float = [%sedlex.regexp? | |||
|
|
|||
| let unsigned_float = [%sedlex.regexp? dec_float]; | |||
|
|
|||
| let uident = [%sedlex.regexp? (lu, Star(xid_continue))]; | |||
| let uident = [%sedlex.regexp? | |||
| (Intersect(xid_start, lu), Star(xid_continue)) | |||
There was a problem hiding this comment.
This is equivalent to what it was before, but I felt it made sense to make it more explicit since it's not obvious that Lu is a subset of xid_start.
ospencer
force-pushed
the
oscar/fix-unicode-spaces
branch
from
3fbf020
to
44562b6
Compare
ospencer
force-pushed
the
oscar/fix-unicode-spaces
branch
2 times, most recently
from
9b823b2
to
7f0f63f
Compare
ospencer
force-pushed
the
oscar/fix-unicode-spaces
branch
from
7f0f63f
to
58d7f6a
Compare
This was referenced Mar 20, 2023
av8ta
pushed a commit
to av8ta/grain
that referenced
this pull request
Apr 11, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
While working on the language reference, I realized that the category used for whitespace wasn't quite right, disallowing some common whitespace while allowing some uncommon ones.
This PR states explicitly that Grain follows https://unicode.org/reports/tr31/#Pattern_Syntax for Unicode allowed in the syntax of the language.
Whitespace in Grain now properly adheres to Pattern_White_Space, with additional Grain semantics described below.
Whitespace includes:
Spaces, namely
U+0009U+000BU+0020U+200EU+200FLine separators, namely
U+000AU+000CU+000DU+0085U+2028U+2029Line separators act as end-of-statement characters in Grain. Note that this is distinct from file line endings—Grain supports only LF and CRLF (relevant for compiler error messages and tooling).