Skip to content

Add grammar for char, string, byte, and raw literals #121

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Sep 26, 2017
Merged

Add grammar for char, string, byte, and raw literals #121

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
74f22eb
Add grammar for char and string literals
brauliobz Sep 24, 2017
2f8877d
Added _ (underscore) separators to Unicode escapes
brauliobz Sep 26, 2017
dbc4ab8
Use a repetition notation for unicode escapes to make it smaller.
brauliobz Sep 26, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 77 additions & 2 deletions src/tokens.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,24 @@ evaluated (primarily) at compile time.

| | Example | `#` sets | Characters | Escapes |
|----------------------------------------------|-----------------|------------|-------------|---------------------|
| [Character](#character-literals) | `'H'` | `N/A` | All Unicode | [Quote](#quote-escapes) & [Byte](#byte-escapes) & [Unicode](#unicode-escapes) |
| [String](#string-literals) | `"hello"` | `N/A` | All Unicode | [Quote](#quote-escapes) & [Byte](#byte-escapes) & [Unicode](#unicode-escapes) |
| [Character](#character-literals) | `'H'` | `N/A` | All Unicode | [Quote](#quote-escapes) & [ASCII](#ascii-escapes) & [Unicode](#unicode-escapes) |
| [String](#string-literals) | `"hello"` | `N/A` | All Unicode | [Quote](#quote-escapes) & [ASCII](#ascii-escapes) & [Unicode](#unicode-escapes) |
| [Raw](#raw-string-literals) | `r#"hello"#` | `0...` | All Unicode | `N/A` |
| [Byte](#byte-literals) | `b'H'` | `N/A` | All ASCII | [Quote](#quote-escapes) & [Byte](#byte-escapes) |
| [Byte string](#byte-string-literals) | `b"hello"` | `N/A` | All ASCII | [Quote](#quote-escapes) & [Byte](#byte-escapes) |
| [Raw byte string](#raw-byte-string-literals) | `br#"hello"#` | `0...` | All ASCII | `N/A` |

#### ASCII escapes

| | Name |
|---|------|
| `\x41` | 7-bit character code (exactly 2 digits, up to 0x7F) |
| `\n` | Newline |
| `\r` | Carriage return |
| `\t` | Tab |
| `\\` | Backslash |
| `\0` | Null |

#### Byte escapes

| | Name |
Expand Down Expand Up @@ -74,12 +85,39 @@ evaluated (primarily) at compile time.

#### Character literals

> **<sup>Lexer</sup>**
> CHAR_LITERAL :
> &nbsp;&nbsp; `'` ( ~[`'` `\` \\n \\r \\t] | QUOTE_ESCAPE | ASCII_ESCAPE | UNICODE_ESCAPE ) `'`
>
> QUOTE_ESCAPE :
> &nbsp;&nbsp; `\'` | `\"`
>
> ASCII_ESCAPE :
> &nbsp;&nbsp; &nbsp;&nbsp; `\x` OCT_DIGIT HEX_DIGIT
> &nbsp;&nbsp; | `\n` | `\r` | `\t` | `\\` | `\0`
>
> UNICODE_ESCAPE :
> &nbsp;&nbsp; `\u{` ( HEX_DIGIT `_`<sup>\*</sup> )<sup>1..6</sup> `}`

A _character literal_ is a single Unicode character enclosed within two
`U+0027` (single-quote) characters, with the exception of `U+0027` itself,
which must be _escaped_ by a preceding `U+005C` character (`\`).

#### String literals

> **<sup>Lexer</sup>**
> STRING_LITERAL :
> &nbsp;&nbsp; `"` (
> &nbsp;&nbsp; &nbsp;&nbsp; ~[`"` `\` _IsolatedCR_]
> &nbsp;&nbsp; &nbsp;&nbsp; | QUOTE_ESCAPE
> &nbsp;&nbsp; &nbsp;&nbsp; | ASCII_ESCAPE
> &nbsp;&nbsp; &nbsp;&nbsp; | UNICODE_ESCAPE
> &nbsp;&nbsp; &nbsp;&nbsp; | STRING_CONTINUE
> &nbsp;&nbsp; )<sup>\*</sup> `"`
>
> STRING_CONTINUE :
> &nbsp;&nbsp; `\` _followed by_ \\n

A _string literal_ is a sequence of any Unicode characters enclosed within two
`U+0022` (double-quote) characters, with the exception of `U+0022` itself,
which must be _escaped_ by a preceding `U+005C` character (`\`).
Expand Down Expand Up @@ -120,6 +158,14 @@ following forms:

#### Raw string literals

> **<sup>Lexer</sup>**
> RAW_STRING_LITERAL :
> &nbsp;&nbsp; `r` RAW_STRING_CONTENT
>
> RAW_STRING_CONTENT :
> &nbsp;&nbsp; &nbsp;&nbsp; `"` ( ~ _IsolatedCR_ )<sup>* (non-greedy)</sup> `"`
> &nbsp;&nbsp; | `#` RAW_STRING_CONTENT `#`

Raw string literals do not process any escapes. They start with the character
`U+0072` (`r`), followed by zero or more of the character `U+0023` (`#`) and a
`U+0022` (double-quote) character. The _raw string body_ can contain any sequence
Expand Down Expand Up @@ -149,6 +195,17 @@ r##"foo #"# bar"##; // foo #"# bar

#### Byte literals

> **<sup>Lexer</sup>**
> BYTE_LITERAL :
> &nbsp;&nbsp; `b'` ( ASCII_FOR_CHAR | BYTE_ESCAPE ) `'`
>
> ASCII_FOR_CHAR :
> &nbsp;&nbsp; _any ASCII (i.e. 0x00 to 0x7F), except_ `'`, `/`, \\n, \\r or \\t
>
> BYTE_ESCAPE :
> &nbsp;&nbsp; &nbsp;&nbsp; `\x` HEX_DIGIT HEX_DIGIT
> &nbsp;&nbsp; | `\n` | `\r` | `\t` | `\\` | `\0`

A _byte literal_ is a single ASCII character (in the `U+0000` to `U+007F`
range) or a single _escape_ preceded by the characters `U+0062` (`b`) and
`U+0027` (single-quote), and followed by the character `U+0027`. If the character
Expand All @@ -158,6 +215,13 @@ _number literal_.

#### Byte string literals

> **<sup>Lexer</sup>**
> BYTE_STRING_LITERAL :
> &nbsp;&nbsp; `b"` ( ASCII_FOR_STRING | BYTE_ESCAPE | STRING_CONTINUE )<sup>\*</sup> `"`
>
> ASCII_FOR_STRING :
> &nbsp;&nbsp; _any ASCII (i.e 0x00 to 0x7F), except_ `"`, `/` _and IsolatedCR_

A non-raw _byte string literal_ is a sequence of ASCII characters and _escapes_,
preceded by the characters `U+0062` (`b`) and `U+0022` (double-quote), and
followed by the character `U+0022`. If the character `U+0022` is present within
Expand All @@ -183,6 +247,17 @@ following forms:

#### Raw byte string literals

> **<sup>Lexer</sup>**
> RAW_BYTE_STRING_LITERAL :
> &nbsp;&nbsp; `br` RAW_BYTE_STRING_CONTENT
>
> RAW_BYTE_STRING_CONTENT :
> &nbsp;&nbsp; &nbsp;&nbsp; `"` ASCII<sup>* (non-greedy)</sup> `"`
> &nbsp;&nbsp; | `#` RAW_STRING_CONTENT `#`
>
> ASCII :
> &nbsp;&nbsp; _any ASCII (i.e. 0x00 to 0x7F)_

Raw byte string literals do not process any escapes. They start with the
character `U+0062` (`b`), followed by `U+0072` (`r`), followed by zero or more
of the character `U+0023` (`#`), and a `U+0022` (double-quote) character. The
Expand Down