Character Literals (#1934)

proposals/p1964.md

@@ -26,6 +26,8 @@ SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
     -   [Supporting prefix declarations](#supporting-prefix-declarations)
     -   [Allowing numeric escape sequences](#allowing-numeric-escape-sequences)
     -   [Supporting formulations of grapheme clusters and non-code-point code-units](#supporting-formulations-of-grapheme-clusters-and-non-code-point-code-units)
+-   [Future Work](#future-work)
+    -   [UTF code unit types proposal](#utf-code-unit-types-proposal)
 
 <!-- tocstop -->
 
@@ -105,8 +107,6 @@ like numeric literals:
 -   Character literals support some back-slash (`\`) escape sequences, including
     `\t`, `\n`, `\r`, `\"`, `\'`, `\\`, `\0`, and `\u{HHHH...}`. See
     [String Literals: Escape sequence](https://github.com/carbon-language/carbon-lang/blob/trunk/proposals/p0199.md#escape-sequences).
--   Character literals implicitly convert to Unicode strings, even though the
-    numeric values of the encoding may change.
 
 We will not support:
 
@@ -115,89 +115,52 @@ We will not support:
 -   "raw" literals (using #'x'#);
 -   `\x` escape sequences;
 -   character literals with a single quote (`'`) or back-slash (`\`), except as
-    part of an escape sequence;
+    part of an escape sequence
 -   empty character literals (`''`);
 -   ASCII control codes (0...31), including whitespace characters other than
     word space (tab, line feed, carriage return, form feed, and vertical tab),
     except when specified with an escape sequence.
 
 ### Types
 
-We will have the types `Char8`, `Char16`, and `Char32` representing code units
-in UTF-8, UTF-16, and UTF-32, but we will not support all code units, but only
-those which map directly to the complete value of a code point. However,
-character literals will use their own types distinct from these:
+For the time being we will support a type `CharN` that will hold both code units
+and code points, and will leave the different UTF-encoding code unit types to
+another proposal. See
+[UTF code unit types proposal](#utf-code-unit-types-proposal))
 
--   We will support value preserving implicit conversions from character
-    literals to code point or code unit types. In particular, a character
-    literal converts to a `Char8` UTF-8 code unit if it is less than or equal to
-    0x7F, and `Char16` UTF-16 code unit if it is less than or equal to 0xFFFF.
--   Conversions from string or character literals to a non-value-preserving
-    encoding must be explicit.
--   Conversions from string literals to Unicode strings are implicit, even
-    though the numeric values of the encoding may change.
-
-We can see whether the particular literal is represented in the variable's type
-by only looking at the types.
+We will have the type `CharN` and only support literals that map directly to the
+complete value of a code point.
 
 ```
-let allowed: Char8 = 'a';
+let allowed: CharN = 'a';
 ```
 
 The above is allowed because the type of `'a'` is the character literal
 consisting of the single Unicode code point 97, which can be converted to
-`Char8` since 97 is less than or equal to 0x7F.
-
-```
-let error1: Char8 = '😃';
-let error2: Char8 = 'AB';
-```
-
-However these should produce errors. The type of `'😃'` is the character literal
-consisting of the single Unicode code point `0x1F603`, which is greater than
-0x7F. The type of `'AB'` is a character literal that is a sequence of two
-Unicode code points, which is not valid.
-
-Literals `'\n'` and `'\u{A}'` represent the same character and so have the same
-type. However, explicitly converting this character literal to another character
-set might result in a character with a different value, but that still
-represents the newline character.
-
-```
-// 0x15 is the new line character in EBCDIC.
-Assert('\n' as EBCDICChar == EBCDICChar.Make(0x15));
-Assert('\u{A}' as EBCDICChar != EBCDICChar.Make(0x0A));
-```
+`CharN` since 97 is less than or equal to 0x7F.
 
 ### Operations
 
 Character literals representing a single code point support the following
 operators:
 
 -   Comparison: `<`, `>`, `<=`, `>=` `==`
--   Plus: `+`. However the behavior of this operator is different from its use
-    with strings. Rather than a concatenation, this will add the value of the
-    two characters:
-    -   If the `+` is used between a character literal representing a single
-        Unicode code point and an integer literal, this should produce a
-        character literal if the result fits in a Unicode code point.
-    -   If the `+` is used between two character literals this will produce an
-        error.
--   Subtract: `-` Similar to the plus operator, this will subtract the value of
-    the two characters.
-
-    -   If the `-` is used between a character literal representing a single
-        code point or code unit and an integer literal, this should produce a
-        character literal as long as the result of the difference is in range.
-    -   If the `-` is used between two character literals, which are both a
-        single code point, this should produce an integer literal representing
-        the difference between the code points.
-
-    In cases where the `+` and `-` operators are used with a character literal
-    and a non-integer non-literal, the character literal should be converted to
-    the type of the other operand if possible. For example, in the expression
-    `w - 'a'`, where `w` is of type `Char8`, the `'a'` literal will be converted
-    to type `Char8`.
+-   Plus: `+`. This doesn't concatenate, but allows numerically adjusting the
+    value:
+    -   Only one operand may be a character literal, the other must be an
+        integer literal.
+    -   The result is the character literal whose numeric value is the sum of
+        numeric value of the operands. If that sum is not a valid Unicode code
+        point, it is an error.
+-   Subtract: `-`. This will subtract the value of the two characters, or a
+    character followed by an integer literal:
+    -   If the `-` is used between two character literals, the result will be a
+        character constant `'z' - 4`.
+    -   If the `-` is used between a character literal, followed by a integer
+        literal this will produce an integer constant `'z' - 'a'`.
+    -   If the `-` is used between a integer literal followed by a character
+        literal `100 - 'a'`, this will be rejected unless the integer is cast to
+        a character.
 
 There is intentionally no implicit conversion from character literals to integer
 types, but explicit conversions are permitted between character literals and
@@ -212,11 +175,11 @@ Adding support for a specific character literal supports clean, readable,
 concise use and is a much more familiar concept that will make it easier to
 adopt Carbon coming from other languages. Have a distinct character literal will
 also allow us support useful operations designed to manipulate the literal's
-value. When working with `String`, we use the `+` operator to concatenate
-multiple `String`s, but say we wanted to advance a character to the next
-literal. Using a `String` will produce a type error, as we are misusing the `+`
-operator: `"a" + 1`. However with a character literal, we can support operations
-for these use cases:
+value. When working with an explicit character type we can use operators that
+have unique behavior, for example say we wanted to advance a character to the
+next literal. In other languages the `+` operator is often used for
+concatenation, so using a `String` will produce a type error: `"a" + 1`. However
+with a character literal, we can support operations for these use cases:
 
 ```
 var b: u8;
@@ -275,6 +238,9 @@ disadvantages:
 -   It's useful for developers to document the intended meaning of a value, and
     using a distinct type is one way to do that.
 
+See [UTF code unit types proposal](#utf-code-unit-types-proposal) for more
+information about UTF encoding types for a future proposal.
+
 ### No distinct character literal
 
 In principle, a character literal can be represented by reusing string literals
@@ -354,3 +320,53 @@ single Unicode code points following a more integer-like model. However this
 topic should be revisited if we find that there is a significant need for the
 additional functionality and attendant complexity for these other character
 formulations.
+
+## Future Work
+
+### UTF code unit types proposal
+
+There have been several ideas and discussions around how we would like to handle
+UTF code units. This section will hopefully provide some guidance for a future
+proposal when the topic is revisited for how we would like to build out
+encoding/decoding for character literals.
+
+We will have the types `Char8`, `Char16`, and `Char32` representing code units
+in UTF-8, UTF-16, and UTF-32, but we will not support all code units, but only
+those which map directly to the complete value of a code point. However,
+character literals will use their own types distinct from these:
+
+-   We will support value preserving implicit conversions from character
+    literals to code point or code unit types. In particular, a character
+    literal converts to a `Char8` UTF-8 code unit if it is less than or equal to
+    0x7F, and `Char16` UTF-16 code unit if it is less than or equal to 0xFFFF.
+-   Conversions from string or character literals to a non-value-preserving
+    encoding must be explicit.
+-   Conversions from string literals to Unicode strings are implicit, even
+    though the numeric values of the encoding may change.
+
+We can see whether the particular literal is represented in the variable's type
+by only looking at the types.
+
+```
+let allowed: Char8 = 'a';
+```
+
+The above is allowed because the type of `'a'` is the character literal
+consisting of the single Unicode code point 97, which can be converted to
+`Char8` since 97 is less than or equal to 0x7F.
+
+```
+let error1: Char8 = '😃';
+let error2: Char8 = 'AB';
+```
+
+However these should produce errors. The type of `'😃'` is the character literal
+consisting of the single Unicode code point `0x1F603`, which is greater than
+0x7F. The type of `'AB'` is a character literal that is a sequence of two
+Unicode code points, which has no conversion to a type that only handles a
+single UTF-8 code unit.
+
+All of `'\n'`, and `'\u{A}'` represent the same character and so have the same
+type. However, explicitly converting this character literal to another character
+set might result in a character with a different value, but that still
+represents the newline character.