Bitwise operators

docs/design/expressions/arithmetic.md

@@ -159,8 +159,12 @@ programming errors:
     will be aborted, or the arithmetic will evaluate to a mathematically
     incorrect result, such as a two's complement result or zero. The program
     might not in all cases be aborted immediately -- for example, multiple
-    overflow checks might be combined into one, and if the result of an
-    arithmetic operation is never observed, the abort may not happen at all.
+    overflow checks might be combined into one -- but no control flow or memory
+    access that depends on the value will be performed.
+
+**TODO:** Unify the description of these programming errors with those of
+bit-shift domain errors, document the behavior in a common place and link to it
+from here.
 
 **TODO:** In a hardened build, should we prefer to trap on overflow, give a
 two's complement result, or produce zero? Using zero may defeat some classes of

docs/design/expressions/bitwise.md

@@ -131,15 +131,23 @@ to a division by a power of two, rounding downwards.
 
 The second operand of a bit-shift is required to be between zero (inclusive) and
 the bit-width of the first operand (exclusive); it is a programming error if the
-second operand is not within that range. In a hardened build, the result will
-have well defined behavior of either stopping program execution immediately or a
-shift of an unspecified number of bits, which if wider than the first operand
-will result in `0` or `-1`. In a performance build, the optimizer may assume
-that this programming error does not occur.
-
-TODO: Need to unify the developer and hardened build behavior for these
-programming errors with those of arithmetic overflow, likely documenting the
-requirements in a common place and linking to it from here.
+second operand is not within that range.
+
+-   In a development build, they will be caught immediately when they happen at
+    runtime.
+-   In a performance build, the optimizer may assume that this programming error
+    does not occur.
+-   In a hardened build, the result will have well the defined behavior of
+    either aborting the program or performing a shift of an unspecified number
+    of bits, which if wider than the first operand will result in `0` or `-1`.
+    In the case where the program is aborted, the program might not in all cases
+    be aborted immediately -- for example, multiple checks might be combined
+    into one -- but no control flow or memory access that depends on the value
+    will be performed.
+
+**TODO:** Unify the description of these programming errors with those of
+arithmetic overflow, document the behavior in a common place and link to it from
+here.
 
 ## Integer constants
 
@@ -266,6 +274,7 @@ to give the semantics described above.
 -   [Use different symbols for bitwise operators](/proposals/p1191.md#use-different-symbols-for-bitwise-operators)
 -   [Provide different operators for arithmetic and logical shifts](/proposals/p1191.md#provide-different-operators-for-arithmetic-and-logical-shifts)
 -   [Provide rotate operators](/proposals/p1191.md#provide-rotate-operators)
+-   [Guarantee the behavior of large shifts](/proposals/p1191.md#guarantee-behavior-of-large-shifts)
 -   [Support shifting a constant by a variable](/proposals/p1191.md#support-shifting-a-constant-by-a-variable)
 
 ## References

proposals/p1191.md

@@ -25,6 +25,7 @@ SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
         -   [Use `~`, or some other symbol, for complement](#use--or-some-other-symbol-for-complement)
     -   [Provide different operators for arithmetic and logical shifts](#provide-different-operators-for-arithmetic-and-logical-shifts)
     -   [Provide rotate operators](#provide-rotate-operators)
+    -   [Guarantee behavior of large shifts](#guarantee-behavior-of-large-shifts)
     -   [Support shifting a constant by a variable](#support-shifting-a-constant-by-a-variable)
     -   [Converting complements to unsigned types](#converting-complements-to-unsigned-types)
 
@@ -240,6 +241,60 @@ is the underlying mathematical model behind the two's complement representation.
 We could provide bitwise rotation operators. However, there doesn't seem to be a
 sufficient need to justify adding another operator symbol for this purpose.
 
+### Guarantee behavior of large shifts
+
+Logically, the behavior of shifts is meaningful for all values of the second
+operand:
+
+-   A shift by an amount greater than or equal to the bit-width of the first
+    operand will shift out all of the original bits, producing a result where
+    all value bits are the same.
+-   A shift in one direction by a negative amount is treated as a shift in the
+    opposite direction by the negation of that amount.
+
+Put another way, we can view the bits of the first operand as an N-bit window
+into an infinite sequence of bits, with infinitely many leading sign bits (all
+zeroes for an unsigned value) and infinitely many trailing zero bits after a
+notional binary point, and a shift moves that window around. Or equivalently, a
+shift is always a multiplication by 2<sup>N</sup> followed by rounding and
+wrapping.
+
+We could provide the correct result for all shifts, regardless of the magnitude
+of the second operand. The primary reason we do not do this is lack of hardware
+support. For example, x86 does not have an instruction to perform this
+operation. Rather, x86 provides shift instructions that mask off all bits of the
+second operand except for the bottom 5 or 6, meaning that a left shift of a
+64-bit operand by 64 will return the operand unchanged rather than producing
+zero. This is the approach taken by Python, except that Python rejects negative
+shift counts.
+
+We could instead provide x86-like behavior, guaranteeing to consider only the
+lowest `N` bits of the second operand when the first operand is an `iN` or `uN`.
+This would provide an operation that can be implemented by a single instruction
+on x86 platforms when `N` is 32 or 64, and for all smaller types and for all
+other platforms the operation can be implemented with two instructions: a mask
+and a shift. For larger types, single-instruction support may not be available,
+but nonetheless the performance will be close to optimal, requiring at most one
+additional mask. There is still some performance cost in some cases, but the
+primary reason we do not do this is the same reason we choose to not define
+signed integer overflow: this masked result is unlikely to be the value that the
+developer actually wanted. This is the approach taken by Java and JavaScript.
+
+Instead of the above options, Carbon treats a second operand that is not in the
+interval [0, N) as a programming error, just like signed integer overflow:
+
+-   Debugging builds can detect and report this error without the risk of false
+    positives.
+-   Performance builds can optimize on the basis that this situation will not
+    occur, and can in particular use the dedicated x86 instructions that ignore
+    the high order bits of the second operand.
+-   Optimized builds guarantee that either the programming error results in
+    program termination or that _some_ value is produced, and moreover that said
+    value is the result of applying _some_ mathematical shift to the input. For
+    example, it's valid for an `i32` shift to be implemented by an x86 64-bit
+    shift that will produce 0 if the second operand is in [32, 63) but that will
+    treat a second operand of, say, 64 or -64 the same as 0.
+
 ### Support shifting a constant by a variable
 
 We considered various ways to support
@@ -268,7 +323,14 @@ We considered the following options:
     new type that carries its left-hand operand as a type parameter and its
     right-hand operand as runtime state, and allow that type to be converted in
     the same way as its integer constant. However, this would introduce
-    substantial complexity, and would allow implicit conversions that notionally
+    substantial complexity: reasonable and expected uses such as
+    ```
+    var mask: u32 = (1 << a) - 1;
+    ```
+    would require a second new type for a shifted value plus an offset, and
+    general support would require a facility analogous to
+    [expression templates](https://en.wikipedia.org/wiki/Expression_templates).
+    Further, this facility would allow implicit conversions that notionally
     overflow, such as would happen in the above example when `a` is greater
     than 32.