The alt_bn128 syscalls do not conform to the EIP-197 standard in the way it handles flags

[samkim-crypto]

Problem

There is currently a difference in the way curve encodings are handled in EIP-197 and the alt_bn128 syscalls on Solana that relies on the arkworks crate.

In the arkworks implementation, a field element is represented as an array [u64; 4]. Since the base field modulus in bn128 (bn254) is a 254-bit number, when field elements or curve points are serialized, the two most significant bits (the 255th and 256th bits) are used as flag bits to store additional information about a curve point (YIsNegative and PointAtInfinity).

In the current alt_bn128 syscalls, since these flags are not relevant for the curve addition, multiplication, and pairings operations, the flag bits are zeroed out before the bytes are deserialized into curve points.

In the ethereum implementations of EIP-197, the flag bits are not used. Therefore, on field inputs where the 255th or 256th bits are set to 1, the function will interpret it as a number that exceeds the modulus and reject the input (instead of zeroing out the flags and deserializing).

Proposed Solution

To make the alt_bn128 syscalls to conform to EIP-197, we can add a simple logic at the beginning of each addition, multiplication, and pairing syscalls to check for the 255th and 256th bits, and reject if any of these bits are set.