kernel: prevent out-of-bound access creating finite fields

The kernel function FiniteField takes a prime <p> and a degree <d> as
argument, and creates a small finite field with <q>:=<p>^<d> elements.
However, it can be called with invalid arguments (e.g. where <p> is not a
prime, or where <q> exceeds 2^16). It thus needs to validate its arguments,
and several function calling it in fact rely on it.

However, the linear interpolation search it used failed to do this properly,
and thus if <q> was not a prime power, or was too large, it could end up
performing out-of-bound accesses to the <SizeFF> array. Depending on the
content of the memory it incorrect accessed, this could lead to an infinite
loop, or to a correct error (because of some end validation), or
hypothetically to nonsense computations (but only if you were *really* unlucky
and the out-bounds-access resulted in *exactly* the right value).

This is now fixed by this commit. In addition, after the linear interpolation
search, we now verify that the <ff> index is not out-of-bounds *before* using
it to access the SizeFF array.

Fixes #1382

src/finfield.c

@@ -529,7 +529,8 @@ FF FiniteField (
 
  /* search through the finite field table */
  l = 1; n = NUM_SHORT_FINITE_FIELDS;
- while (l <= n) {
+ ff = 0;
+ while (l <= n && SizeFF[l] <= q && q <= SizeFF[n]) {
  /* interpolation search */
  /* cuts iterations roughly in half compared to binary search at
  * the expense of additional divisions. */
@@ -542,10 +543,10 @@ FF FiniteField (
  else
  n = ff-1;
  }
+ if (ff < 1 || ff > NUM_SHORT_FINITE_FIELDS)
+ return 0;
  if (SizeFF[ff] != q)
  return 0;
- if (ff > NUM_SHORT_FINITE_FIELDS)
- return 0;
 #ifdef HPCGAP
  /* Important correctness concern here:
  *