[3.11] gh-95778: Correctly pre-check for int-to-str conversion (GH-96537)

[miss-islington]

Converting a large enough int to a decimal string raises ValueError as expected. However, the raise comes after the quadratic-time base-conversion algorithm has run to completion. For effective DOS prevention, we need some kind of check before entering the quadratic-time loop. Oops! =)

The quick fix: essentially we catch most values that exceed the threshold up front. Those that slip through will still be on the small side (read: sufficiently fast), and will get caught by the existing check so that the limit remains exact.

The justification for the current check. The C code check is:

max_str_digits / (3 * PyLong_SHIFT) <= (size_a - 11) / 10

In GitHub markdown math-speak, writing $M$ for max_str_digits, $L$ for PyLong_SHIFT and $s$ for size_a, that check is:
$$\left\lfloor\frac{M}{3L}\right\rfloor \le \left\lfloor\frac{s - 11}{10}\right\rfloor$$

From this it follows that
$$\frac{M}{3L} < \frac{s-1}{10}$$
hence that
$$\frac{L(s-1)}{M} > \frac{10}{3} > \log_2(10).$$
So
$$2^{L(s-1)} > 10^M.$$
But our input integer $a$ satisfies $|a| \ge 2^{L(s-1)}$, so $|a|$ is larger than $10^M$. This shows that we don't accidentally capture anything below the intended limit in the check.

• Issue: CVE-2020-10735: Prevent DoS by large int<->str conversions #95778

Co-authored-by: Gregory P. Smith [Google LLC] greg@krypto.org
(cherry picked from commit b126196)

Co-authored-by: Mark Dickinson dickinsm@gmail.com

Automerge-Triggered-By: GH:gpshead

[miss-islington]

Status check is done, and it's a failure ❌.