Accuracy issues of sum() specialization for floats/complexes

[skirpichev]

Bug report

Bug description:

Unfortunately, #121176 was merged with a bug:

cpython/Python/bltinmodule.c

Lines 2749 to 2755 in e968121

	if (PyFloat_Check(item)) {
	double value = PyFloat_AS_DOUBLE(item);
	re_sum.hi += value;
	im_sum.hi += 0.0;
	_Py_DECREF_SPECIALIZED(item, _PyFloat_ExactDealloc);
	continue;
	}

L2751 lacks cs_add(). Sorry for that. Reproducer: sum([2j, 1., 10E100, 1., -10E100]) (should be 2+2j). I'll provide a patch.

But maybe cases for integer arguments also should use compensated summation? E.g.:

cpython/Python/bltinmodule.c

Lines 2689 to 2698 in e968121

	if (PyLong_Check(item)) {
	long value;
	int overflow;
	value = PyLong_AsLongAndOverflow(item, &overflow);
	if (!overflow) {
	re_sum.hi += (double)value;
	Py_DECREF(item);
	continue;
	}
	}

on L2694 (and use PyLong_AsDouble()). An example:

>>> sum([1.0, 10E100, 1.0, -10E100])
2.0
>>> sum([1.0, 10**100, 1.0, -10**100])  # huh?
0.0

I would guess, that integer values in this case are treated as exact and they are allowed to smash floating-point result to garbage. But... This looks as a bug for me. fsum() also chooses 2.0:

>>> math.fsum([1.0, 10**100, 1.0, -10**100])
2.0

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Linked PRs

• gh-122234: fix accuracy issues for sum() #122236
• gh-122234: Add DECREFs to error paths #122406

[skirpichev]

CC @rhettinger for the second part (integer values). Is this an issue or a feature?

[picnixz]

Actually, even without the sum, we have:

>>> x = 10 ** 100
>>> 1.0 + x + 1.0 - x
0.0

[skirpichev]

@picnixz, that's a feature.

Ok, it seems that in the second case PyLong_AsLongAndOverflow() just overflows and we fallback to the generic sum. That's something copied from the specialization for integers (in 8ce8a78) and it wasn't changed in #100426. Maybe it should?

[skirpichev]

Hmm, with PyLong_AsDouble() + compensated summation it's even faster!

./python -m timeit -r11 -s 'xs=[1.0, 10**100, 1.0, -10**100]' 'sum(xs)'
500000 loops, best of 11: 634 nsec per loop

while in the main:

$ ./python -m timeit -r11 -s 'xs=[1.0, 10**100, 1.0, -10**100]' 'sum(xs)'
500000 loops, best of 11: 801 nsec per loop

[skirpichev]

PR is ready for review: #122236

It combines fixes for floats and integers. If second case requires more discussion or is "a feature", I can quickly revert that part.

[encukou]

The added error returns leak references. This was caught by the noGIL refleak buildbot (which runs more often than refleaks for regular-builds): https://buildbot.python.org/#/builders/1226/builds/2352/steps/6/logs/stdio

[Eclips4]

The added error returns leak references. This was caught by the noGIL refleak buildbot (which runs more often than refleaks for regular-builds): https://buildbot.python.org/#/builders/1226/builds/2352/steps/6/logs/stdio

PR is ready: #122405

[encukou]

My PR #122406 looks exactly the same :)
IMO, it's better to associate the PR with this issue, so the fix-up PR is kept together with the original.

[picnixz]

Closing since the refleak is now fixed.