Unable to pip install multidict with gcc-14

[psss]

Long story short

With the latest Fedora Rawhide which freshly contains gcc-14 I'm not able to install multidict.

Expected behaviour

Package successfully installed.

Actual behaviour

I see the following error:

gcc -fno-strict-overflow -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -fexceptions -fcf-protection -fexceptions -fcf-protection -fexceptions -fcf-protection -fPIC -I/tmp/venv/include -I/usr/include/python3.12 -c multidict/_multidict.c -o build/temp.linux-x86_64-cpython-312/multidict/_multidict.o -O2 -std=c99 -Wall -Wsign-compare -Wconversion -fno-strict-aliasing -pedantic
multidict/_multidict.c: In function ‘multidict_getall’:
multidict/_multidict.c:458:37: error: initialization of ‘int’ from ‘void *’ makes integer from pointer without a cast [-Wint-conversion]
  458 |     static _PyArg_Parser _parser = {NULL, _keywords, "getall", 0};
      |                                     ^~~~
multidict/_multidict.c:458:37: note: (near initialization for ‘_parser.initialized’)
multidict/_multidict.c:458:43: error: initialization of ‘const char *’ from incompatible pointer type ‘const char * const*’ [-Wincompatible-pointer-types]
  458 |     static _PyArg_Parser _parser = {NULL, _keywords, "getall", 0};
      |                                           ^~~~~~~~~
multidict/_multidict.c:458:43: note: (near initialization for ‘_parser.format’)
multidict/_multidict.c:458:54: error: initialization of ‘const char * const*’ from incompatible pointer type ‘char *’ [-Wincompatible-pointer-types]
  458 |     static _PyArg_Parser _parser = {NULL, _keywords, "getall", 0};
      |                                                      ^~~~~~~~
multidict/_multidict.c:458:54: note: (near initialization for ‘_parser.keywords’)
multidict/_multidict.c: In function ‘multidict_getone’:
multidict/_multidict.c:503:37: error: initialization of ‘int’ from ‘void *’ makes integer from pointer without a cast [-Wint-conversion]
  503 |     static _PyArg_Parser _parser = {NULL, _keywords, "getone", 0};
      |                                     ^~~~
multidict/_multidict.c:503:37: note: (near initialization for ‘_parser.initialized’)
multidict/_multidict.c:503:43: error: initialization of ‘const char *’ from incompatible pointer type ‘const char * const*’ [-Wincompatible-pointer-types]
  503 |     static _PyArg_Parser _parser = {NULL, _keywords, "getone", 0};
      |                                           ^~~~~~~~~
multidict/_multidict.c:503:43: note: (near initialization for ‘_parser.format’)
multidict/_multidict.c:503:54: error: initialization of ‘const char * const*’ from incompatible pointer type ‘char *’ [-Wincompatible-pointer-types]
  503 |     static _PyArg_Parser _parser = {NULL, _keywords, "getone", 0};
      |                                                      ^~~~~~~~
multidict/_multidict.c:503:54: note: (near initialization for ‘_parser.keywords’)
multidict/_multidict.c: In function ‘multidict_get’:
multidict/_multidict.c:538:37: error: initialization of ‘int’ from ‘void *’ makes integer from pointer without a cast [-Wint-conversion]
  538 |     static _PyArg_Parser _parser = {NULL, _keywords, "get", 0};
      |                                     ^~~~
multidict/_multidict.c:538:37: note: (near initialization for ‘_parser.initialized’)
multidict/_multidict.c:538:43: error: initialization of ‘const char *’ from incompatible pointer type ‘const char * const*’ [-Wincompatible-pointer-types]
  538 |     static _PyArg_Parser _parser = {NULL, _keywords, "get", 0};
      |                                           ^~~~~~~~~
multidict/_multidict.c:538:43: note: (near initialization for ‘_parser.format’)
multidict/_multidict.c:538:54: error: initialization of ‘const char * const*’ from incompatible pointer type ‘char *’ [-Wincompatible-pointer-types]
  538 |     static _PyArg_Parser _parser = {NULL, _keywords, "get", 0};
      |                                                      ^~~~~
multidict/_multidict.c:538:54: note: (near initialization for ‘_parser.keywords’)
multidict/_multidict.c: In function ‘multidict_tp_dealloc’:
multidict/_multidict.c:712:5: warning: ‘UsingDeprecatedTrashcanMacro’ is deprecated [-Wdeprecated-declarations]
  712 |     Py_TRASHCAN_SAFE_BEGIN(self);
      |     ^~~~~~~~~~~~~~~~~~~~~~
multidict/_multidict.c: In function ‘multidict_add’:
multidict/_multidict.c:780:37: error: initialization of ‘int’ from ‘void *’ makes integer from pointer without a cast [-Wint-conversion]
  780 |     static _PyArg_Parser _parser = {NULL, _keywords, "add", 0};
      |                                     ^~~~
multidict/_multidict.c:780:37: note: (near initialization for ‘_parser.initialized’)
multidict/_multidict.c:780:43: error: initialization of ‘const char *’ from incompatible pointer type ‘const char * const*’ [-Wincompatible-pointer-types]
  780 |     static _PyArg_Parser _parser = {NULL, _keywords, "add", 0};
      |                                           ^~~~~~~~~
multidict/_multidict.c:780:43: note: (near initialization for ‘_parser.format’)
multidict/_multidict.c:780:54: error: initialization of ‘const char * const*’ from incompatible pointer type ‘char *’ [-Wincompatible-pointer-types]
  780 |     static _PyArg_Parser _parser = {NULL, _keywords, "add", 0};
      |                                                      ^~~~~
multidict/_multidict.c:780:54: note: (near initialization for ‘_parser.keywords’)
multidict/_multidict.c: In function ‘multidict_setdefault’:
multidict/_multidict.c:839:37: error: initialization of ‘int’ from ‘void *’ makes integer from pointer without a cast [-Wint-conversion]
  839 |     static _PyArg_Parser _parser = {NULL, _keywords, "setdefault", 0};
      |                                     ^~~~
multidict/_multidict.c:839:37: note: (near initialization for ‘_parser.initialized’)
multidict/_multidict.c:839:43: error: initialization of ‘const char *’ from incompatible pointer type ‘const char * const*’ [-Wincompatible-pointer-types]
  839 |     static _PyArg_Parser _parser = {NULL, _keywords, "setdefault", 0};
      |                                           ^~~~~~~~~
multidict/_multidict.c:839:43: note: (near initialization for ‘_parser.format’)
multidict/_multidict.c:839:54: error: initialization of ‘const char * const*’ from incompatible pointer type ‘char *’ [-Wincompatible-pointer-types]
  839 |     static _PyArg_Parser _parser = {NULL, _keywords, "setdefault", 0};
      |                                                      ^~~~~~~~~~~~
multidict/_multidict.c:839:54: note: (near initialization for ‘_parser.keywords’)
multidict/_multidict.c: In function ‘multidict_popone’:
multidict/_multidict.c:875:37: error: initialization of ‘int’ from ‘void *’ makes integer from pointer without a cast [-Wint-conversion]
  875 |     static _PyArg_Parser _parser = {NULL, _keywords, "popone", 0};
      |                                     ^~~~
multidict/_multidict.c:875:37: note: (near initialization for ‘_parser.initialized’)
multidict/_multidict.c:875:43: error: initialization of ‘const char *’ from incompatible pointer type ‘const char * const*’ [-Wincompatible-pointer-types]
  875 |     static _PyArg_Parser _parser = {NULL, _keywords, "popone", 0};
      |                                           ^~~~~~~~~
multidict/_multidict.c:875:43: note: (near initialization for ‘_parser.format’)
multidict/_multidict.c:875:54: error: initialization of ‘const char * const*’ from incompatible pointer type ‘char *’ [-Wincompatible-pointer-types]
  875 |     static _PyArg_Parser _parser = {NULL, _keywords, "popone", 0};
      |                                                      ^~~~~~~~
multidict/_multidict.c:875:54: note: (near initialization for ‘_parser.keywords’)
multidict/_multidict.c: In function ‘multidict_pop’:
multidict/_multidict.c:922:37: error: initialization of ‘int’ from ‘void *’ makes integer from pointer without a cast [-Wint-conversion]
  922 |     static _PyArg_Parser _parser = {NULL, _keywords, "pop", 0};
      |                                     ^~~~
multidict/_multidict.c:922:37: note: (near initialization for ‘_parser.initialized’)
multidict/_multidict.c:922:43: error: initialization of ‘const char *’ from incompatible pointer type ‘const char * const*’ [-Wincompatible-pointer-types]
  922 |     static _PyArg_Parser _parser = {NULL, _keywords, "pop", 0};
      |                                           ^~~~~~~~~
multidict/_multidict.c:922:43: note: (near initialization for ‘_parser.format’)
multidict/_multidict.c:922:54: error: initialization of ‘const char * const*’ from incompatible pointer type ‘char *’ [-Wincompatible-pointer-types]
  922 |     static _PyArg_Parser _parser = {NULL, _keywords, "pop", 0};
      |                                                      ^~~~~
multidict/_multidict.c:922:54: note: (near initialization for ‘_parser.keywords’)
multidict/_multidict.c: In function ‘multidict_popall’:
multidict/_multidict.c:970:37: error: initialization of ‘int’ from ‘void *’ makes integer from pointer without a cast [-Wint-conversion]
  970 |     static _PyArg_Parser _parser = {NULL, _keywords, "popall", 0};
      |                                     ^~~~
multidict/_multidict.c:970:37: note: (near initialization for ‘_parser.initialized’)
multidict/_multidict.c:970:43: error: initialization of ‘const char *’ from incompatible pointer type ‘const char * const*’ [-Wincompatible-pointer-types]
  970 |     static _PyArg_Parser _parser = {NULL, _keywords, "popall", 0};
      |                                           ^~~~~~~~~
multidict/_multidict.c:970:43: note: (near initialization for ‘_parser.format’)
multidict/_multidict.c:970:54: error: initialization of ‘const char * const*’ from incompatible pointer type ‘char *’ [-Wincompatible-pointer-types]
  970 |     static _PyArg_Parser _parser = {NULL, _keywords, "popall", 0};
      |                                                      ^~~~~~~~
multidict/_multidict.c:970:54: note: (near initialization for ‘_parser.keywords’)
error: command '/usr/bin/gcc' failed with exit code 1

Steps to reproduce

Install the latest rawhide.

python3 -m venv /tmp/venv
/tmp/venv/bin/pip install multidict

Your environment

Fedora-Cloud-Base-Rawhide-20240117.n.0.x86_64.qcow2
gcc-14.0.1-0.1.fc40.x86_64

[webknjaz]

Thanks for the report! Could you confirm this against master?

I don't know what GCC version is used in CI but it doesn't error out.

P.S. There's another issue tracking the release of wheels for py312.

[psss]

Yes, I'm seeing the same error with 68b080e.

[hroncok]

_parser.initialized is an int. I suppose replacing all the NULLs with zeros will get rid of the -Wint-conversion one

[hroncok]

I would also guess that passing _keywords as the second value seems bogus, because the second value is format, not keywords...

Oh. Since Python 3.12, the first int value was added. See python/cpython@6f6a4e6 -- hence the entire order of values here needs to be shifted.

[hroncok]

Perhaps initializing like this makes more sense?

multidict/multidict/_multidict.c

Lines 865 to 869 in 68b080e

	static _PyArg_Parser _parser = {
	.keywords = _keywords,
	.fname = "add",
	.kwtuple = NULL,
	};

[hroncok]

The bit I don't understand is this: How did this ever work on Python 3.12 in the first place?

[charles-dyfis-net]

This isn't limited to gcc-14; the same occurs building against Python 2.13 3.12 with clang 16.0.6 on MacOS.

[charles-dyfis-net]

One way to fix this, albeit with a performance penalty, is to extend the PR dropping use of METH_FASTCALL from Python 3.13 to drop it from 3.12 as well; I have a branch doing this as charles-dyfis-net/multidict@ad11386

[webknjaz]

Feel free to submit a PR, with maybe some tests that would show that it works as expected?
I didn't write that original code, so I don't fully understand all the inner workings. Hopefully, the partial revert I made @ #909 is correct. At least the CI passes (and it made the 3.13 jobs green).

Any idea why that is? Does the older GCC compile things differently? After all, the wheels being tested are built in the manylinux images with a GCC version that's hardly modern. The tests (except for the 3.13 job) don't recompile the project but install those wheels for testing.

[jonaslb]

Does the older GCC compile things differently?

On gcc 13 / python 3.12, I see the same compiler messages, but they are only warnings, not errors.

[webknjaz]

That's weird. I may consider #929, but would like to understand why or how this works right now. As @hroncok mentioned — this is a mystery, and I'd rather not make sudden mystery-driven hotfixing without understanding fully what's going now.

Any ideas?

P.S. @charles-dyfis-net @psss is this problematic under CPython 3.9 with clang 16.0.6 / GCC 14?

[jonaslb]

One observation is that these compiler warnings/errors are about kwargs parsing, but (from a quick glance) the multidict methods such as getall are only ever called using positional arguments in the tests, and then in the c code, the args are only used as eg. args[0]. In other words, I'm thinking that _PyArg_UnpackKeywords doesn't do anything because kwargs is empty, so we never see that the parser struct containing expected kwargs is somehow wrongly constructed. Probably, to trigger a bug from this, actually using kwargs would be required. So something like d.getall(default="abc", key="xyz").

[webknjaz]

@jonaslb any chance you could verify that guess?

I think we need test coverage for such cases before the release.

I was actually trying to come up with a way to measure coverage in the C code using gcovr, but wasn't successful — it just shows zero coverage. I can't figure out the compilation env vars to make it work..

If anybody has example repos implementing this (without setup.py build_ext --inplace), plz share the links/insights..

[jonaslb]

Yeah it's verified, I get a segmentation fault when I run the test using kwargs (python 3.12/gcc13, but I think it's not really about gcc but about the change in Python linked further up).

I have a branch where I used the named member syntax fix proposed in #926 (comment) throughout _multidict.c. This seems to fix the issue - tests pass and there's no segmentation fault. It "should" also work on older Pythons (but I haven't tested that here).

I can send a PR for it, if you prefer it. I should say though that I think that #929 seems a very reasonable solution, especially given that _PyArg_Parser is not stable ABI and was even removed in 3.13.

[hroncok]

See also this (using the Fedora RPM package built with GCC 13):

Python 3.12.1 (main, Dec 18 2023, 00:00:00) [GCC 13.2.1 20231205 (Red Hat 13.2.1-6)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import multidict
>>> m = multidict.MultiDict()
>>> m.getall()
Segmentation fault (core dumped)

[hoodmane]

Pyodide is also affected by this.

[webknjaz]

@jonaslb I'm going to accept #929 once it's polished, but if you're up for it, I'd love that PR with the named member syntax too. Thanks in advance!

[tales-aparecida]

Thanks for fixing this! ❤️

Praying for the 6.0.5 release pipeline to pass, you had some weird errors in there 😬

[webknjaz]

@tales-aparecida feel free to watch https://github.com/aio-libs/multidict/actions/runs/7746701945 now.

[hoodmane]

Thanks @webknjaz!

[psss]

Thanks for the quick fix!