[BUG] bundled distutils produces incorrect paths to headers with relocatable pythons

[asottile-sentry]

setuptools version

65.5.0

Python version

3.11.0

OS

macos

Additional environment information

the pythons can be downloaded from here if you'd like to reproduce exactly:

arm64: https://storage.googleapis.com/sentry-dev-infra-assets/prebuilt-pythons/python-3.11.0+0-macosx_12_0_arm64.tgz
x86_64: https://storage.googleapis.com/sentry-dev-infra-assets/prebuilt-pythons/python-3.11.0+0-macosx_12_0_x86_64.tgz

the pythons are built from https://github.com/getsentry/prebuilt-pythons

Description

building a trivial C extension fails as it looks for the headers at the original path that the binaries were built at rather than the path after installation. the stdlib implementation of distutils seems to handle this correctly

# setup.py
from setuptools import setup
from setuptools import Extension


setup(
    name='hello-lib',
    version='1',
    ext_modules=[Extension('_hello', ['_hello.c'])],
)

/* _hello.c */
#include <Python.h>

static PyObject* _hello_world(PyObject* self) {
    return PyUnicode_FromString("hello world");  // python c api
}

static struct PyMethodDef methods[] = {
    {"hello_world", (PyCFunction)_hello_world, METH_NOARGS},
    {NULL, NULL}
};

static struct PyModuleDef module = {
    PyModuleDef_HEAD_INIT,
    "_hello",
    NULL,
    -1,
    methods
};

PyMODINIT_FUNC PyInit__hello(void) {
    return PyModule_Create(&module);
}

Expected behavior

I expect them to be able to build and identify the prefix from the current executable location -- for example using SETUPTOOLS_USE_DISTUTILS=stdlib:

$ SETUPTOOLS_USE_DISTUTILS=stdlib ./venv311/bin/pip wheel .
Processing /private/tmp/y
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: hello-lib
  Building wheel for hello-lib (setup.py) ... done
  Created wheel for hello-lib: filename=hello_lib-1-cp311-cp311-macosx_12_0_arm64.whl size=2690 sha256=6ff81f5734d30f9c0f99bc341806e16f9d870135971f81aef068409ffccf13b9
  Stored in directory: /private/var/folders/1c/5r_6j4rs0j56vnt8kq83tmvh0000gn/T/pip-ephem-wheel-cache-d8pr4u8_/wheels/b5/10/da/93caa535f3d99a26b32fc168ccfc527888b41536b9e286bf4f
Successfully built hello-lib

How to Reproduce

note that this script has the url above for arm64 -- substitute for x86_64 instead

#!/usr/bin/env bash
set -euxo pipefail

rm -rf work
mkdir work
cd work

curl --silent --location https://storage.googleapis.com/sentry-dev-infra-assets/prebuilt-pythons/python-3.11.0+0-macosx_12_0_arm64.tgz |
   tar -xf -

*/bin/python3 -m venv venv
venv/bin/pip install setuptools wheel --upgrade

cat > setup.py <<EOF
from setuptools import setup
from setuptools import Extension


setup(
    name='hello-lib',
    version='1',
    ext_modules=[Extension('_hello', ['_hello.c'])],
)
EOF

cat > _hello.c << EOF
#include <Python.h>

static PyObject* _hello_world(PyObject* self) {
    return PyUnicode_FromString("hello world");  // python c api
}

static struct PyMethodDef methods[] = {
    {"hello_world", (PyCFunction)_hello_world, METH_NOARGS},
    {NULL, NULL}
};

static struct PyModuleDef module = {
    PyModuleDef_HEAD_INIT,
    "_hello",
    NULL,
    -1,
    methods
};

PyMODINIT_FUNC PyInit__hello(void) {
    return PyModule_Create(&module);
}
EOF

venv/bin/pip wheel .

Output

$ SETUPTOOLS_USE_DISTUTILS=local bash t.sh 
+ rm -rf work
+ mkdir work
+ cd work
+ curl --silent --location https://storage.googleapis.com/sentry-dev-infra-assets/prebuilt-pythons/python-3.11.0+0-macosx_12_0_arm64.tgz
+ tar -xf -
+ python-3.11.0+0-macosx_12_0_arm64/bin/python3 -m venv venv
+ venv/bin/pip install setuptools wheel --upgrade
Requirement already satisfied: setuptools in ./venv/lib/python3.11/site-packages (65.5.0)
Collecting wheel
  Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
Installing collected packages: wheel
Successfully installed wheel-0.37.1
+ cat
+ cat
+ venv/bin/pip wheel .
Processing /private/tmp/y/y/work
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: hello-lib
  Building wheel for hello-lib (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [12 lines of output]
      running bdist_wheel
      running build
      running build_ext
      building '_hello' extension
      creating build
      creating build/temp.macosx-12.5-arm64-cpython-311
      gcc -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/private/tmp/y/y/work/venv/include -I/var/folders/76/zy5ktkns50v6gt5g8r0sf6sc0000gn/T/tmp7csdie6a/prefix/include/python3.11 -c _hello.c -o build/temp.macosx-12.5-arm64-cpython-311/_hello.o
      _hello.c:1:10: fatal error: 'Python.h' file not found
      #include <Python.h>
               ^~~~~~~~~~
      1 error generated.
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for hello-lib
  Running setup.py clean for hello-lib
Failed to build hello-lib
ERROR: Failed to build one or more wheels

[roblg]

I think we're experiencing the same error, and by bisection based on https://setuptools.pypa.io/en/latest/history.html, it might have been introduced in #3553 (setuptools v65.2.0)

# correct
$ /platform/python3/bin/python3 /platform/python3/bin/pip3 install setuptools==65.0.0
<success>
$ /platform/python3/bin/python3 -c "from distutils import sysconfig; print(sysconfig.get_python_inc())"
/platform/python3/include/python3.7m

# correct 
$ /platform/python3/bin/python3 /platform/python3/bin/pip3 install setuptools==65.1.1
<success>
$ /platform/python3/bin/python3 -c "from distutils import sysconfig; print(sysconfig.get_python_inc())"
/platform/python3/include/python3.7m

# INCORRECT (or at least differs in a way that breaks pip compilation of subsequent native modules)
$ /platform/python3/bin/python3 /platform/python3/bin/pip3 install setuptools==65.2.0
<success>
$ /platform/python3/bin/python3 -c "from distutils import sysconfig; print(sysconfig.get_python_inc())"
/root/python-3.7.2/include/python3.7m

note: /platform --> /root in 65.2.0

[edmorley]

We encountered this breaking change too, when upgrading from setuptools 65.1.1 to 65.2.0.

It appears to have been caused by:
pypa/distutils#173

Which is under discussion at:
pypa/distutils#178

[ice-tong]

I meet the same issue, what is the solution~ 🤣

[edmorley]

This is fixed for me now (at least with my configuration) in setuptools 67.2.0, thanks to #3809 which pulled in the fix from distutils (pypa/distutils/pull/200).

[asottile-sentry]

indeed, the script above now succeeds