Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include Python development headers #530

Open
Jackenmen opened this issue Mar 25, 2023 · 2 comments
Open

Include Python development headers #530

Jackenmen opened this issue Mar 25, 2023 · 2 comments
Labels
bug Something isn't working

Comments

@Jackenmen
Copy link

Jackenmen commented Mar 25, 2023

Describe the Enhancement

Currently, the buildpack does not contain Python development headers which makes it hard to install any Python packages implemented in C that don't provide wheels for the platform. One such package is psycopg2 which purposefully doesn't provide source distribution to allow usage of system's own libraries on production systems.

In order to see the problem, you can run these commands:

git clone https://github.com/paketo-buildpacks/samples
cd samples/python/pip
echo "psycopg2==2.9.5" >> requirements.txt
pack build pip-sample --buildpack paketo-buildpacks/python --builder paketobuildpacks/builder:full

which results in an error about missing Python headers:

      ./psycopg/psycopg.h:35:10: fatal error: Python.h: No such file or directory
       #include <Python.h>
                ^~~~~~~~~~
      compilation terminated.

Possible Solution

Include development headers in appropriate environments (potential candidates: CPATH, CPPPATH, PKG_CONFIG_PATH, INCLUDE_PATH) or symlink them into one of the default include directories (not sure that will work properly).

Motivation

The lack of development headers makes it hard to install any Python packages implemented in C (C extensions) that don't provide wheels for the platform. One such package is psycopg2 which purposefully doesn't provide source distribution to allow usage of system's own libraries on production systems.

@robdimsdale
Copy link
Member

@Jackenmen thanks for reporting this. I agree it's an issue. Thank you for provide a clear, simple reproduction. I was able to reproduce the issue quickly.

I'll prioritize this for us to fix, and keep you updated with our progress.


For maintainers' reference, the header files are present on the resultant image, so the pre-compiled CPython dependency is probably fine. It's more likely that we need to set/append an environment variable to point to the correct location (as suggested above).

git clone https://github.com/paketo-buildpacks/samples
cd samples/python/pip
# no modifications to the requirements.txt file
pack build pip-sample --buildpack paketo-buildpacks/python --builder paketobuildpacks/builder:full
docker run --rm -it --entrypoint launcher pip-sample ls -alh /layers/paketo-buildpacks_cpython/cpython/include/python3.10/Python.h
-rw-r--r-- 1 cnb cnb 3.2K Jan  1  1980 /layers/paketo-buildpacks_cpython/cpython/include/python3.10/Python.h

Also for reference, this is the run-time environment using the same app image as above. It might be slightly different from the build-time environment, but should be a useful starting-off point.

docker run --rm -it --entrypoint launcher pip-sample env
LD_LIBRARY_PATH=/layers/paketo-buildpacks_pip-install/packages/lib:/layers/paketo-buildpacks_cpython/cpython/lib
HOSTNAME=00e87b622780
OLDPWD=/workspace
PWD=/workspace
HOME=/home/cnb
PYTHONPYCACHEPREFIX=/home/cnb/.pycache
TERM=xterm
SHLVL=1
PYTHONPATH=/layers/paketo-buildpacks_pip-install/packages/lib/python3.10/site-packages:/layers/paketo-buildpacks_cpython/cpython
PATH=/layers/paketo-buildpacks_pip-install/packages/bin:/layers/paketo-buildpacks_cpython/cpython/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
_=/usr/bin/env

@robdimsdale robdimsdale added the bug Something isn't working label Mar 28, 2023
@github-project-automation github-project-automation bot moved this to ❓Not scoped in Paketo Workstreams Mar 28, 2023
@robdimsdale robdimsdale moved this from ❓Not scoped to 📝 Todo in Paketo Workstreams Mar 28, 2023
@edmorley
Copy link

edmorley commented May 22, 2023

Btw this is likely due to pypa/setuptools#3657 which was a setuptools regression in header autodetection, which first appeared in setuptools 65.2.0 and was fixed in setuptools 67.2.0. (Though I haven't tested the Paketo Python CNB to confirm.)

The Heroku Python CNB currently explicitly sets CPATH to work around the issue (though it's less important now that the issue has been fixed in setuptools):
https://github.com/heroku/buildpacks-python/blob/e618353b041f00c4b57201ecfbcfd9b69d3fcc58/src/layers/python.rs#L270-L283

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: 📝 Todo
Development

No branches or pull requests

3 participants