Compiler build paths and related environment variables are ignored for native builds

[AlexGrund]

BPO	38549
Nosy	@vstinner

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-10-21.14:21:13.761>
labels = ['3.8', 'build', '3.7', '3.9']
title = 'Compiler build paths and related environment variables are ignored for native builds'
updated_at = <Date 2019-10-21.14:35:20.179>
user = 'https://bugs.python.org/AlexGrund'

bugs.python.org fields:

activity = <Date 2019-10-21.14:35:20.179>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Build']
creation = <Date 2019-10-21.14:21:13.761>
creator = 'Alex Grund'
dependencies = []
files = []
hgrepos = []
issue_num = 38549
keywords = []
message_count = 3.0
messages = ['355073', '355075', '355076']
nosy_count = 2.0
nosy_names = ['vstinner', 'Alex Grund']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue38549'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6', 'Python 3.7', 'Python 3.8', 'Python 3.9']

[AlexGrund]

In e.g. Linux users can set CPATH and LIBRARY_PATH to a list of paths considered by e.g. GCC, Clang, ... as if they were passed to -I, -L

These paths show up in the verbose compiler output too.

However in native builds (not cross-compiling) these variables/directories are ignored which leads to failure of e.g. building the BZip2 extension (among others).

Use case: On HPC modules are used, which means e.g. BZip2 is not installed into a system directory but into some other folder and CPATH, LIBRARY_PATH, LD_LIBRARY_PATH, ... are modified so it can be found by compiler and runtime. However the Python build script does not find it as it ignores those "compiler paths".

This did work ages ago, but was broken when it was decided that compiler paths (output of gcc -E -v) is no longer used for native builds and that function was later even renamed: 32f5fdd#diff-2eeaed663bd0d25b7e608891384b7298R514

I propose to rename the function back to its original name and use it always.

[vstinner]

This did work ages ago, but was broken when it was decided that compiler paths (output of gcc -E -v) is no longer used for native builds and that function was later even renamed: 32f5fdd#diff-2eeaed663bd0d25b7e608891384b7298R514

Can you please elaborate?

setup.py of Python 2.7:

        if cross_compiling:
            self.add_gcc_paths()

setup.py of Python 3.5:

        # only change this for cross builds for 3.3, issues on Mageia
        if cross_compiling:
            self.add_gcc_paths()

In the 2.7, this code was changed in 2013 to fix bpo-17990:

"For bpo-17086, 8ee6d96a1019 backported some cross-build patches to the 2.7 branch. The changes to setup.py detect_modules differ in the backport from those in default. In particular, in default, changes to the library and include directory lists used to build extensions modules are only made conditionally when cross-compiling. But the 2.7 backport makes these changes unconditionally."

commit 6166709
Author: Ned Deily <nad@acm.org>
Date: Wed May 15 18:00:45 2013 -0700

Issue bpo-17990: Only modify include and library search paths when cross-compiling.

[vstinner]

However in native builds (not cross-compiling) these variables/directories are ignored which leads to failure of e.g. building the BZip2 extension (among others).

Python does not explicitly ignore these environment variable, so I'm not sure how setup.py prevent that.

For example, the _bz2 module is built by setup.py by default in the master branch of Python.

[mkoeppe]

However in native builds (not cross-compiling) these variables/directories are ignored which leads to failure of e.g. building the BZip2 extension (among others).

Python does not explicitly ignore these environment variable, so I'm not sure how setup.py prevent that.

Simply because setup.py tries to discover header files via direct file system lookup (find_file) instead of running the compiler, which would respect these environment variables.

[mkoeppe]

Previously reported in #65770.