Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove pyconfig.h header parsing #1521

Merged
merged 2 commits into from
Mar 28, 2021
Merged

Remove pyconfig.h header parsing #1521

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
80c747d
Remove `pyconfig.h` header parsing
ravenexp Mar 26, 2021
eca20fe
Merge branch 'main' into remove-header-parsing
kngwyu Mar 27, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
- `PyObject_Check`, `PySuper_Check`, and `FreeFunc` [#1438](https://github.com/PyO3/pyo3/pull/1438)
- Remove pyclass implementation details `Type`, `DESCRIPTION`, and `FLAGS` from `PyTypeInfo`. [#1456](https://github.com/PyO3/pyo3/pull/1456)
- Remove `__doc__` from module's `__all__`. [#1509](https://github.com/PyO3/pyo3/pull/1509)
- Remove `PYO3_CROSS_INCLUDE_DIR` environment variable and the associated C header parsing functionality.

### Fixed
- Remove FFI definition `PyCFunction_ClearFreeList` for Python 3.9 and later. [#1425](https://github.com/PyO3/pyo3/pull/1425)
Expand Down
74 changes: 5 additions & 69 deletions build.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@ use std::{
collections::{HashMap, HashSet},
convert::AsRef,
env,
fs::{self, DirEntry, File},
io::{self, BufRead, BufReader},
fs::{self, DirEntry},
io,
path::{Path, PathBuf},
process::{Command, Stdio},
str::FromStr,
Expand Down Expand Up @@ -109,24 +109,15 @@ impl GetPrimitive for HashMap<String, String> {

struct CrossCompileConfig {
lib_dir: PathBuf,
include_dir: Option<PathBuf>,
version: Option<String>,
os: String,
arch: String,
}

impl CrossCompileConfig {
fn both() -> Result<Self> {
Ok(CrossCompileConfig {
include_dir: env::var_os("PYO3_CROSS_INCLUDE_DIR").map(Into::into),
..CrossCompileConfig::lib_only()?
})
}

fn lib_only() -> Result<Self> {
fn new() -> Result<Self> {
Ok(CrossCompileConfig {
lib_dir: CrossCompileConfig::validate_variable("PYO3_CROSS_LIB_DIR")?,
include_dir: None,
os: env::var("CARGO_CFG_TARGET_OS").unwrap(),
arch: env::var("CARGO_CFG_TARGET_ARCH").unwrap(),
version: env::var_os("PYO3_CROSS_PYTHON_VERSION").map(|s| s.into_string().unwrap()),
Expand Down Expand Up @@ -183,13 +174,8 @@ fn cross_compiling() -> Result<Option<CrossCompileConfig>> {
return Ok(None);
}

if env::var("CARGO_CFG_TARGET_FAMILY")? == "windows" {
// Windows cross-compile uses both header includes and sysconfig
return Ok(Some(CrossCompileConfig::both()?));
}

// Cross-compiling on any other platform
Ok(Some(CrossCompileConfig::lib_only()?))
Ok(Some(CrossCompileConfig::new()?))
}

/// A list of python interpreter compile-time preprocessor defines that
Expand Down Expand Up @@ -300,23 +286,6 @@ impl BuildFlags {
}
}

/// Attempts to parse the header at the given path, returning a map of definitions to their values.
/// Each entry in the map directly corresponds to a `#define` in the given header.
fn parse_header_defines(header_path: impl AsRef<Path>) -> Result<HashMap<String, String>> {
let header_reader = BufReader::new(File::open(header_path.as_ref())?);
let mut definitions = HashMap::new();
for maybe_line in header_reader.lines() {
let line = maybe_line?;
let mut i = line.trim().split_whitespace();
if i.next() == Some("#define") {
if let (Some(key), Some(value), None) = (i.next(), i.next(), i.next()) {
definitions.insert(key.into(), value.into());
}
}
}
Ok(definitions)
}

fn parse_script_output(output: &str) -> HashMap<String, String> {
output
.lines()
Expand Down Expand Up @@ -500,36 +469,6 @@ fn load_cross_compile_from_sysconfigdata(
Ok((interpreter_config, build_flags))
}

fn load_cross_compile_from_headers(
cross_compile_config: CrossCompileConfig,
) -> Result<(InterpreterConfig, BuildFlags)> {
let python_include_dir = cross_compile_config.include_dir.unwrap();
let python_include_dir = Path::new(&python_include_dir);
let patchlevel_defines = parse_header_defines(python_include_dir.join("patchlevel.h"))?;

let major = patchlevel_defines.get_numeric("PY_MAJOR_VERSION")?;
let minor = patchlevel_defines.get_numeric("PY_MINOR_VERSION")?;

let python_version = PythonVersion { major, minor };

let config_data = parse_header_defines(python_include_dir.join("pyconfig.h"))?;

let interpreter_config = InterpreterConfig {
version: python_version,
libdir: cross_compile_config.lib_dir.to_str().map(String::from),
shared: config_data.get_bool("Py_ENABLE_SHARED").unwrap_or(false),
ld_version: format!("{}.{}", major, minor),
base_prefix: "".to_string(),
executable: PathBuf::new(),
calcsize_pointer: None,
implementation: PythonInterpreterKind::CPython,
};

let build_flags = BuildFlags::from_config_map(&config_data);

Ok((interpreter_config, build_flags))
}

fn windows_hardcoded_cross_compile(
cross_compile_config: CrossCompileConfig,
) -> Result<(InterpreterConfig, BuildFlags)> {
Expand All @@ -549,7 +488,7 @@ fn windows_hardcoded_cross_compile(
} else if let Some(minor_version) = get_abi3_minor_version() {
(3, minor_version)
} else {
bail!("One of PYO3_CROSS_INCLUDE_DIR, PYO3_CROSS_PYTHON_VERSION, or an abi3-py3* feature must be specified when cross-compiling for Windows.")
bail!("PYO3_CROSS_PYTHON_VERSION or an abi3-py3* feature must be specified when cross-compiling for Windows.")
};

let python_version = PythonVersion { major, minor };
Expand All @@ -576,9 +515,6 @@ fn load_cross_compile_info(
if target_family == "unix" {
// Configure for unix platforms using the sysconfigdata file
load_cross_compile_from_sysconfigdata(cross_compile_config)
} else if cross_compile_config.include_dir.is_some() {
// Must configure by headers on windows platform
load_cross_compile_from_headers(cross_compile_config)
} else {
windows_hardcoded_cross_compile(cross_compile_config)
}
Expand Down
16 changes: 7 additions & 9 deletions guide/src/building_and_distribution.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,17 +84,13 @@ Cross compiling PyO3 modules is relatively straightforward and requires a few pi
* The appropriate options in your Cargo `.config` for the platform you're targeting and the toolchain you are using.
* A Python interpreter that's already been compiled for your target.
* A Python interpreter that is built for your host and available through the `PATH` or setting the [`PYO3_PYTHON`](#python-version) variable.
* The headers that match the above interpreter.

See https://github.com/japaric/rust-cross for a primer on cross compiling Rust in general.
See [github.com/japaric/rust-cross](https://github.com/japaric/rust-cross) for a primer on cross compiling Rust in general.

After you've obtained the above, you can build a cross compiled PyO3 module by setting a few extra environment variables:

* `PYO3_CROSS_LIB_DIR`: This variable must be set to the directory containing the target's libpython DSO and the associated `_sysconfigdata*.py` file.
* `PYO3_CROSS_PYTHON_VERSION`: Major and minor version (e.g. 3.9) of the target Python installation. This variable is only needed if pyo3 cannot determine the version to target by other means:
- From `PYO3_CROSS_INCLUDE_DIR` or abi3-py3* features when targeting Windows, or
- if there are multiple versions of python present in `PYO3_CROSS_LIB_DIR` when targeting unix.
* `PYO3_CROSS_INCLUDE_DIR`: This variable can optionally be set to the directory containing the headers for the target's Python interpreter when targeting Windows.
* `PYO3_CROSS_LIB_DIR`: This variable must be set to the directory containing the target's libpython DSO and the associated `_sysconfigdata*.py` file for Unix-like targets, or the Python DLL import libraries for the Windows target.
* `PYO3_CROSS_PYTHON_VERSION`: Major and minor version (e.g. 3.9) of the target Python installation. This variable is only needed if PyO3 cannot determine the version to target from `abi3-py3*` features, or if there are multiple versions of Python present in `PYO3_CROSS_LIB_DIR`.

An example might look like the following (assuming your target's sysroot is at `/home/pyo3/cross/sysroot` and that your target is `armv7`):

Expand All @@ -112,14 +108,16 @@ export PYO3_CROSS_LIB_DIR="/home/pyo3/cross/sysroot/usr/lib"
cargo build --target armv7-unknown-linux-gnueabihf
```

Or another example with the same sys root but building for windows:
Or another example with the same sys root but building for Windows:
```sh
export PYO3_CROSS_INCLUDE_DIR="/home/pyo3/cross/sysroot/usr/include"
export PYO3_CROSS_PYTHON_VERSION=3.9
export PYO3_CROSS_LIB_DIR="/home/pyo3/cross/sysroot/usr/lib"

cargo build --target x86_64-pc-windows-gnu
```

Any of the `abi3-py3*` features can be enabled instead of setting `PYO3_CROSS_PYTHON_VERSION` in the above examples.

## Bazel

For an example of how to build python extensions using Bazel, see https://github.com/TheButlah/rules_pyo3
Expand Down