diff --git a/guide/src/SUMMARY.md b/guide/src/SUMMARY.md index 0227dbd4313..2198c3792e9 100644 --- a/guide/src/SUMMARY.md +++ b/guide/src/SUMMARY.md @@ -4,35 +4,37 @@ --- -- [Getting Started](getting_started.md) -- [Python Modules](module.md) -- [Python Functions](function.md) -- [Python Classes](class.md) +- [Getting started](getting_started.md) +- [Python modules](module.md) +- [Python functions](function.md) + - [Function signatures](function/signature.md) + - [Error handling](function/error_handling.md) +- [Python classes](class.md) - [Class customizations](class/protocols.md) - [Basic object customization](class/object.md) - [Emulating numeric types](class/numeric.md) - [Emulating callable objects](class/call.md) -- [Type Conversions](conversions.md) +- [Type conversions](conversions.md) - [Mapping of Rust types to Python types](conversions/tables.md)] - [Conversion traits](conversions/traits.md)] -- [Python Exceptions](exception.md) +- [Python exceptions](exception.md) - [Calling Python from Rust](python_from_rust.md) - [GIL, mutability and object types](types.md) - [Parallelism](parallelism.md) - [Debugging](debugging.md) -- [Features Reference](features.md) -- [Memory Management](memory.md) -- [Advanced Topics](advanced.md) -- [Building and Distribution](building_and_distribution.md) +- [Features reference](features.md) +- [Memory management](memory.md) +- [Advanced topics](advanced.md) +- [Building and distribution](building_and_distribution.md) - [Supporting multiple Python versions](building_and_distribution/multiple_python_versions.md) -- [Useful Crates](ecosystem.md) +- [Useful crates](ecosystem.md) - [Logging](ecosystem/logging.md) - - [Async / Await](ecosystem/async-await.md) -- [FAQ & Troubleshooting](faq.md) + - [Using `async` and `await`](ecosystem/async-await.md) +- [FAQ and troubleshooting](faq.md) --- -[Appendix A: Migration Guide](migration.md) +[Appendix A: Migration guide](migration.md) [Appendix B: PyO3 and rust-cpython](rust_cpython.md) [Appendix C: Trait bounds](trait_bounds.md) [Appendix D: Python typing hints](python_typing_hints.md) diff --git a/guide/src/advanced.md b/guide/src/advanced.md index 327790d47a7..8264c14dd17 100644 --- a/guide/src/advanced.md +++ b/guide/src/advanced.md @@ -6,7 +6,7 @@ PyO3 exposes much of Python's C API through the `ffi` module. The C API is naturally unsafe and requires you to manage reference counts, errors and specific invariants yourself. Please refer to the [C API Reference Manual](https://docs.python.org/3/c-api/) and [The Rustonomicon](https://doc.rust-lang.org/nightly/nomicon/ffi.html) before using any function from that API. -## Memory Management +## Memory management PyO3's `&PyAny` "owned references" and `Py` smart pointers are used to access memory stored in Python's heap. This memory sometimes lives for longer diff --git a/guide/src/building_and_distribution.md b/guide/src/building_and_distribution.md index e59525b2158..2c39f2dbd82 100644 --- a/guide/src/building_and_distribution.md +++ b/guide/src/building_and_distribution.md @@ -1,4 +1,4 @@ -# Building and Distribution +# Building and distribution This chapter of the guide goes into detail on how to build and distribute projects using PyO3. The way to achieve this is very different depending on whether the project is a Python module implemented in Rust, or a Rust binary embedding Python. For both types of project there are also common problems such as the Python version to build for and the [linker](https://en.wikipedia.org/wiki/Linker_(computing)) arguments to use. diff --git a/guide/src/class.md b/guide/src/class.md index a9b026fc3b1..58400267862 100644 --- a/guide/src/class.md +++ b/guide/src/class.md @@ -1,4 +1,4 @@ -# Python Classes +# Python classes PyO3 exposes a group of attributes powered by Rust's proc macro system for defining Python classes as Rust structs. @@ -614,29 +614,10 @@ impl MyClass { ## Method arguments -By default, PyO3 uses function signatures to determine which arguments are required. Then it scans -the incoming `args` and `kwargs` parameters. If it can not find all required -parameters, it raises a `TypeError` exception. It is possible to override the default behavior -with the `#[args(...)]` attribute. This attribute accepts a comma separated list of parameters in -the form of `attr_name="default value"`. Each parameter has to match the method parameter by name. - -Each parameter can be one of the following types: - - * `"/"`: positional-only arguments separator, each parameter defined before `"/"` is a - positional-only parameter. - Corresponds to python's `def meth(arg1, arg2, ..., /, argN..)`. - * `"*"`: var arguments separator, each parameter defined after `"*"` is a keyword-only parameter. - Corresponds to python's `def meth(*, arg1.., arg2=..)`. - * `args="*"`: "args" is var args, corresponds to Python's `def meth(*args)`. Type of the `args` - parameter has to be `&PyTuple`. - * `kwargs="**"`: "kwargs" receives keyword arguments, corresponds to Python's `def meth(**kwargs)`. - The type of the `kwargs` parameter has to be `Option<&PyDict>`. - * `arg="Value"`: arguments with default value. Corresponds to Python's `def meth(arg=Value)`. - If the `arg` argument is defined after var arguments, it is treated as a keyword-only argument. - Note that `Value` has to be valid rust code, PyO3 just inserts it into the generated - code unmodified. - -Example: +Similar to `#[pyfunction]`, the `#[args]` attribute can be used to specify the way that `#[pymethods]` accept arguments. Consult the documentation for [`function signatures`](./function/signature.md) to see the parameters this attribute accepts. + +The following example defines a class `MyClass` with a method `method`. This method has an `#[args]` attribute which sets default values for `num` and `name`, and indicates that `py_args` should collect all extra positional arguments and `py_kwargs` all extra keyword arguments: + ```rust # use pyo3::prelude::*; use pyo3::types::{PyDict, PyTuple}; @@ -665,35 +646,26 @@ impl MyClass { name: &str, py_args: &PyTuple, py_kwargs: Option<&PyDict>, - ) -> PyResult { - self.num = num; - Ok(format!( - "py_args={:?}, py_kwargs={:?}, name={}, num={}", - py_args, py_kwargs, name, self.num - )) - } - - fn make_change(&mut self, num: i32) -> PyResult { + ) -> String { + let num_before = self.num; self.num = num; - Ok(format!("num={}", self.num)) + format!( + "py_args={:?}, py_kwargs={:?}, name={}, num={} num_before={}", + py_args, py_kwargs, name, self.num, num_before, + ) } } ``` -N.B. the position of the `"/"` and `"*"` arguments (if included) control the system of handling positional and keyword arguments. In Python: -```python -import mymodule -mc = mymodule.MyClass() -print(mc.method(44, False, "World", 666, x=44, y=55)) -print(mc.method(num=-1, name="World")) -print(mc.make_change(44, False)) -``` -Produces output: -```text -py_args=('World', 666), py_kwargs=Some({'x': 44, 'y': 55}), name=Hello, num=44 -py_args=(), py_kwargs=None, name=World, num=-1 -num=44 -num=-1 +In Python this might be used like: + +```python +>>> import mymodule +>>> mc = mymodule.MyClass() +>>> print(mc.method(44, False, "World", 666, x=44, y=55)) +py_args=('World', 666), py_kwargs=Some({'x': 44, 'y': 55}), name=Hello, num=44, num_before=-1 +>>> print(mc.method(num=-1, name="World")) +py_args=(), py_kwargs=None, name=World, num=-1, num_before=44 ``` ## Making class method signatures available to Python @@ -951,8 +923,8 @@ struct MyClass { # #[allow(dead_code)] num: i32, } -unsafe impl ::pyo3::type_object::PyTypeInfo for MyClass { - type AsRefTarget = ::pyo3::PyCell; +unsafe impl pyo3::type_object::PyTypeInfo for MyClass { + type AsRefTarget = pyo3::PyCell; const NAME: &'static str = "MyClass"; const MODULE: ::std::option::Option<&'static str> = ::std::option::Option::None; #[inline] @@ -963,27 +935,27 @@ unsafe impl ::pyo3::type_object::PyTypeInfo for MyClass { } } -impl ::pyo3::PyClass for MyClass { +impl pyo3::PyClass for MyClass { type Frozen = pyo3::pyclass::boolean_struct::False; } -impl<'a, 'py> ::pyo3::impl_::extract_argument::PyFunctionArgument<'a, 'py> for &'a MyClass +impl<'a, 'py> pyo3::impl_::extract_argument::PyFunctionArgument<'a, 'py> for &'a MyClass { - type Holder = ::std::option::Option<::pyo3::PyRef<'py, MyClass>>; + type Holder = ::std::option::Option>; #[inline] - fn extract(obj: &'py ::pyo3::PyAny, holder: &'a mut Self::Holder) -> ::pyo3::PyResult { - ::pyo3::impl_::extract_argument::extract_pyclass_ref(obj, holder) + fn extract(obj: &'py pyo3::PyAny, holder: &'a mut Self::Holder) -> pyo3::PyResult { + pyo3::impl_::extract_argument::extract_pyclass_ref(obj, holder) } } -impl<'a, 'py> ::pyo3::impl_::extract_argument::PyFunctionArgument<'a, 'py> for &'a mut MyClass +impl<'a, 'py> pyo3::impl_::extract_argument::PyFunctionArgument<'a, 'py> for &'a mut MyClass { - type Holder = ::std::option::Option<::pyo3::PyRefMut<'py, MyClass>>; + type Holder = ::std::option::Option>; #[inline] - fn extract(obj: &'py ::pyo3::PyAny, holder: &'a mut Self::Holder) -> ::pyo3::PyResult { - ::pyo3::impl_::extract_argument::extract_pyclass_ref_mut(obj, holder) + fn extract(obj: &'py pyo3::PyAny, holder: &'a mut Self::Holder) -> pyo3::PyResult { + pyo3::impl_::extract_argument::extract_pyclass_ref_mut(obj, holder) } } @@ -1001,9 +973,9 @@ impl pyo3::impl_::pyclass::PyClassImpl for MyClass { type BaseType = PyAny; type ThreadChecker = pyo3::impl_::pyclass::ThreadCheckerStub; type PyClassMutability = <::PyClassMutability as pyo3::impl_::pycell::PyClassMutability>::MutableChild; - type Dict = ::pyo3::impl_::pyclass::PyClassDummySlot; - type WeakRef = ::pyo3::impl_::pyclass::PyClassDummySlot; - type BaseNativeType = ::pyo3::PyAny; + type Dict = pyo3::impl_::pyclass::PyClassDummySlot; + type WeakRef = pyo3::impl_::pyclass::PyClassDummySlot; + type BaseNativeType = pyo3::PyAny; fn items_iter() -> pyo3::impl_::pyclass::PyClassItemsIter { use pyo3::impl_::pyclass::*; diff --git a/guide/src/class/object.md b/guide/src/class/object.md index fa1f4ef76f9..d5afe5fd604 100644 --- a/guide/src/class/object.md +++ b/guide/src/class/object.md @@ -45,10 +45,10 @@ It can't even print an user-readable representation of itself! We can fix that b ```rust # use pyo3::prelude::*; -# +# # #[pyclass] # struct Number(i32); -# +# #[pymethods] impl Number { // For `__repr__` we want to return a string that Python code could use to recreate @@ -83,10 +83,10 @@ use std::collections::hash_map::DefaultHasher; use std::hash::{Hash, Hasher}; # use pyo3::prelude::*; -# +# # #[pyclass] # struct Number(i32); -# +# #[pymethods] impl Number { fn __hash__(&self) -> u64 { @@ -110,7 +110,7 @@ impl Number { > By default, all `#[pyclass]` types have a default hash implementation from Python. > Types which should not be hashable can override this by setting `__hash__` to None. > This is the same mechanism as for a pure-Python class. This is done like so: -> +> > ```rust > # use pyo3::prelude::*; > #[pyclass] @@ -174,16 +174,40 @@ impl Number { It checks that the `std::cmp::Ordering` obtained from Rust's `Ord` matches the given `CompareOp`. +Alternatively, if you want to leave some operations unimplemented, you can +return `py.NotImplemented()` for some of the operations: + + +```rust +use pyo3::class::basic::CompareOp; + +# use pyo3::prelude::*; +# +# #[pyclass] +# struct Number(i32); +# +#[pymethods] +impl Number { + fn __richcmp__(&self, other: &Self, op: CompareOp, py: Python<'_>) -> PyObject { + match op { + CompareOp::Eq => (self.0 == other.0).into_py(py), + CompareOp::Ne => (self.0 != other.0).into_py(py), + _ => py.NotImplemented(), + } + } +} +``` + ### Truthyness We'll consider `Number` to be `True` if it is nonzero: ```rust # use pyo3::prelude::*; -# +# # #[pyclass] # struct Number(i32); -# +# #[pymethods] impl Number { fn __bool__(&self) -> bool { diff --git a/guide/src/class/protocols.md b/guide/src/class/protocols.md index a541b5e1b2a..f43e7f21c59 100644 --- a/guide/src/class/protocols.md +++ b/guide/src/class/protocols.md @@ -44,7 +44,7 @@ given signatures should be interpreted as follows: - `__hash__() -> isize` - Objects that compare equal must have the same hash value. + Objects that compare equal must have the same hash value. Any type up to 64 bits may be returned instead of `isize`, PyO3 will convert to an isize automatically (wrapping unsigned types like `u64` and `usize`).
Disabling Python's default hash By default, all `#[pyclass]` types have a default hash implementation from Python. Types which should not be hashable can override this by setting `__hash__` to `None`. This is the same mechanism as for a pure-Python class. This is done like so: @@ -67,6 +67,8 @@ given signatures should be interpreted as follows: Overloads Python comparison operations (`==`, `!=`, `<`, `<=`, `>`, and `>=`). The `CompareOp` argument indicates the comparison operation being performed. + + _Note that implementing `__richcmp__` will cause Python not to generate a default `__hash__` implementation, so consider implementing `__hash__` when implementing `__richcmp__`._
Return type The return type will normally be `PyResult`, but any Python object can be returned. diff --git a/guide/src/conversions.md b/guide/src/conversions.md index f535e083666..991c2061042 100644 --- a/guide/src/conversions.md +++ b/guide/src/conversions.md @@ -1,3 +1,3 @@ -# Type Conversions +# Type conversions In this portion of the guide we'll talk about the mapping of Python types to Rust types offered by PyO3, as well as the traits available to perform conversions between them. diff --git a/guide/src/ecosystem.md b/guide/src/ecosystem.md index 0937a048953..478edaece43 100644 --- a/guide/src/ecosystem.md +++ b/guide/src/ecosystem.md @@ -1,4 +1,4 @@ -# The PyO3 Ecosystem +# The PyO3 ecosystem This portion of the guide is dedicated to crates which are external to the main PyO3 project and provide additional functionality you might find useful. diff --git a/guide/src/ecosystem/async-await.md b/guide/src/ecosystem/async-await.md index 262b27e682c..1ab50065541 100644 --- a/guide/src/ecosystem/async-await.md +++ b/guide/src/ecosystem/async-await.md @@ -1,4 +1,4 @@ -# Async / Await +# Using `async` and `await` If you are working with a Python library that makes use of async functions or wish to provide Python bindings for an async Rust library, [`pyo3-asyncio`](https://github.com/awestlake87/pyo3-asyncio) diff --git a/guide/src/exception.md b/guide/src/exception.md index 68b4846eee0..385be99838a 100644 --- a/guide/src/exception.md +++ b/guide/src/exception.md @@ -1,4 +1,4 @@ -# Python Exceptions +# Python exceptions ## Defining a new exception @@ -52,35 +52,9 @@ fn mymodule(py: Python<'_>, m: &PyModule) -> PyResult<()> { ## Raising an exception -To raise an exception from `pyfunction`s and `pymethods`, you should return an `Err(PyErr)`. -If returned to Python code, this [`PyErr`] will then be raised as a Python exception. Many PyO3 APIs also return [`PyResult`]. +As described in the [function error handling](./function/error_handling.md) chapter, to raise an exception from a `#[pyfunction]` or `#[pymethods]`, return an `Err(PyErr)`. PyO3 will automatically raise this exception for you when returing the result to Python. -If a Rust type exists for the exception, then it is possible to use the `new_err` method. -For example, each standard exception defined in the `pyo3::exceptions` module -has a corresponding Rust type and exceptions defined by [`create_exception!`] and [`import_exception!`] macro have Rust types as well. - -```rust -use pyo3::exceptions::PyZeroDivisionError; -use pyo3::prelude::*; - -#[pyfunction] -fn divide(a: i32, b: i32) -> PyResult { - match a.checked_div(b) { - Some(q) => Ok(q), - None => Err(PyZeroDivisionError::new_err("division by zero")), - } -} -# -# fn main(){ -# Python::with_gil(|py|{ -# let fun = pyo3::wrap_pyfunction!(divide, py).unwrap(); -# fun.call1((1,0)).unwrap_err(); -# fun.call1((1,1)).unwrap(); -# }); -# } -``` - -You can manually write and fetch errors in the Python interpreter's global state: +You can also manually write and fetch errors in the Python interpreter's global state: ```rust use pyo3::{Python, PyErr}; @@ -93,8 +67,6 @@ Python::with_gil(|py| { }); ``` -If you already have a Python exception object, you can use [`PyErr::from_value`] to create a `PyErr` from it. - ## Checking exception types Python has an [`isinstance`](https://docs.python.org/3/library/functions.html#isinstance) method to check an object's type. @@ -123,104 +95,6 @@ err.is_instance_of::(py); # }); ``` -## Handling Rust errors - -The vast majority of operations in this library will return -[`PyResult`]({{#PYO3_DOCS_URL}}/pyo3/prelude/type.PyResult.html), -which is an alias for the type `Result`. - -A [`PyErr`] represents a Python exception. Errors within the PyO3 library are also exposed as -Python exceptions. - -If your code has a custom error type, adding an implementation of `std::convert::From for PyErr` -is usually enough. PyO3 will then automatically convert your error to a Python exception when needed. - -The following code snippet defines a Rust error named `CustomIOError`. In its `From for PyErr` -implementation it returns a `PyErr` representing Python's `OSError`. - -```rust -use pyo3::exceptions::PyOSError; -use pyo3::prelude::*; -use std::fmt; - -#[derive(Debug)] -struct CustomIOError; - -impl std::error::Error for CustomIOError {} - -impl fmt::Display for CustomIOError { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - write!(f, "Oh no!") - } -} - -impl std::convert::From for PyErr { - fn from(err: CustomIOError) -> PyErr { - PyOSError::new_err(err.to_string()) - } -} - -pub struct Connection { /* ... */} - -fn bind(addr: String) -> Result { - if &addr == "0.0.0.0"{ - Err(CustomIOError) - } else { - Ok(Connection{ /* ... */}) - } -} - -#[pyfunction] -fn connect(s: String) -> Result<(), CustomIOError> { - bind(s)?; - Ok(()) -} - -fn main() { - Python::with_gil(|py| { - let fun = pyo3::wrap_pyfunction!(connect, py).unwrap(); - let err = fun.call1(("0.0.0.0",)).unwrap_err(); - assert!(err.is_instance_of::(py)); - }); -} -``` - -This has been implemented for most of Rust's standard library errors, so that you can use the `?` -("try") operator with them. The following code snippet will raise a `ValueError` in Python if -`String::parse()` returns an error. - -```rust -use pyo3::prelude::*; - -fn parse_int(s: String) -> PyResult { - Ok(s.parse::()?) -} -# -# use pyo3::exceptions::PyValueError; -# -# fn main() { -# Python::with_gil(|py| { -# assert_eq!(parse_int(String::from("1")).unwrap(), 1); -# assert_eq!(parse_int(String::from("1337")).unwrap(), 1337); -# -# assert!(parse_int(String::from("-1")) -# .unwrap_err() -# .is_instance_of::(py)); -# assert!(parse_int(String::from("foo")) -# .unwrap_err() -# .is_instance_of::(py)); -# assert!(parse_int(String::from("13.37")) -# .unwrap_err() -# .is_instance_of::(py)); -# }) -# } -``` - -If lazy construction of the Python exception instance is desired, the -[`PyErrArguments`]({{#PYO3_DOCS_URL}}/pyo3/trait.PyErrArguments.html) -trait can be implemented. In that case, actual exception argument creation is delayed -until the `PyErr` is needed. - ## Using exceptions defined in Python code It is possible to use an exception defined in Python code as a native Rust type. diff --git a/guide/src/faq.md b/guide/src/faq.md index 6b0dd57ff7f..1389f131569 100644 --- a/guide/src/faq.md +++ b/guide/src/faq.md @@ -1,4 +1,4 @@ -# Frequently Asked Questions / Troubleshooting +# Frequently Asked Questions and troubleshooting ## I'm experiencing deadlocks using PyO3 with lazy_static or once_cell! diff --git a/guide/src/features.md b/guide/src/features.md index 5974ba0f450..8cb5b0bc407 100644 --- a/guide/src/features.md +++ b/guide/src/features.md @@ -1,4 +1,4 @@ -# Features Reference +# Features reference PyO3 provides a number of Cargo features to customise functionality. This chapter of the guide provides detail on each of them. diff --git a/guide/src/function.md b/guide/src/function.md index 41550c18414..a370254d756 100644 --- a/guide/src/function.md +++ b/guide/src/function.md @@ -1,4 +1,4 @@ -# Python Functions +# Python functions The `#[pyfunction]` attribute is used to define a Python function from a Rust function. Once defined, the function needs to be added to a [module](./module.md) using the `wrap_pyfunction!` macro. @@ -19,17 +19,20 @@ fn my_extension(py: Python<'_>, m: &PyModule) -> PyResult<()> { } ``` -This chapter of the guide explains full usage of the `#[pyfunction]` attribute. The following topics are covered: +This chapter of the guide explains full usage of the `#[pyfunction]` attribute. In this first section, the following topics are covered: - [Function options](#function-options) - [`#[pyo3(name = "...")]`](#name) - [`#[pyo3(text_signature = "...")]`](#text_signature) - [`#[pyo3(pass_module)]`](#pass_module) -- [Argument parsing](#argument-parsing) - - [`#[pyo3(from_py_with = "...")]`](#from_py_with) +- [Per-argument options](#per-argument-options) - [Advanced function patterns](#advanced-function-patterns) - [`#[pyfn]` shorthand](#pyfn-shorthand) +There are also additional sections on the following topics: + +- [Function Signatures](./function/signature.md) + ## Function options The `#[pyo3]` attribute can be used to modify properties of the generated Python function. It can take any combination of the following options: @@ -118,27 +121,7 @@ The `#[pyo3]` attribute can be used to modify properties of the generated Python } ``` -## Argument parsing - -The `#[pyfunction]` attribute supports specifying details of argument parsing. The details are given in the section ["Method arguments" of the Classes chapter](class.md#method-arguments). Here is an example for a function that accepts arbitrary keyword arguments (`**kwargs` in Python syntax) and returns the number that was passed: - -```rust -use pyo3::prelude::*; -use pyo3::types::PyDict; - -#[pyfunction(kwds="**")] -fn num_kwds(kwds: Option<&PyDict>) -> usize { - kwds.map_or(0, |dict| dict.len()) -} - -#[pymodule] -fn module_with_functions(py: Python<'_>, m: &PyModule) -> PyResult<()> { - m.add_function(wrap_pyfunction!(num_kwds, m)?).unwrap(); - Ok(()) -} -``` - -### Per-argument options +## Per-argument options The `#[pyo3]` attribute can be used on individual arguments to modify properties of them in the generated function. It can take any combination of the following options: @@ -214,12 +197,6 @@ Docstring: This function adds two unsigned 64-bit integers. Type: builtin_function_or_method ``` -### Closures - -Currently, there are no conversions between `Fn`s in Rust and callables in Python. This would -definitely be possible and very useful, so contributions are welcome. In the meantime, you can do -the following: - ### Calling Python functions in Rust You can pass Python `def`'d functions and built-in functions to Rust functions [`PyFunction`] @@ -234,13 +211,12 @@ with only positional args. ### Calling Rust functions in Python -If you have a static function, you can expose it with `#[pyfunction]` and use [`wrap_pyfunction!`] -to get the corresponding [`PyCFunction`]. For dynamic functions, e.g. lambdas and functions that -were passed as arguments, you must put them in some kind of owned container, e.g. a `Box`. -(A long-term solution will be a special container similar to wasm-bindgen's `Closure`). You can -then use a `#[pyclass]` struct with that container as a field as a way to pass the function over -the FFI barrier. You can even make that class callable with `__call__` so it looks like a function -in Python code. +The ways to convert a Rust function into a Python object vary depending on the function: + +- Named functions, e.g. `fn foo()`: add `#[pyfunction]` and then use [`wrap_pyfunction!`] to get the corresponding [`PyCFunction`]. +- Anonymous functions (or closures), e.g. `foo: fn()` either: + - use a `#[pyclass]` struct which stores the function as a field and implement `__call__` to call the stored function. + - use `PyFunction::new_closure` to create an object directly from the function. [`PyAny::is_callable`]: {{#PYO3_DOCS_URL}}/pyo3/struct.PyAny.html#tymethod.is_callable [`PyAny::call`]: {{#PYO3_DOCS_URL}}/pyo3/struct.PyAny.html#tymethod.call diff --git a/guide/src/function/error_handling.md b/guide/src/function/error_handling.md new file mode 100644 index 00000000000..1dd4ac9d106 --- /dev/null +++ b/guide/src/function/error_handling.md @@ -0,0 +1,222 @@ +# Error handling + +This chapter contains a little background of error handling in Rust and how PyO3 integrates this with Python exceptions. + +This covers enough detail to create a `#[pyfunction]` which raises Python exceptions from errors originating in Rust. + +There is a later section of the guide on [Python exceptions](../exception.md) which covers exception types in more detail. + +## Representing Python exceptions + +Rust code uses the generic [`Result`] enum to propagate errors. The error type `E` is chosen by the code author to describe the possible errors which can happen. + +PyO3 has the [`PyErr`] type which represents a Python exception. If a PyO3 API could result in a Python exception being raised, the return type of that `API` will be [`PyResult`], which is an alias for the type `Result`. + +In summary: +- When Python exceptions are raised and caught by PyO3, the exception will stored in the `Err` variant of the `PyResult`. +- Passing Python exceptions through Rust code then uses all the "normal" techniques such as the `?` operator, with `PyErr` as the error type. +- Finally, when a `PyResult` crosses from Rust back to Python via PyO3, if the result is an `Err` variant the contained exception will be raised. + +(There are many great tutorials on Rust error handling and the `?` operator, so this guide will not go into detail on Rust-specific topics.) + +## Raising an exception from a function + +As indicated in the previous section, when a `PyResult` containing an `Err` crosses from Rust to Python, PyO3 will raise the exception contained within. + +Accordingly, to raise an exception from a `#[pyfunction]`, change the return type `T` to `PyResult`. When the function returns an `Err` it will raise a Python exception. (Other `Result` types can be used as long as the error `E` has a `From` conversion for `PyErr`, see [implementing a conversion](#implementing-an-error-conversion) below.) + +This also works for functions in `#[pymethods]`. + +For example, the following `check_positive` function raises a `ValueError` when the input is negative: + +```rust +use pyo3::exceptions::PyValueError; +use pyo3::prelude::*; + +#[pyfunction] +fn check_positive(x: i32) -> PyResult<()> { + if x < 0 { + Err(PyValueError::new_err("x is negative")) + } else { + Ok(()) + } +} +# +# fn main(){ +# Python::with_gil(|py|{ +# let fun = pyo3::wrap_pyfunction!(check_positive, py).unwrap(); +# fun.call1((-1,)).unwrap_err(); +# fun.call1((1)).unwrap(); +# }); +# } +``` + +All built-in Python exception types are defined in the [`pyo3::exceptions`] module. They have a `new_err` constructor to directly build a `PyErr`, as seen in the example above. + +## Custom Rust error types + +PyO3 will automatically convert a `Result` returned by a `#[pyfunction]` into a `PyResult` as long as there is an implementation of `std::from::From for PyErr`. Many error types in the Rust standard library have a [`From`] conversion defined in this way. + +If the type `E` you are handling is defined in a third-party crate, see the section on [foreign rust error types](#foreign-rust-error-types) below for ways to work with this error. + +The following example makes use of the implementation of `From for PyErr` to raise exceptions encountered when parsing strings as integers: + +```rust +# use pyo3::prelude::*; +use std::num::ParseIntError; + +#[pyfunction] +fn parse_int(x: &str) -> Result { + x.parse() +} +``` + +When passed a string which doesn't contain a floating-point number, the exception raised will look like the below: + +```python +>>> parse_int("bar") +Traceback (most recent call last): + File "", line 1, in +ValueError: invalid digit found in string +``` + +As a more complete example, the following snippet defines a Rust error named `CustomIOError`. It then defines a `From for PyErr`, which returns a `PyErr` representing Python's `OSError`. Finally, it + +```rust +use pyo3::exceptions::PyOSError; +use pyo3::prelude::*; +use std::fmt; + +#[derive(Debug)] +struct CustomIOError; + +impl std::error::Error for CustomIOError {} + +impl fmt::Display for CustomIOError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "Oh no!") + } +} + +impl std::convert::From for PyErr { + fn from(err: CustomIOError) -> PyErr { + PyOSError::new_err(err.to_string()) + } +} + +pub struct Connection { /* ... */} + +fn bind(addr: String) -> Result { + if &addr == "0.0.0.0"{ + Err(CustomIOError) + } else { + Ok(Connection{ /* ... */}) + } +} + +#[pyfunction] +fn connect(s: String) -> Result<(), CustomIOError> { + bind(s)?; + Ok(()) +} + +fn main() { + Python::with_gil(|py| { + let fun = pyo3::wrap_pyfunction!(connect, py).unwrap(); + let err = fun.call1(("0.0.0.0",)).unwrap_err(); + assert!(err.is_instance_of::(py)); + }); +} +``` + +If lazy construction of the Python exception instance is desired, the +[`PyErrArguments`]({{#PYO3_DOCS_URL}}/pyo3/trait.PyErrArguments.html) +trait can be implemented instead of `From`. In that case, actual exception argument creation is delayed +until the `PyErr` is needed. + +A final note is that any errors `E` which have a `From` conversion can be used with the `?` +("try") operator with them. An alternative implementation of the above `parse_int` which instead returns `PyResult` is below: + +```rust +use pyo3::prelude::*; + +fn parse_int(s: String) -> PyResult { + let x = s.parse()?; + Ok(x) +} +# +# use pyo3::exceptions::PyValueError; +# +# fn main() { +# Python::with_gil(|py| { +# assert_eq!(parse_int(String::from("1")).unwrap(), 1); +# assert_eq!(parse_int(String::from("1337")).unwrap(), 1337); +# +# assert!(parse_int(String::from("-1")) +# .unwrap_err() +# .is_instance_of::(py)); +# assert!(parse_int(String::from("foo")) +# .unwrap_err() +# .is_instance_of::(py)); +# assert!(parse_int(String::from("13.37")) +# .unwrap_err() +# .is_instance_of::(py)); +# }) +# } +``` + +## Foreign Rust error types + +The Rust compiler will not permit implementation of traits for types outside of the crate where the type is defined. (This is known as the "orphan rule".) + +Given a type `OtherError` which is defined in third-party code, there are two main strategies available to integrate it with PyO3: + +- Create a newtype wrapper, e.g. `MyOtherError`. Then implement `From for PyErr` (or `PyErrArguments`), as well as `From` for `MyOtherError`. +- Use Rust's Result combinators such as `map_err` to write code freely to convert `OtherError` into whatever is needed. This requires boilerplate at every usage however gives unlimited flexibility. + +To detail the newtype strategy a little further, the key trick is to return `Result` from the `#[pyfunction]`. This means that PyO3 will make use of `From for PyErr` to create Python exceptions while the `#[pyfunction]` implementation can use `?` to convert `OtherError` to `MyOtherError` automatically. + +The following example demonstrates this for some imaginary third-party crate `some_crate` with a function `get_x` returning `Result`: + +```rust +# mod some_crate { +# struct OtherError(()); +# impl OtherError { +# fn message() -> &'static str { "some error occurred" } +# } +# fn get_x() -> Result +# } + +use pyo3::prelude::*; +use some_crate::{OtherError, get_x}; + +struct MyOtherError(OtherError); + +impl From for PyErr { + fn from(error: MyOtherError) -> Self { + PyValueError::new_err(self.0.message()) + } +} + +impl From for MyOtherError { + fn from(other: OtherError) -> Self { + Self(other) + } +} + +#[pyfunction] +fn wrapped_get_x() -> Result { + // get_x is a function returning Result + let x: i32 = get_x()?; + Ok(x) +} +``` + + +[`From`]: https://doc.rust-lang.org/stable/std/convert/trait.From.html +[`Result`]: https://doc.rust-lang.org/stable/std/result/enum.Result.html + +[`PyResult`]: {{#PYO3_DOCS_URL}}/pyo3/prelude/type.PyResult.html +[`PyResult`]: {{#PYO3_DOCS_URL}}/pyo3/prelude/type.PyResult.html +[`PyErr`]: {{#PYO3_DOCS_URL}}/pyo3/struct.PyErr.html +[`pyo3::exceptions`]: {{#PYO3_DOCS_URL}}/pyo3/exceptions/index.html diff --git a/guide/src/function/signature.md b/guide/src/function/signature.md new file mode 100644 index 00000000000..259fda19c86 --- /dev/null +++ b/guide/src/function/signature.md @@ -0,0 +1,97 @@ +# Function signatures + +The `#[pyfunction]` attribute also accepts parameters to control how the generated Python function accepts arguments. Just like in Python, arguments can be positional-only, keyword-only, or accept either. `*args` lists and `**kwargs` dicts can also be accepted. These parameters also work for `#[pymethods]` which will be introduced in the [Python Classes](../class.md) section of the guide. + +Like Python, by default PyO3 accepts all arguments as either positional or keyword arguments. The extra arguments to `#[pyfunction]` modify this behaviour. For example, below is a function that accepts arbitrary keyword arguments (`**kwargs` in Python syntax) and returns the number that was passed: + +```rust +use pyo3::prelude::*; +use pyo3::types::PyDict; + +#[pyfunction(kwds="**")] +fn num_kwds(kwds: Option<&PyDict>) -> usize { + kwds.map_or(0, |dict| dict.len()) +} + +#[pymodule] +fn module_with_functions(py: Python<'_>, m: &PyModule) -> PyResult<()> { + m.add_function(wrap_pyfunction!(num_kwds, m)?).unwrap(); + Ok(()) +} +``` + +The following parameters can be passed to the `#[pyfunction]` attribute: + + * `"/"`: positional-only arguments separator, each parameter defined before `"/"` is a + positional-only parameter. + Corresponds to python's `def meth(arg1, arg2, ..., /, argN..)`. + * `"*"`: var arguments separator, each parameter defined after `"*"` is a keyword-only parameter. + Corresponds to python's `def meth(*, arg1.., arg2=..)`. + * `args="*"`: "args" is var args, corresponds to Python's `def meth(*args)`. Type of the `args` + parameter has to be `&PyTuple`. + * `kwargs="**"`: "kwargs" receives keyword arguments, corresponds to Python's `def meth(**kwargs)`. + The type of the `kwargs` parameter has to be `Option<&PyDict>`. + * `arg="Value"`: arguments with default value. Corresponds to Python's `def meth(arg=Value)`. + If the `arg` argument is defined after var arguments, it is treated as a keyword-only argument. + Note that `Value` has to be valid rust code, PyO3 just inserts it into the generated + code unmodified. + +Example: +```rust +# use pyo3::prelude::*; +use pyo3::types::{PyDict, PyTuple}; +# +# #[pyclass] +# struct MyClass { +# num: i32, +# } +#[pymethods] +impl MyClass { + #[new] + #[args(num = "-1")] + fn new(num: i32) -> Self { + MyClass { num } + } + + #[args( + num = "10", + py_args = "*", + name = "\"Hello\"", + py_kwargs = "**" + )] + fn method( + &mut self, + num: i32, + name: &str, + py_args: &PyTuple, + py_kwargs: Option<&PyDict>, + ) -> PyResult { + self.num = num; + Ok(format!( + "py_args={:?}, py_kwargs={:?}, name={}, num={}", + py_args, py_kwargs, name, self.num + )) + } + + fn make_change(&mut self, num: i32) -> PyResult { + self.num = num; + Ok(format!("num={}", self.num)) + } +} +``` +N.B. the position of the `"/"` and `"*"` arguments (if included) control the system of handling positional and keyword arguments. In Python: +```python +import mymodule + +mc = mymodule.MyClass() +print(mc.method(44, False, "World", 666, x=44, y=55)) +print(mc.method(num=-1, name="World")) +print(mc.make_change(44, False)) +``` +Produces output: +```text +py_args=('World', 666), py_kwargs=Some({'x': 44, 'y': 55}), name=Hello, num=44 +py_args=(), py_kwargs=None, name=World, num=-1 +num=44 +num=-1 +``` diff --git a/guide/src/getting_started.md b/guide/src/getting_started.md index 16bb98ef0e8..66ee841ac4c 100644 --- a/guide/src/getting_started.md +++ b/guide/src/getting_started.md @@ -1,33 +1,32 @@ -# Instalation +# Installation -To get started using PyO3 you will need three things: a rust toolchain, a python environment, and a way to build. We'll cover each of these below. +To get started using PyO3 you will need three things: a rust toolchain, a python environment, and a way to build. We'll cover each of these below. ## Rust First, make sure you have rust installed on your system. If you haven't already done so you can do so by following the instructions [here](https://www.rust-lang.org/tools/install). PyO3 runs on both the `stable` and `nightly` versions so you can choose whichever one fits you best. The minimum required rust version is Rust 1.48. -if you can run `rustc --version` and the version is high enough you're good to go! +if you can run `rustc --version` and the version is high enough you're good to go! ## Python -To use PyO3 you need at least Python 3.7. While you can simply use the default Python version on your system, it is recommended to use a virtual environment. +To use PyO3 you need at least Python 3.7. While you can simply use the default Python version on your system, it is recommended to use a virtual environment. +## Virtualenvs -## Virtualenvs - -While you can use any virtualenv manager you like, we recommend the use of `pyenv` especially if you want to develop or test for multiple different python versions, so that is what the examples in this book will use. The installation instructions for `pyenv` can be found [here](https://github.com/pyenv/pyenv#getting-pyenv). +While you can use any virtualenv manager you like, we recommend the use of `pyenv` especially if you want to develop or test for multiple different python versions, so that is what the examples in this book will use. The installation instructions for `pyenv` can be found [here](https://github.com/pyenv/pyenv#getting-pyenv). Note that when using `pyenv` you should also set the following environment variable ```bash PYTHON_CONFIGURE_OPTS="--enable-shared" ``` -### Building +### Building -There are a number of build and python package management systems such as [`setuptools-rust`](https://github.com/PyO3/setuptools-rust) or [manually](https://pyo3.rs/latest/building_and_distribution.html#manual-builds) we recommend the use of `maturin` which you can install [here](https://maturin.rs/installation.html). It is developed to work with PyO3 and is the most "batteries included" experience. `maturin` is just a python package so you can add it in any way that you install python packages. +There are a number of build and python package management systems such as [`setuptools-rust`](https://github.com/PyO3/setuptools-rust) or [manually](https://pyo3.rs/latest/building_and_distribution.html#manual-builds) we recommend the use of `maturin` which you can install [here](https://maturin.rs/installation.html). It is developed to work with PyO3 and is the most "batteries included" experience. `maturin` is just a python package so you can add it in any way that you install python packages. -System Python: +System Python: ```bash pip install maturin --user ``` @@ -48,12 +47,11 @@ poetry: poetry add -D maturin ``` -after installation, you can run `maturin --version` to check that you have correctly installed it. - +after installation, you can run `maturin --version` to check that you have correctly installed it. # Starting a new project -Firstly you should create the folder and virtual environment that are going to contain your new project. Here we will use the recommended `pyenv`: +Firstly you should create the folder and virtual environment that are going to contain your new project. Here we will use the recommended `pyenv`: ```bash mkdir pyo3-example @@ -61,19 +59,19 @@ cd pyo3-example pyenv virtualenv pyo3 pyenv local pyo3 ``` -after this, you should install your build manager. In this example, we will use `maturin`. After you've activated your virtualenv add `maturin` to it: +after this, you should install your build manager. In this example, we will use `maturin`. After you've activated your virtualenv add `maturin` to it: ```bash pip install maturin ``` -After this, you can initialise the new project +After this, you can initialise the new project ```bash maturin init ``` -If `maturin` is already installed you can create a new project using that directly as well: +If `maturin` is already installed you can create a new project using that directly as well: ```bash maturin new -b pyo3 pyo3-example @@ -84,16 +82,16 @@ pyenv local pyo3 # Adding to an existing project -Sadly currently `maturin` cannot be run in existing projects, so if you want to use python in an existing project you basically have two options: +Sadly currently `maturin` cannot be run in existing projects, so if you want to use python in an existing project you basically have two options: 1. create a new project as above and move your existing code into that project -2. Manually edit your project configuration as necessary. +2. Manually edit your project configuration as necessary. -If you are opting for the second option, here are the things you need to pay attention to: +If you are opting for the second option, here are the things you need to pay attention to: ## Cargo.toml -Make sure that the rust you want to be able to access from Python is compiled into a library. You can have a binary output as well, but the code you want to access from python has to be in the library. Also, make sure that the crate type is `cdylib` and add PyO3 as a dependency as so: +Make sure that the rust you want to be able to access from Python is compiled into a library. You can have a binary output as well, but the code you want to access from python has to be in the library. Also, make sure that the crate type is `cdylib` and add PyO3 as a dependency as so: ```toml @@ -111,7 +109,8 @@ pyo3 = { version = "0.16.5", features = ["extension-module"] } ``` ## pyproject.toml -You should also create a `pyproject.toml` with the following contents: + +You should also create a `pyproject.toml` with the following contents: ```toml [build-system] @@ -130,7 +129,7 @@ classifiers = [ ## Running code -After this you can setup rust code to be available in python as such: +After this you can setup rust code to be available in python as such: ```rust use pyo3::prelude::*; @@ -151,7 +150,7 @@ fn string_sum(_py: Python<'_>, m: &PyModule) -> PyResult<()> { } ``` -After this you can run `maturin develop` to prepare the python package after which you can use it like so: +After this you can run `maturin develop` to prepare the python package after which you can use it like so: ```bash $ maturin develop @@ -162,4 +161,4 @@ $ python '25' ``` -For more instructions on how to use python code from rust see the [Python from Rust](python_from_rust.md) page. \ No newline at end of file +For more instructions on how to use python code from rust see the [Python from Rust](python_from_rust.md) page. diff --git a/guide/src/memory.md b/guide/src/memory.md index 5b43dd78362..aca624ab492 100644 --- a/guide/src/memory.md +++ b/guide/src/memory.md @@ -1,4 +1,4 @@ -# Memory Management +# Memory management Rust and Python have very different notions of memory management. Rust has a strict memory model with concepts of ownership, borrowing, and lifetimes, @@ -13,7 +13,7 @@ PyO3 bridges the Rust and Python memory models with two different strategies for accessing memory allocated on Python's heap from inside Rust. These are GIL-bound, or "owned" references, and GIL-independent `Py` smart pointers. -## GIL-bound Memory +## GIL-bound memory PyO3's GIL-bound, "owned references" (`&PyAny` etc.) make PyO3 more ergonomic to use by ensuring that their lifetime can never be longer than the duration the @@ -119,7 +119,7 @@ dropped you do not retain access to any owned references created after the [documentation for `Python::new_pool()`]({{#PYO3_DOCS_URL}}/pyo3/prelude/struct.Python.html#method.new_pool) for more information on safety. -## GIL-independent Memory +## GIL-independent memory Sometimes we need a reference to memory on Python's heap that can outlive the GIL. Python's `Py` is analogous to `Rc`, but for variables whose diff --git a/guide/src/module.md b/guide/src/module.md index 4146fb183a2..9e52ab93e2b 100644 --- a/guide/src/module.md +++ b/guide/src/module.md @@ -1,4 +1,4 @@ -# Python Modules +# Python modules You can create a module using `#[pymodule]`: @@ -64,7 +64,7 @@ print(my_extension.__doc__) ## Python submodules -You can create a module hierarchy within a single extension module by using +You can create a module hierarchy within a single extension module by using [`PyModule.add_submodule()`]({{#PYO3_DOCS_URL}}/pyo3/prelude/struct.PyModule.html#method.add_submodule). For example, you could define the modules `parent_module` and `parent_module.child_module`. @@ -99,9 +99,9 @@ fn func() -> String { # }) ``` -Note that this does not define a package, so this won’t allow Python code to directly import -submodules by using `from parent_module import child_module`. For more information, see -[#759](https://github.com/PyO3/pyo3/issues/759) and +Note that this does not define a package, so this won’t allow Python code to directly import +submodules by using `from parent_module import child_module`. For more information, see +[#759](https://github.com/PyO3/pyo3/issues/759) and [#1517](https://github.com/PyO3/pyo3/issues/1517#issuecomment-808664021). It is not necessary to add `#[pymodule]` on nested modules, which is only required on the top-level module. diff --git a/src/marker.rs b/src/marker.rs index 7540b615b2c..494814b1428 100644 --- a/src/marker.rs +++ b/src/marker.rs @@ -288,6 +288,10 @@ impl Python<'_> { #[cfg_attr(PyPy, doc = "`prepare_freethreaded_python`")] /// for details. /// + /// If the current thread does not yet have a Python "thread state" associated with it, + /// a new one will be automatically created before `F` is executed and destroyed after `F` + /// completes. + /// /// # Panics /// /// - If the [`auto-initialize`] feature is not enabled and the Python interpreter is not diff --git a/src/pycell.rs b/src/pycell.rs index aaf4c384f42..2dd6e4f79a2 100644 --- a/src/pycell.rs +++ b/src/pycell.rs @@ -57,9 +57,9 @@ //! # //! // This function is exported to Python. //! unsafe extern "C" fn __pymethod_increment__( -//! _slf: *mut ::pyo3::ffi::PyObject, -//! _args: *mut ::pyo3::ffi::PyObject, -//! ) -> *mut ::pyo3::ffi::PyObject { +//! _slf: *mut pyo3::ffi::PyObject, +//! _args: *mut pyo3::ffi::PyObject, +//! ) -> *mut pyo3::ffi::PyObject { //! use :: pyo3 as _pyo3; //! let gil = _pyo3::GILPool::new(); //! let _py = gil.python();