Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: better document FromPyObject for &str changes in migration guide #3974

Merged
merged 4 commits into from
Mar 22, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions guide/src/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ The rough order of material in this user guide is as follows:
Please choose from the chapters on the left to jump to individual topics, or continue below to start with PyO3's README.

<div class="warning">

⚠️ Warning: API update in progress 🛠️

PyO3 0.21 has introduced a significant new API, termed the "Bound" API after the new smart pointer `Bound<T>`.
Expand Down
5 changes: 4 additions & 1 deletion guide/src/memory.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Memory management

<div class="warning">

⚠️ Warning: API update in progress 🛠️

PyO3 0.21 has introduced a significant new API, termed the "Bound" API after the new smart pointer `Bound<T>`.
Expand Down Expand Up @@ -84,6 +85,7 @@ the end of the `with_gil()` closure, at which point the 10 copies of `hello`
are finally released to the Python garbage collector.

<div class="warning">

⚠️ Warning: `GILPool` is no longer the preferred way to manage memory with PyO3 🛠️

PyO3 0.21 has introduced a new API known as the Bound API, which doesn't have the same surprising results. Instead, each `Bound<T>` smart pointer releases the Python reference immediately on drop. See [the smart pointer types](./types.md#pyo3s-smart-pointers) for more details.
Expand Down Expand Up @@ -121,7 +123,7 @@ this is unsafe.
# fn main() -> PyResult<()> {
Python::with_gil(|py| -> PyResult<()> {
for _ in 0..10 {
#[allow(deprecated)] // `new_pool` is not needed in code not using the GIL Refs API
#[allow(deprecated)] // `new_pool` is not needed in code not using the GIL Refs API
let pool = unsafe { py.new_pool() };
let py = pool.python();
#[allow(deprecated)] // py.eval() is part of the GIL Refs API
Expand Down Expand Up @@ -155,6 +157,7 @@ a few objects, meaning this doesn't have a significant impact. Occasionally func
with long complex loops may need to use `Python::new_pool` as shown above.

<div class="warning">

⚠️ Warning: `GILPool` is no longer the preferred way to manage memory with PyO3 🛠️

PyO3 0.21 has introduced a new API known as the Bound API, which doesn't have the same surprising results. Instead, each `Bound<T>` smart pointer releases the Python reference immediately on drop. See [the smart pointer types](./types.md#pyo3s-smart-pointers) for more details.
Expand Down
71 changes: 65 additions & 6 deletions guide/src/migration.md
Original file line number Diff line number Diff line change
Expand Up @@ -245,13 +245,19 @@ Because the new `Bound<T>` API brings ownership out of the PyO3 framework and in
- `Bound<PyList>` and `Bound<PyTuple>` cannot support indexing with `list[0]`, you should use `list.get_item(0)` instead.
- `Bound<PyTuple>::iter_borrowed` is slightly more efficient than `Bound<PyTuple>::iter`. The default iteration of `Bound<PyTuple>` cannot return borrowed references because Rust does not (yet) have "lending iterators". Similarly `Bound<PyTuple>::get_borrowed_item` is more efficient than `Bound<PyTuple>::get_item` for the same reason.
- `&Bound<T>` does not implement `FromPyObject` (although it might be possible to do this in the future once the GIL Refs API is completely removed). Use `bound_any.downcast::<T>()` instead of `bound_any.extract::<&Bound<T>>()`.
- To convert between `&PyAny` and `&Bound<PyAny>` you can use the `as_borrowed()` method:
- `Bound<PyString>::to_str` now borrows from the `Bound<PyString>` rather than from the `'py` lifetime, so code will need to store the smart pointer as a value in some cases where previously `&PyString` was just used as a temporary. (There are some more details relating to this in [the section below](#deactivating-the-gil-refs-feature).)

To convert between `&PyAny` and `&Bound<PyAny>` you can use the `as_borrowed()` method:

```rust,ignore
let gil_ref: &PyAny = ...;
let bound: &Bound<PyAny> = &gil_ref.as_borrowed();
```

<div class="warning">

⚠️ Warning: dangling pointer trap 💣

> Because of the ownership changes, code which uses `.as_ptr()` to convert `&PyAny` and other GIL Refs to a `*mut pyo3_ffi::PyObject` should take care to avoid creating dangling pointers now that `Bound<PyAny>` carries ownership.
>
> For example, the following pattern with `Option<&PyAny>` can easily create a dangling pointer when migrating to the `Bound<PyAny>` smart pointer:
Expand All @@ -267,6 +273,7 @@ let bound: &Bound<PyAny> = &gil_ref.as_borrowed();
> let opt: Option<Bound<PyAny>> = ...;
> let p: *mut ffi::PyObject = opt.as_ref().map_or(std::ptr::null_mut(), Bound::as_ptr);
> ```
<div>

#### Migrating `FromPyObject` implementations

Expand Down Expand Up @@ -312,14 +319,66 @@ Despite a large amount of deprecations warnings produced by PyO3 to aid with the

As a final step of migration, deactivating the `gil-refs` feature will set up code for best performance and is intended to set up a forward-compatible API for PyO3 0.22.

At this point code which needed to manage GIL Ref memory can safely remove uses of `GILPool` (which are constructed by calls to `Python::new_pool` and `Python::with_pool`). Deprecation warnings will highlight these cases.
At this point code that needed to manage GIL Ref memory can safely remove uses of `GILPool` (which are constructed by calls to `Python::new_pool` and `Python::with_pool`). Deprecation warnings will highlight these cases.

There is one notable API removed when this feature is disabled. `FromPyObject` trait implementations for types which borrow directly from the input data cannot be implemented by PyO3 without GIL Refs (while the migration is ongoing). These types are `&str`, `Cow<'_, str>`, `&[u8]`, `Cow<'_, u8>`.
There is just one case of code that changes upon disabling these features: `FromPyObject` trait implementations for types that borrow directly from the input data cannot be implemented by PyO3 without GIL Refs (while the GIL Refs API is in the process of being removed). The main types affected are `&str`, `Cow<'_, str>`, `&[u8]`, `Cow<'_, u8>`.

To ease pain during migration, these types instead implement a new temporary trait `FromPyObjectBound` which is the expected future form of `FromPyObject`. The new temporary trait ensures is that `obj.extract::<&str>()` continues to work, as well for these types in `#[pyfunction]` arguments.
To make PyO3's core functionality continue to work while the GIL Refs API is in the process of being removed, disabling the `gil-refs` feature moves the implementations of `FromPyObject` for `&str`, `Cow<'_, str>`, `&[u8]`, `Cow<'_, u8>` to a new temporary trait `FromPyObjectBound`. This trait is the expected future form of `FromPyObject` and has an additional lifetime `'a` to enable these types to borrow data from Python objects.

An unfortunate final point here is that PyO3 cannot offer this new implementation for `&str` on `abi3` builds for Python older than 3.10. On code which needs `abi3` builds for these older Python versions, many cases of `.extract::<&str>()` may need to be replaced with `.extract::<PyBackedStr>()`, which is string data which borrows from the Python `str` object. Alternatively, use `.extract::<Cow<str>>()` ro `.extract::<String>()` to copy the data into Rust for these versions.
</details>
A key thing to note here is because extracting to these types now ties them to the input lifetime, some extremely common patterns may need to be split into multiple Rust lines. For example, the following snippet of calling `.extract::<&str>()` directly on the result of `.getattr()` needs to be adjusted when deactivating the `gil-refs-migration` feature.

Before:

```rust
# #[cfg(feature = "gil-refs-migration")] {
# use pyo3::prelude::*;
# use pyo3::types::{PyList, PyType};
# fn example<'py>(py: Python<'py>) -> PyResult<()> {
#[allow(deprecated)] // GIL Ref API
let obj: &'py PyType = py.get_type::<PyList>();
let name: &'py str = obj.getattr("__name__")?.extract()?;
assert_eq!(name, "list");
# Ok(())
# }
# Python::with_gil(example).unwrap();
# }
```

After:

```rust
# #[cfg(any(not(Py_LIMITED_API), Py_3_10))] {
# use pyo3::prelude::*;
# use pyo3::types::{PyList, PyType};
# fn example<'py>(py: Python<'py>) -> PyResult<()> {
let obj: Bound<'py, PyType> = py.get_type_bound::<PyList>();
let name_obj: Bound<'py, PyAny> = obj.getattr("__name__")?;
// the lifetime of the data is no longer `'py` but the much shorter
// lifetime of the `name_obj` smart pointer above
let name: &'_ str = name_obj.extract()?;
assert_eq!(name, "list");
# Ok(())
# }
# Python::with_gil(example).unwrap();
# }
```

To avoid needing to worry about lifetimes at all, it is also possible to use the new `PyBackedStr` type, which stores a reference to the Python `str` without a lifetime attachment. In particular, `PyBackedStr` helps for `abi3` builds for Python older than 3.10. Due to limitations in the `abi3` CPython API for those older versions, PyO3 cannot offer a `FromPyObjectBound` implementation for `&str` on those versions. The easiest way to migrate for older `abi3` builds is to replace any cases of `.extract::<&str>()` with `.extract::<PyBackedStr>()`. Alternatively, use `.extract::<Cow<str>>()`, `.extract::<String>()` to copy the data into Rust.

The following example uses the same snippet as those just above, but this time the final extracted type is `PyBackedStr`:

```rust
# use pyo3::prelude::*;
# use pyo3::types::{PyList, PyType};
# fn example<'py>(py: Python<'py>) -> PyResult<()> {
use pyo3::pybacked::PyBackedStr;
let obj: Bound<'py, PyType> = py.get_type_bound::<PyList>();
let name: PyBackedStr = obj.getattr("__name__")?.extract()?;
assert_eq!(&*name, "list");
# Ok(())
# }
# Python::with_gil(example).unwrap();
```

## from 0.19.* to 0.20

Expand Down
Loading