PyErr drops the traceback on into_py(py).

[zakstucke]

Bug Description

As mentioned in pydantic/pydantic-core#780.

Because into_py(py) seems to directly convert the pvalue, rather than the full state, the traceback is dropped.

Can be fixed by manually re-attaching the tb to __traceback__ of the output object.

Steps to Reproduce

use pyo3::prelude::*;
use pyo3::wrap_pyfunction;

#[pyfunction]
fn broken_run_cb(py: Python, cb: &PyAny) -> PyObject {
    match cb.call0() {
        Ok(result) => result.into_py(py),
        Err(e) => {
            e.into_py(py)
        }
    }
}

#[pyfunction]
fn fixed_run_cb(py: Python, cb: &PyAny) -> PyObject {
    match cb.call0() {
        Ok(result) => result.into_py(py),
        Err(e) => {
            let tb = e.traceback(py);
            let out = e.clone_ref(py).into_py(py);
            out.setattr(py, "__traceback__", tb).unwrap();
            out
        }
    }
}

#[pymodule]
fn rust(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(broken_run_cb, m)?)?;
    m.add_function(wrap_pyfunction!(fixed_run_cb, m)?)?;
    Ok(())
}

import rust

def callback():
    raise ValueError("Python exc")

# BROKEN
raise ValueError("Outer") from rust.broken_run_cb(callback)
"""
ValueError: Python exc

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/bugfix-pyo3/example.py", line 6, in <module>
    raise ValueError("Outer") from rust.broken_run_cb(callback)
ValueError: Outer
"""

# FIXED
raise ValueError("Outer") from rust.fixed_run_cb(callback)
"""
Traceback (most recent call last):
  File "/bugfix-pyo3/example.py", line 4, in callback
    raise ValueError("Python exc")
ValueError: Python exc

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/bugfix-pyo3/example.py", line 18, in <module>
    raise ValueError("Outer") from rust.fixed_run_cb(callback)
ValueError: Outer
"""

Backtrace

No response

Your operating system and version

Ubuntu 22.04

Your Python version (python --version)

3.11.4

Your Rust version (rustc --version)

1.71.0

Your PyO3 version

0.19.1

How did you install python? Did you use a virtualenv?

apt

Additional Info

No response

[davidhewitt]

Thanks @zakstucke for the report. Based on your snippet above, I was able to create two following failing tests:

    #[test]
    fn test_err_from_value() {
        Python::with_gil(|py| {
            let locals = PyDict::new(py);
            // Produce an error from python so that it has a traceback
            py.run(
                r"
try:
    raise ValueError('raised exception')
except Exception as e:
    err = e
",
                None,
                Some(locals),
            )
            .unwrap();
            let err = PyErr::from_value(locals.get_item("err").unwrap());
            let traceback = err.value(py).getattr("__traceback__").unwrap();
            assert!(err.traceback(py).unwrap().is(traceback));
        })
    }

    #[test]
    fn test_err_into_py() {
        Python::with_gil(|py| {
            let locals = PyDict::new(py);
            // Produce an error from python so that it has a traceback
            py.run(
                r"
def f():
    raise ValueError('raised exception')
",
                None,
                Some(locals),
            )
            .unwrap();
            let f = locals.get_item("f").unwrap();
            let err = f.call0().unwrap_err();
            let traceback = err.traceback(py).unwrap();
            let err_object = err.clone_ref(py).into_py(py).into_ref(py);

            assert!(err_object.getattr("__traceback__").unwrap().is(traceback));
        })
    }

I think there's actually two separate bugs here:

• PyErr::from_value will ignore the traceback (test 1)
• PyErr::into_py will ignore the traceback (test 2)

I suspect these might work on Python 3.12 (after #3306), because on 3.12 the exception isn't split up inside the interpreter internals.

So I would argue these are bugs. I suggest we make two PRs to fix them each in turn. I can try get to this after #3306 is merged to avoid pain of rebasing. Alternatively if you are interested in submitting PRs here, the help is appreciated :)

[zakstucke]

@davidhewitt opened PR, seemed straightforward enough!