PyMapping is inconsistent with typing.Mapping

[Gobot1234]

Bug Description

The conversion table for PyMapping suggests anything that is a typing.Mapping should be able to be converted to PyMapping, this however is not the case seemly list and str are also downcast-able to a PyMapping.

This causes errors when you try to access things like items on the PyMapping as you get a AttributeError str has no attribute items.

Steps to Reproduce

use pyo3::prelude::*;
use pyo3::types::PyMapping;

#[pyfunction]
fn downcast_(a: &PyAny) -> PyResult<&PyMapping> {
    Ok(a.downcast::<PyMapping>()?)
}

#[pymodule]
fn downcast(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(downcast_, m)?)?;

    Ok(())
}

>>> import typing, downcast
>>> isinstance({}, typing.Mapping)
True
>>> downcast.downcast_({})
{}
>>> isinstance("", typing.Mapping)
False
>>> downcast.downcast_("")
''

Backtrace

No response

Your operating system and version

MacOS 11.1 (20C69)

Your Python version (python --version)

3.9.3

Your Rust version (rustc --version)

rustc 1.53.0-nightly (07e0e2ec2 2021-03-24)

Your PyO3 version

0.15.1

How did you install python? Did you use a virtualenv?

venv

Additional Info

No response

[davidhewitt]

Yikes, thanks for reporting.

So I'm happy to take the fall for this one. This is the C-API definition of PyMapping_Check:

.. c:function:: int PyMapping_Check(PyObject *o)

   Return ``1`` if the object provides mapping protocol or supports slicing,
   and ``0`` otherwise.  Note that it returns ``1`` for Python classes with
   a :meth:`__getitem__` method since in general case it is impossible to
   determine what type of keys it supports. This function always succeeds.

Reading this again, it's such a broad definition that it doesn't really seem practically useful. And you're right that it's very different to typing.Mapping.

On Python 3.10 and up we could potentially make use of Py_TPFLAGS_MAPPING and Py_TPFLAGS_SEQUENCE to accept the same set of types as the match keyword. That seems generally correct.

However I'm not fond of having behaviour that's different according to the Python version.

Alternatively, we could call into Python to pretty much literally do isinstance(<thing>, collections.abc.Mapping), but that seems like a waste of performance. This might be a solution for Python 3.9 and backwards if we wanted to use Py_TPFLAGS_MAPPING.

Ideas on a solution very welcome!

[Gobot1234]

I like the sound of using both the tp flags and the instance checks depending on your version

[davidhewitt]

I think this is awkward enough of a footgun (and also new enough, having been added in 0.15) that it would be good if we could come to a decision about what to do for this in time for the 0.16 release.

All opinions very much wanted!

[birkenfeld]

Would be interesting to see what C extensions do.

[aviramha]

I think that we should consider trying to resolve this upstream in CPython. The proposed behavior (checking of tpflags/isinstance) feels like something that PyMapping_Check should do. Opinions?

[bharel]

I think that we should consider trying to resolve this upstream in CPython. The proposed behavior (checking of tpflags/isinstance) feels like something that PyMapping_Check should do. Opinions?

Remember guys, PyMapping_Check in CPython is used quite frequently and is meant to be as optimized as possible, while isinstance does not only check for inheritance.

Modifying it in CPython will cause a discrepancy across versions, and since you're not actually checking a __subclasshook__, user made classes will still not be taken into account. After all, not every user made mapping inherits from the appropriate ABC.

You can try blacklisting builtins but it's a whack a mole and will cause inconsistencies.

All in all, unless you take __subclasshook__ into account, which you don't want as it's a custom Python function, it'll never be the same as isinstance.

The only possible solution is to maybe covert the ABCs and respective hook methods to C in order to optimize them, but even then it'll be much slower.

[aganders3]

I found some seemingly relevant discussion on BPO https://bugs.python.org/issue5945

One suggestion there is to use PyMapping_Check() && !PySequence_Check() for a true mapping check. Would that work here?

[aviramha]

@aganders3 Nice find! I'll try to implement it and see how it behaves.