Typing of multiprocessing.Value broken since mypy 0.981

[prauscher]

Hello everyone,

after updating to mypy 0.981, i received a mypy error for multiprocessing.Value:

from multiprocessing import Value
import ctypes

mapper = Value(ctypes.c_float, float("nan"))
mapper.value = 2.3  # error: "SynchronizedBase[c_float]" has no attribute "value"

As far as I can tell, the issue is that Value() gives a SynchronizedBase which is not typed to have a value attribute. Synchronized however would offer this attribute, as it is present in the real world.

Am I missing something or should

typeshed/stdlib/multiprocessing/context.pyi

Line 89 in 9544447

def Value(self, typecode_or_type: type[_CT], *args: Any, lock: Literal[True] | _LockLike = ...) -> SynchronizedBase[_CT]: ...

be changed to Synchronized instead of SynchronizedBase ?

Update: The issue was not there in 0.971 and is now present in 0.981

[srittau]

I assume that this was introduced in #8330 by @hauntsaninja. Before that PR, the fourth overload was matched, which just returned Any from Value(), now the second overload is (correctly) matched. The implementation of BaseContext.Value() actually returns the concrete type multiprocessing.sharedctypes.Value, not the (abstract?) SynchronizedBase type. I wonder whether it would make sense to

• either annotate these methods with the concrete types (and use # type: ignore[override] in sub-classes if necessary); or
• override these methods in DefaultContext, which is what is eventually used for multiprocessing.Value().

[Joshix-1]

Array() has the same problem

"SynchronizedArray[c_char]" has no attribute "value"
"SynchronizedArray[c_char]" has no attribute "raw"

https://docs.python.org/3.10/library/multiprocessing.html#multiprocessing.Array

Note that an array of ctypes.c_char has value and raw attributes which allow one to use it to store and retrieve strings.

[angelo-daumas]

I've been meaning to address this problem in a PR (as well as other problems in #4266), however, I have a couple of questions:

The main issue is concerning the Synchronized type. At present it is parametrized by the python type, not the ctypes type.

typeshed/stdlib/multiprocessing/sharedctypes.pyi

Lines 90 to 91 in 9544447

	class Synchronized(SynchronizedBase[_SimpleCData[_T]], Generic[_T]):
	value: _T

Therefore, the correct usage would be: Synchronized[int], not Synchronized[ctypes.c_int]. However, I feel like the latter would be more correct, because otherwise you're losing information about what underlying ctype is being used. Note that SynchronizedBase is parametrized by the ctypes type, so SynchronizedBase[ctypes.c_int] would be correct. It also seems weird that they would be parametrized differently.

Unfortunately, I don't know how to type hint it so the Synchronized.value property returned the correct type (e.g. an int for something like ctypes.c_int or bool for ctypes.c_bool).

_SCT = TypeVar("_SCT", bound=_SimpleCData[Any])

class Synchronized(Generic[_SCT]):
    value: ?# What could go here? In C++ one could use decltype(self._obj.value), but there's no Py equivalent.

The second question is about the current usage of _SimpleCData and _CData. They are imported in the following file:

typeshed/stdlib/multiprocessing/sharedctypes.pyi

Line 3 in 9544447

from ctypes import _CData, _SimpleCData, c_char

Is it really okay to use these, even though they are prefixed with an underscore? Seems to me like it would mark them as private. There's a related issue about this (#6018). Also, please note that _CData is not actually accessible at runtime from ctypes, it's only available in the stub file. That also seems pretty counterintuitive.

--- TL;DR ---

If the following questions are answered, I can start working on a PR to resolve this issue:

1. Is Synchronized[int] over Synchronized[ctypes.c_int] really the best option?
2. If it is not, is there a way to make Synchronized[ctypes.c_int] work while keeping the correct type for Synchronized.value?
3. Is it really okay to use the private types from ctypes, especially _CData, which is not exported at runtime?

[angelo-daumas]

I assume that this was introduced in #8330 by @hauntsaninja. Before that PR, the fourth overload was matched, which just returned Any from Value(), now the second overload is (correctly) matched. The implementation of BaseContext.Value() actually returns the concrete type multiprocessing.sharedctypes.Value, not the (abstract?) SynchronizedBase type. I wonder whether it would make sense to

* either annotate these methods with the concrete types (and use `# type: ignore[override]` in sub-classes if necessary); or

* override these methods in `DefaultContext`, which is what is eventually used for `multiprocessing.Value()`.

As far as I can see, multiprocessing.sharedctypes.Value is a factory function. It's been unfortunately named with an uppercase letter. SynchronizedBase and Synchronized are the types returned. The multiprocessing.Value export is a method of the default context that simply calls multiprocessing.sharedctypes.Value. Please correct me if I'm wrong.