Did you know that py::buffer_info::format has a different meaning on Windows?

[jpivarski]

On Windows (both 32-bit and 64-bit Python),

std::string f(py::array a) {
    py::buffer_info info = a.request();
    return info.format;
}

returns "l" for a numpy.int32 array and "q" for a numpy.int64 array, whereas MacOS and Linux return "i" and "l", respectively. (That is, "l" is ambiguous.)

To be safe against misinterpretations, I'm using

auto b = a.cast<py::array_t<int64_t, py::array::c_style | py::array::forcecast>>();

to convert whatever I have into int64_t (and similarly for int32_t), if necessary. (Then, choosing int32_t vs int64_t based on platform is just an optimization, not needed for correctness.)

[jpivarski]

The same is true on a 32-bit Docker image (i386/ubuntu):

• numpy.int32 → format == "l"
• numpy.int64 → format == "q"

whereas on 64-bit Linux:

• numpy.int32 → format == "i"
• numpy.int64 → format == "l"

Windows, regardless of whether it's 32-bit or 64-bit, take the same meanings as 32-bit Linux.

[jpivarski]

Idioms like the following are successful on 32-bit Linux, 64-bit Linux, 32-bit Windows, and 64-bit Windows (and 64-bit MacOS; haven't tested 32-bit MacOS). It's possible that the __i386__ macro is only defined on gcc, so that's another thing I'll have to consider.

    if (format_.compare("d") == 0) {
      out = std::make_shared<PrimitiveType>(parameters_, PrimitiveType::float64);
    }
    else if (format_.compare("f") == 0) {
      out = std::make_shared<PrimitiveType>(parameters_, PrimitiveType::float32);
    }
#if defined _MSC_VER || defined __i386__
    else if (format_.compare("q") == 0) {
#else
    else if (format_.compare("l") == 0) {
#endif
      out = std::make_shared<PrimitiveType>(parameters_, PrimitiveType::int64);
    }
#if defined _MSC_VER || defined __i386__
    else if (format_.compare("Q") == 0) {
#else
    else if (format_.compare("L") == 0) {
#endif
      out = std::make_shared<PrimitiveType>(parameters_, PrimitiveType::uint64);
    }
#if defined _MSC_VER || defined __i386__
    else if (format_.compare("l") == 0) {
#else
    else if (format_.compare("i") == 0) {
#endif
      out = std::make_shared<PrimitiveType>(parameters_, PrimitiveType::int32);
    }
#if defined _MSC_VER || defined __i386__
    else if (format_.compare("L") == 0) {
#else
    else if (format_.compare("I") == 0) {
#endif
      out = std::make_shared<PrimitiveType>(parameters_, PrimitiveType::uint32);
    }
    else if (format_.compare("h") == 0) {
      out = std::make_shared<PrimitiveType>(parameters_, PrimitiveType::int16);
    }
    else if (format_.compare("H") == 0) {
      out = std::make_shared<PrimitiveType>(parameters_, PrimitiveType::uint16);
    }
    else if (format_.compare("b") == 0) {
      out = std::make_shared<PrimitiveType>(parameters_, PrimitiveType::int8);
    }
    else if (format_.compare("B") == 0  ||  format_.compare("c") == 0) {
      out = std::make_shared<PrimitiveType>(parameters_, PrimitiveType::uint8);
    }
    else if (format_.compare("?") == 0) {
      out = std::make_shared<PrimitiveType>(parameters_, PrimitiveType::boolean);
    }

[YannickJadoul]

I believe this is a combination of C/C++ not defining the sizes of integer types and the way numpy handles this.

Is there any particular issue when using this from pybind11, that doesn't match C/C++/numpy? Or do you suggest we add something to the docs somewhere, or ...?

[jpivarski]

Ultimately, all I need is a work-around, so if I'm given a buffer_info, I know how to cast it using stdint types. Above, I gave my work-around in the hope that it would help others, but recently, we've been running into more format related bugs that we'll cross-reference to this issue. At the moment, we don't know what's going wrong in the current work-around, so I don't have a specific request yet.

If this does get figured out, documentation is probably all that's needed, maybe a small section at the end of this page, since that's where we learn about the buffer_info interface and how to work with NumPy arrays. Maybe the best thing would be to have a listing of all the possible values format can take, with how each should be cast? (I just ran into 'e' for float16, as a result of taking the square root of an int8 in NumPy. It would be better not to discover each case, one by one, in production code.)

[YannickJadoul]

This does seem to mainly be a numpy feature/issue, though, where the meaning of i, l, q is int, long, long long, the size of these being platform-dependent, and np.int32 and np.int64 are aliases:

Testing out with 64-bit manylinux2010 Docker image:

$ docker run -it --rm quay.io/pypa/manylinux2010_x86_64 bash
[root@9962fd909c16 /]# /opt/python/cp38-cp38/bin/pip install numpy
Collecting numpy
  Downloading numpy-1.19.0-cp38-cp38-manylinux2010_x86_64.whl (14.6 MB)
     |████████████████████████████████| 14.6 MB 97 kB/s 
Installing collected packages: numpy
Successfully installed numpy-1.19.0
[root@9962fd909c16 /]# /opt/python/cp38-cp38/bin/python
Python 3.8.3 (default, Jun 16 2020, 21:10:27) 
[GCC 8.3.1 20190311 (Red Hat 8.3.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.dtype(np.int32).char
'i'
>>> np.dtype(np.int64).char
'l'
>>> 

32-bit:

$ docker run -it --rm quay.io/pypa/manylinux2010_i686 bash
[root@795a929a35e8 /]# /opt/python/cp38-cp38/bin/pip install numpy
Collecting numpy
  Downloading numpy-1.19.0-cp38-cp38-manylinux2010_i686.whl (12.3 MB)
     |████████████████████████████████| 12.3 MB 513 kB/s 
Installing collected packages: numpy
Successfully installed numpy-1.19.0
[root@795a929a35e8 /]# /opt/python/cp38-cp38/bin/python
Python 3.8.3 (default, Jul  4 2020, 10:49:13) 
[GCC 8.3.1 20190311 (Red Hat 8.3.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.dtype(np.int32).char
'l'
>>> np.dtype(np.int64).char
'q'
>>> 

See e.g., here: https://numpy.org/doc/stable/reference/arrays.scalars.html#built-in-scalar-types

I guess we could refer to these numpy docs, in some warning, when talking about dtypes in pybind11? If so, do you have time to make a quick PR on that?

[jpivarski]

I guess we could refer to these numpy docs, in some warning, when talking about dtypes in pybind11? If so, do you have time to make a quick PR on that?

I could. In the meantime, I'll use this space to try figuring things out, out in the open.

To try to get a handle on all the possible numeric types NumPy can handle, I scanned all objects attached to the numpy module, looking for array types, like this:

>>> [x for x in dir(numpy)
...  if isinstance(getattr(numpy, x), type) and issubclass(getattr(numpy, x), numpy.generic)]

then excluded any types that are not a leaf in the class hierarchy with:

[x for x in dir(numpy)
...  if isinstance(getattr(numpy, x), type) and issubclass(getattr(numpy, x), numpy.generic)]

Accounting for multiple names for the same type objects, nothing on this list has been left out of the above. Creating an array of each of these types, passing it into buffer_info, this is what I get for the format strings:

	Linux 32-bit	Linux 64-bit	MacOS 64-bit	Windows 32-bit	Windows 64-bit	diff?
bool_	?	?	?	?	?
int8	b	b	b	b	b
int16	h	h	h	h	h
int32	l	i	i	l	l	yes
int64	q	l	l	q	q	yes
uint8	B	B	B	B	B
uint16	H	H	H	H	H
uint32	L	I	I	L	L	yes
uint64	Q	L	L	Q	Q	yes
intc	i	i	i	i	i
uintc	I	I	I	I	I
longlong	q	q	q	q	q
ulonglong	Q	Q	Q	Q	Q
float16	e	e	e	e	e
float32	f	f	f	f	f
float64	d	d	d	d	d
float128	N/A	g	g	N/A	N/A	yes
complex64	Zf	Zf	Zf	Zf	Zf
complex128	Zd	Zd	Zd	Zd	Zd
complex256	N/A	Zg	Zg	N/A	N/A	yes
datetime64	M	M	M	M	M
timedelta64	m	m	m	m	m
bytes_	3s	3s	3s	3s	3s
str_	3w	3w	3w	3w	3w
record	T{...}	T{...}	T{...}	T{...}	T{...}
object_	O	O	O	O	O

On all systems where I could test it (Linux 64-bit, MacOS 64-bit, Windows 32-bit, and Windows 64-bit), Python 2.7 differed from Python 3.6+ only in that bytes_ and str_ are not distinguished (both are 3s). Numeric types are unaffected.

datetime64 and timedelta64 (M and m) both raised "ValueError: cannot include dtype 'M' in a buffer"; I'm using pybind 2.4.3.

A bug that we recently encountered involves a distinction between int64 and longlong. On 64-bit Linux, numpy.int64 is identically equal to numpy.intp, but numpy.longlong is a distinct object. I wonder why.

>>> numpy.int32, numpy.int64, numpy.intp, numpy.longlong
(<class 'numpy.int32'>, <class 'numpy.int64'>, <class 'numpy.int64'>, <class 'numpy.longlong'>)

[bstaletic]

On 64-bit Linux, numpy.int64 is identically equal to numpy.intp, but numpy.longlong is a distinct object. I wonder why.

I guess because intptr_t and int64_t are typedefs of long and long long is a distinct type.

[YannickJadoul]

I could. In the meantime, I'll use this space to try figuring things out, out in the open.

Thanks! And yes, please do; no problem at all to aggregate everything together in one place!

Creating an array of each of these types, passing it into buffer_info, this is what I get for the format strings:

Nice overview! So the main thing to note - I believe - is that intc, uintc, longlong, ulonglong, etc (did you somehow miss long and ulong?) are consistent across platforms (even if they have different physical sizes). So numpy defines the format strings based on the C types, and not based on the actual size of each integer type. We can debate on whether that's a good idea or not (very likely depends on the actual case), but there's not a lot we can do about it.

Note that you can compare dtype objects, btw:

>>> np.dtype(np.intc) == np.dtype(np.long)
False
>>> np.dtype(np.long) == np.dtype(np.longlong)
True

By some weird coincidence, I once made a PR on numpy about these dtype docstrings (during/after the sprints at EuroSciPy), and I while I don't remember all details, even writing docs for this was quite hairy given that you have these aliases. Maybe this is useful to look at: numpy/numpy#11858?

datetime64 and timedelta64 (M and m) both raised "ValueError: cannot include dtype 'M' in a buffer"; I'm using pybind 2.4.3.

If I'm not mistaken, there are some issues/PRs open on this, that we haven't gotten to yet.

[bstaletic]

I believe - is that intc, uintc, longlong, ulonglong, etc (did you somehow miss long and ulong?) are consistent across platforms (even if they have different physical sizes).

I think your conclusion is wrong here.

https://en.cppreference.com/w/cpp/language/types

Unix-like OS's always have int == 32bits, as does Win64. Same thing for long long - always 64 bits. That's why the format is consistent.

[YannickJadoul]

Unix-like OS's always have int == 32bits, as does Win64. Same thing for long long - always 64 bits. That's why the format is consistent.

Huh, interesting. But I don't think it changes anything. The thing is just that i is defined to be the format character for intc, according to https://numpy.org/doc/stable/reference/arrays.scalars.html#built-in-scalar-types, while the other, sized types (int32, etc) are just aliases to a type with that amount of bits.

[EricCousineau-TRI]

@jpivarski I believe your table effectively arises from the (rather complex) set of aliasing rules from this header file:
https://github.com/numpy/numpy/blob/6ef5ec39cdfaf77aa4600ec2e3bf9f679a4fd527/numpy/core/include/numpy/npy_common.h

In #1329, I thought I had aligned this with how it was defined in the latest version of NumPy (at least in time of writing); however, it does seem like that assumption may have been rather brittle, as noted by this issue.

@YannickJadoul pointed it out in this review comment:
#1329 (review)

I'm just wondering if there's a robust way to test this out in CI...