BUG: 54445 interferes with EAs that support complex128 #54540
Conversation
When a loc indexer goes to create a fresh array to hold values from an Extension Array, it makes the array allocation based on the na_value of the EA, but that na_value may be smaller than the size of the things that the EA can hold (such as complex128). Note that np.nan is itself a legitimate complex128 value. If the allocated array cannot hold the values from the block manager, and if the EA is not immutable, it will try a second strategy of allocating an array based on the dtype of the values of the blocks. If the blocks hold complex numbers, the array will be properly sized. This should close pandas-dev#54445. Signed-off-by: Michael Tiemann <72577720+MichaelTiemannOSC@users.noreply.github.com>
Make mypy happy. Signed-off-by: Michael Tiemann <72577720+MichaelTiemannOSC@users.noreply.github.com>
|
I think the numpy failure is due to other CI/CD problems and not this PR. |
| result[rl] = blk.iget((i, loc)) | ||
| except TypeError: | ||
| if isinstance(dtype, ExtensionDtype) and not immutable_ea: | ||
| values = [v[loc] for v in self.arrays] |
There was a problem hiding this comment.
this will only work in some fairly specific all-EA cases
There was a problem hiding this comment.
im skeptical this is the way to go, will discuss in the #54445 to try to get an idea of where the underlying problem is
There was a problem hiding this comment.
Maybe the solution is to use a (np.nan+0j) as the na_value, and maybe na_values should be the largest thing an array should be required to hold, not the smallest or simplest?
|
This pull request is stale because it has been open for thirty days with no activity. Please update and respond to this comment if you're still interested in working on this. |
|
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen. |
When a loc indexer goes to create a fresh array to hold values from an Extension Array, it makes the array allocation based on the na_value of the EA, but that na_value may be smaller than the size of the things that the EA can hold (such as complex128). Note that np.nan is itself a legitimate complex128 value.
If the allocated array cannot hold the values from the block manager, and if the EA is not immutable, it will try a second strategy of allocating an array based on the dtype of the values of the blocks. If the blocks hold complex numbers, the array will be properly sized.
This should close #54445.
doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.