BUG: fix concat of Sparse with non-sparse dtypes #34338
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jorisvandenbossche
added
Bug
Reshaping
Co-authored-by: Tom Augspurger <TomAugspurger@users.noreply.github.com>
|
OK, will go forward with this PR as is (matching the previous behaviour) to unblock the other concat PRs, but will open a set of follow-up issues. |
| @@ -1063,7 +1063,8 @@ def astype(self, dtype=None, copy=True): | |||
| """ | |||
| dtype = self.dtype.update_dtype(dtype) | |||
| subtype = dtype._subtype_with_str | |||
| sp_values = astype_nansafe(self.sp_values, subtype, copy=copy) | |||
| # TODO copy=False is broken for astype_nansafe with int -> float | |||
| sp_values = astype_nansafe(self.sp_values, subtype, copy=True) | |||
There was a problem hiding this comment.
Opened #34456 for this (and added link in the comment)
| if is_sparse(arr.dtype) and not is_sparse(dtype): | ||
| # problem case: SparseArray.astype(dtype) doesn't follow the specified | ||
| # dtype exactly, but converts this to Sparse[dtype] -> first manually | ||
| # convert to dense array |
There was a problem hiding this comment.
Opened #34457 for this
|
And opened #34459 for the general "what should concat(sparse, categorical) do?" question |
pandas/core/dtypes/concat.py
Outdated
| @@ -20,7 +21,7 @@ | |||
| ) | |||
| from pandas.core.dtypes.generic import ABCCategoricalIndex, ABCRangeIndex, ABCSeries | |||
|
|
|||
| from pandas.core.arrays import ExtensionArray | |||
| from pandas.core.arrays import ExtensionArray, SparseArray | |||
There was a problem hiding this comment.
There was a problem hiding this comment.
Yeah, I already pushed a fix.
I don't really understand why, though, as SparseArray is included in the pandas.core.arrays init, just as ExtensionArray.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #34336
This adds tests with the behaviour as it was on 0.25.3 / 1.0.3, and some changes to get back to that behaviour.
(but whether this behaviour is fully "sane", I am not sure ..)