BUG: concat losing columns dtypes for join=outer #47586
Conversation
phofl
added
Bug
Reshaping
| @@ -995,6 +995,7 @@ Reshaping | |||
| - Bug in :func:`get_dummies` that selected object and categorical dtypes but not string (:issue:`44965`) | |||
| - Bug in :meth:`DataFrame.align` when aligning a :class:`MultiIndex` to a :class:`Series` with another :class:`MultiIndex` (:issue:`46001`) | |||
| - Bug in concatenation with ``IntegerDtype``, or ``FloatingDtype`` arrays where the resulting dtype did not mirror the behavior of the non-nullable dtypes (:issue:`46379`) | |||
| - Bug in :func:`concat` losing dtype of columns when ``join="outer"`` and ``sort=True`` (:issue:`47329`) | |||
There was a problem hiding this comment.
issue is marked as 1.4.4. ok to leave in 1.5 though (if so just change the issue and ignore this comment)
There was a problem hiding this comment.
Changed the milestones.
initially we though this happens only
for ea dtypes, but this was wrong. Occurs also for numpy dtypes
|
thanks @phofl |
| """ | ||
| dtypes = [idx.dtype for idx in indexes if isinstance(idx, Index)] | ||
| if dtypes: | ||
| dtype = find_common_type(dtypes) |
There was a problem hiding this comment.
I assume we never pass a mixed list of Indexes and lists? could add type annotations here for clarity.
There was a problem hiding this comment.
A parent function uses list[list[Hashable] | Index], so not sure
There was a problem hiding this comment.
I assumed that if we could pass a mixed list, then the logic added here would not account for the types in a list and only use the Indexes to find the common dtype. We could therefore expect this to raise in those cases?
There was a problem hiding this comment.
A parent function uses
list[list[Hashable] | Index], so not sure
we may need to change to list[list[Hashable]] | list[Index]
| @@ -223,7 +224,7 @@ def union_indexes(indexes, sort: bool | None = True) -> Index: | |||
|
|
|||
| indexes, kind = _sanitize_and_check(indexes) | |||
|
|
|||
| def _unique_indices(inds) -> Index: | |||
| def _unique_indices(inds, dtype) -> Index: | |||
There was a problem hiding this comment.
the docstring also needs updating at some point.
doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.@simonjayhawkins This also happens for regular dtypes, so would not backport