Skip to content

Commit 8fd8d0d

Browse files
authored
ENH: Integer NA Extension Array (#21160)
* ENH: add integer-na support via an ExtensionArray closes #20700 closes #20747
1 parent 8c50682 commit 8fd8d0d

30 files changed

+1634
-119
lines changed

doc/source/whatsnew/v0.24.0.txt

+58
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ v0.24.0 (Month XX, 2018)
1313
New features
1414
~~~~~~~~~~~~
1515

16+
1617
- ``ExcelWriter`` now accepts ``mode`` as a keyword argument, enabling append to existing workbooks when using the ``openpyxl`` engine (:issue:`3441`)
1718

1819
.. _whatsnew_0240.enhancements.extension_array_operators:
@@ -31,6 +32,62 @@ See the :ref:`ExtensionArray Operator Support
3132
<extending.extension.operator>` documentation section for details on both
3233
ways of adding operator support.
3334

35+
.. _whatsnew_0240.enhancements.intna:
36+
37+
Optional Integer NA Support
38+
^^^^^^^^^^^^^^^^^^^^^^^^^^^
39+
40+
Pandas has gained the ability to hold integer dtypes with missing values. This long requested feature is enabled through the use of :ref:`extension types <extending.extension-types>`.
41+
Here is an example of the usage.
42+
43+
We can construct a ``Series`` with the specified dtype. The dtype string ``Int64`` is a pandas ``ExtensionDtype``. Specifying a list or array using the traditional missing value
44+
marker of ``np.nan`` will infer to integer dtype. The display of the ``Series`` will also use the ``NaN`` to indicate missing values in string outputs. (:issue:`20700`, :issue:`20747`)
45+
46+
.. ipython:: python
47+
48+
s = pd.Series([1, 2, np.nan], dtype='Int64')
49+
s
50+
51+
52+
Operations on these dtypes will propagate ``NaN`` as other pandas operations.
53+
54+
.. ipython:: python
55+
56+
# arithmetic
57+
s + 1
58+
59+
# comparison
60+
s == 1
61+
62+
# indexing
63+
s.iloc[1:3]
64+
65+
# operate with other dtypes
66+
s + s.iloc[1:3].astype('Int8')
67+
68+
# coerce when needed
69+
s + 0.01
70+
71+
These dtypes can operate as part of of ``DataFrame``.
72+
73+
.. ipython:: python
74+
75+
df = pd.DataFrame({'A': s, 'B': [1, 1, 3], 'C': list('aab')})
76+
df
77+
df.dtypes
78+
79+
80+
These dtypes can be merged & reshaped & casted.
81+
82+
.. ipython:: python
83+
84+
pd.concat([df[['A']], df[['B', 'C']]], axis=1).dtypes
85+
df['A'].astype(float)
86+
87+
.. warning::
88+
89+
The Integer NA support currently uses the captilized dtype version, e.g. ``Int8`` as compared to the traditional ``int8``. This may be changed at a future date.
90+
3491
.. _whatsnew_0240.enhancements.read_html:
3592

3693
``read_html`` Enhancements
@@ -258,6 +315,7 @@ Previous Behavior:
258315
ExtensionType Changes
259316
^^^^^^^^^^^^^^^^^^^^^
260317

318+
- ``ExtensionArray`` has gained the abstract methods ``.dropna()`` (:issue:`21185`)
261319
- ``ExtensionDtype`` has gained the ability to instantiate from string dtypes, e.g. ``decimal`` would instantiate a registered ``DecimalDtype``; furthermore
262320
the ``ExtensionDtype`` has gained the method ``construct_array_type`` (:issue:`21185`)
263321
- The ``ExtensionArray`` constructor, ``_from_sequence`` now take the keyword arg ``copy=False`` (:issue:`21185`)

pandas/core/arrays/__init__.py

+3
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,10 @@
11
from .base import (ExtensionArray, # noqa
2+
ExtensionOpsMixin,
23
ExtensionScalarOpsMixin)
34
from .categorical import Categorical # noqa
45
from .datetimes import DatetimeArrayMixin # noqa
56
from .interval import IntervalArray # noqa
67
from .period import PeriodArrayMixin # noqa
78
from .timedeltas import TimedeltaArrayMixin # noqa
9+
from .integer import ( # noqa
10+
IntegerArray, to_integer_array)

pandas/core/arrays/base.py

+9-6
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,8 @@
1212
from pandas.errors import AbstractMethodError
1313
from pandas.compat.numpy import function as nv
1414
from pandas.compat import set_function_name, PY3
15-
from pandas.core.dtypes.common import is_list_like
1615
from pandas.core import ops
16+
from pandas.core.dtypes.common import is_list_like
1717

1818
_not_implemented_message = "{} does not implement {}."
1919

@@ -88,16 +88,19 @@ class ExtensionArray(object):
8888
# Constructors
8989
# ------------------------------------------------------------------------
9090
@classmethod
91-
def _from_sequence(cls, scalars, copy=False):
91+
def _from_sequence(cls, scalars, dtype=None, copy=False):
9292
"""Construct a new ExtensionArray from a sequence of scalars.
9393
9494
Parameters
9595
----------
9696
scalars : Sequence
9797
Each element will be an instance of the scalar type for this
9898
array, ``cls.dtype.type``.
99+
dtype : dtype, optional
100+
Construct for this particular dtype. This should be a Dtype
101+
compatible with the ExtensionArray.
99102
copy : boolean, default False
100-
if True, copy the underlying data
103+
If True, copy the underlying data.
101104
Returns
102105
-------
103106
ExtensionArray
@@ -378,7 +381,7 @@ def fillna(self, value=None, method=None, limit=None):
378381
func = pad_1d if method == 'pad' else backfill_1d
379382
new_values = func(self.astype(object), limit=limit,
380383
mask=mask)
381-
new_values = self._from_sequence(new_values)
384+
new_values = self._from_sequence(new_values, dtype=self.dtype)
382385
else:
383386
# fill with value
384387
new_values = self.copy()
@@ -407,7 +410,7 @@ def unique(self):
407410
from pandas import unique
408411

409412
uniques = unique(self.astype(object))
410-
return self._from_sequence(uniques)
413+
return self._from_sequence(uniques, dtype=self.dtype)
411414

412415
def _values_for_factorize(self):
413416
# type: () -> Tuple[ndarray, Any]
@@ -559,7 +562,7 @@ def take(self, indices, allow_fill=False, fill_value=None):
559562
560563
result = take(data, indices, fill_value=fill_value,
561564
allow_fill=allow_fill)
562-
return self._from_sequence(result)
565+
return self._from_sequence(result, dtype=self.dtype)
563566
"""
564567
# Implementer note: The `fill_value` parameter should be a user-facing
565568
# value, an instance of self.dtype.type. When passed `fill_value=None`,

pandas/core/arrays/categorical.py

+2-2
Original file line numberDiff line numberDiff line change
@@ -488,8 +488,8 @@ def _constructor(self):
488488
return Categorical
489489

490490
@classmethod
491-
def _from_sequence(cls, scalars):
492-
return Categorical(scalars)
491+
def _from_sequence(cls, scalars, dtype=None, copy=False):
492+
return Categorical(scalars, dtype=dtype)
493493

494494
def copy(self):
495495
""" Copy constructor. """

0 commit comments

Comments
 (0)