From dcd125b8c411e1bf01ec81124c7dcbbcffe5adf0 Mon Sep 17 00:00:00 2001 From: Tom Augspurger Date: Wed, 28 Feb 2018 06:19:16 -0600 Subject: [PATCH 1/8] Updated docs --- doc/source/developer.rst | 43 ------ doc/source/extending.rst | 240 ++++++++++++++++++++++++++++++++++ doc/source/index.rst.template | 1 + doc/source/internals.rst | 157 +--------------------- 4 files changed, 243 insertions(+), 198 deletions(-) create mode 100644 doc/source/extending.rst diff --git a/doc/source/developer.rst b/doc/source/developer.rst index 0ef097da090f2..b8bb2b2fcbe2f 100644 --- a/doc/source/developer.rst +++ b/doc/source/developer.rst @@ -140,46 +140,3 @@ As an example of fully-formed metadata: 'metadata': None} ], 'pandas_version': '0.20.0'} - -.. _developer.register-accessors: - -Registering Custom Accessors ----------------------------- - -Libraries can use the decorators -:func:`pandas.api.extensions.register_dataframe_accessor`, -:func:`pandas.api.extensions.register_series_accessor`, and -:func:`pandas.api.extensions.register_index_accessor`, to add additional "namespaces" to -pandas objects. All of these follow a similar convention: you decorate a class, providing the name of attribute to add. The -class's `__init__` method gets the object being decorated. For example: - -.. code-block:: python - - @pd.api.extensions.register_dataframe_accessor("geo") - class GeoAccessor(object): - def __init__(self, pandas_obj): - self._obj = pandas_obj - - @property - def center(self): - # return the geographic center point of this DataFarme - lon = self._obj.latitude - lat = self._obj.longitude - return (float(lon.mean()), float(lat.mean())) - - def plot(self): - # plot this array's data on a map, e.g., using Cartopy - pass - -Now users can access your methods using the `geo` namespace: - - >>> ds = pd.DataFrame({'longitude': np.linspace(0, 10), - ... 'latitude': np.linspace(0, 20)}) - >>> ds.geo.center - (5.0, 10.0) - >>> ds.geo.plot() - # plots data on a map - -This can be a convenient way to extend pandas objects without subclassing them. -If you write a custom accessor, make a pull request adding it to our -:ref:`ecosystem` page. diff --git a/doc/source/extending.rst b/doc/source/extending.rst new file mode 100644 index 0000000000000..2778ea9867a07 --- /dev/null +++ b/doc/source/extending.rst @@ -0,0 +1,240 @@ +.. _extending: + +**************** +Extending Pandas +**************** + +While pandas provides a rich set of methods, containers, and data types, your +needs may not be fully satisfied. Pandas offers a few options for extending +pandas. + +.. _extending.register-accessors: + +Registering Custom Accessors +---------------------------- + +Libraries can use the decorators +:func:`pandas.api.extensions.register_dataframe_accessor`, +:func:`pandas.api.extensions.register_series_accessor`, and +:func:`pandas.api.extensions.register_index_accessor`, to add additional +"namespaces" to pandas objects. All of these follow a similar convention: you +decorate a class, providing the name of attribute to add. The class's `__init__` +method gets the object being decorated. For example: + +.. code-block:: python + + @pd.api.extensions.register_dataframe_accessor("geo") + class GeoAccessor(object): + def __init__(self, pandas_obj): + self._obj = pandas_obj + + @property + def center(self): + # return the geographic center point of this DataFarme + lon = self._obj.latitude + lat = self._obj.longitude + return (float(lon.mean()), float(lat.mean())) + + def plot(self): + # plot this array's data on a map, e.g., using Cartopy + pass + +Now users can access your methods using the `geo` namespace: + + >>> ds = pd.DataFrame({'longitude': np.linspace(0, 10), + ... 'latitude': np.linspace(0, 20)}) + >>> ds.geo.center + (5.0, 10.0) + >>> ds.geo.plot() + # plots data on a map + +This can be a convenient way to extend pandas objects without subclassing them. +If you write a custom accessor, make a pull request adding it to our +:ref:`ecosystem` page. + +Extension Arrays +---------------- + +Pandas defines an interface for implementing data types and arrays that *extend* +NumPy's type system. Pandas uses itself for some types that aren't built into +NumPy (categorical, period, interval, datetime with timezone). + +Libraries can define an custom array and data type. When pandas encounters these +objects, they will be handled properly (i.e. not converted to an ndarray of +objects). Many methods like :func:`pandas.isna` will dispatch to the extension +type's implementation. + +The interface consists of two classes. + +``ExtensionDtype`` +"""""""""""""""""" + +An ``ExtensionDtype`` is similar to a ``numpy.dtype`` object. It describes the +data type. Implementors are responsible for a few unique items like the name. + +One particularly important item is the ``type`` property. This should be the +class that is the scalar type for your data. For example, if you were writing an +extension array for IP Address data, this might be ``ipaddress.IPv4Address``. + +See ``pandas/core/dtypes/base.py`` for interface definition. + +``ExtensionArray`` +"""""""""""""""""" + +This is the main object. + + +See ``pandas/core/arrays/base.py`` for the interface definition. + +.. _ref-subclassing-pandas: + +Subclassing pandas Data Structures +---------------------------------- + +.. warning:: There are some easier alternatives before considering subclassing ``pandas`` data structures. + + 1. Extensible method chains with :ref:`pipe ` + + 2. Use *composition*. See `here `_. + + 3. Extending by :ref:`registering an accessor ` + +This section describes how to subclass ``pandas`` data structures to meet more specific needs. There are 2 points which need attention: + +1. Override constructor properties. +2. Define original properties + +.. note:: You can find a nice example in `geopandas `_ project. + +Override Constructor Properties +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Each data structure has constructor properties to specifying data constructors. By overriding these properties, you can retain defined-classes through ``pandas`` data manipulations. + +There are 3 constructors to be defined: + +- ``_constructor``: Used when a manipulation result has the same dimesions as the original. +- ``_constructor_sliced``: Used when a manipulation result has one lower dimension(s) as the original, such as ``DataFrame`` single columns slicing. +- ``_constructor_expanddim``: Used when a manipulation result has one higher dimension as the original, such as ``Series.to_frame()`` and ``DataFrame.to_panel()``. + +Following table shows how ``pandas`` data structures define constructor properties by default. + +=========================== ======================= =================== ======================= +Property Attributes ``Series`` ``DataFrame`` ``Panel`` +=========================== ======================= =================== ======================= +``_constructor`` ``Series`` ``DataFrame`` ``Panel`` +``_constructor_sliced`` ``NotImplementedError`` ``Series`` ``DataFrame`` +``_constructor_expanddim`` ``DataFrame`` ``Panel`` ``NotImplementedError`` +=========================== ======================= =================== ======================= + +Below example shows how to define ``SubclassedSeries`` and ``SubclassedDataFrame`` overriding constructor properties. + +.. code-block:: python + + class SubclassedSeries(Series): + + @property + def _constructor(self): + return SubclassedSeries + + @property + def _constructor_expanddim(self): + return SubclassedDataFrame + + class SubclassedDataFrame(DataFrame): + + @property + def _constructor(self): + return SubclassedDataFrame + + @property + def _constructor_sliced(self): + return SubclassedSeries + +.. code-block:: python + + >>> s = SubclassedSeries([1, 2, 3]) + >>> type(s) + + + >>> to_framed = s.to_frame() + >>> type(to_framed) + + + >>> df = SubclassedDataFrame({'A', [1, 2, 3], 'B': [4, 5, 6], 'C': [7, 8, 9]}) + >>> df + A B C + 0 1 4 7 + 1 2 5 8 + 2 3 6 9 + + >>> type(df) + + + >>> sliced1 = df[['A', 'B']] + >>> sliced1 + A B + 0 1 4 + 1 2 5 + 2 3 6 + >>> type(sliced1) + + + >>> sliced2 = df['A'] + >>> sliced2 + 0 1 + 1 2 + 2 3 + Name: A, dtype: int64 + >>> type(sliced2) + + +Define Original Properties +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To let original data structures have additional properties, you should let ``pandas`` know what properties are added. ``pandas`` maps unknown properties to data names overriding ``__getattribute__``. Defining original properties can be done in one of 2 ways: + +1. Define ``_internal_names`` and ``_internal_names_set`` for temporary properties which WILL NOT be passed to manipulation results. +2. Define ``_metadata`` for normal properties which will be passed to manipulation results. + +Below is an example to define 2 original properties, "internal_cache" as a temporary property and "added_property" as a normal property + +.. code-block:: python + + class SubclassedDataFrame2(DataFrame): + + # temporary properties + _internal_names = pd.DataFrame._internal_names + ['internal_cache'] + _internal_names_set = set(_internal_names) + + # normal properties + _metadata = ['added_property'] + + @property + def _constructor(self): + return SubclassedDataFrame2 + +.. code-block:: python + + >>> df = SubclassedDataFrame2({'A': [1, 2, 3], 'B': [4, 5, 6], 'C': [7, 8, 9]}) + >>> df + A B C + 0 1 4 7 + 1 2 5 8 + 2 3 6 9 + + >>> df.internal_cache = 'cached' + >>> df.added_property = 'property' + + >>> df.internal_cache + cached + >>> df.added_property + property + + # properties defined in _internal_names is reset after manipulation + >>> df[['A', 'B']].internal_cache + AttributeError: 'SubclassedDataFrame2' object has no attribute 'internal_cache' + + # properties defined in _metadata are retained + >>> df[['A', 'B']].added_property + property diff --git a/doc/source/index.rst.template b/doc/source/index.rst.template index eff1227e98994..6e70fcd903f05 100644 --- a/doc/source/index.rst.template +++ b/doc/source/index.rst.template @@ -152,5 +152,6 @@ See the package overview for more detail about what's in the library. {% if not single_doc -%} developer internals + extending release {% endif -%} diff --git a/doc/source/internals.rst b/doc/source/internals.rst index 957f82fd9eba7..d398a72ccf357 100644 --- a/doc/source/internals.rst +++ b/doc/source/internals.rst @@ -15,7 +15,8 @@ Internals ********* -This section will provide a look into some of pandas internals. +This section will provide a look into some of pandas internals. It's primarily +intended for developers of pandas itself. Indexing -------- @@ -106,157 +107,3 @@ containers (``Index`` classes and ``Series``) we have the following convention: So, for example, ``Series[category]._values`` is a ``Categorical``, while ``Series[category]._ndarray_values`` is the underlying codes. - - -.. _ref-subclassing-pandas: - -Subclassing pandas Data Structures ----------------------------------- - -.. warning:: There are some easier alternatives before considering subclassing ``pandas`` data structures. - - 1. Extensible method chains with :ref:`pipe ` - - 2. Use *composition*. See `here `_. - - 3. Extending by :ref:`registering an accessor ` - -This section describes how to subclass ``pandas`` data structures to meet more specific needs. There are 2 points which need attention: - -1. Override constructor properties. -2. Define original properties - -.. note:: You can find a nice example in `geopandas `_ project. - -Override Constructor Properties -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Each data structure has constructor properties to specifying data constructors. By overriding these properties, you can retain defined-classes through ``pandas`` data manipulations. - -There are 3 constructors to be defined: - -- ``_constructor``: Used when a manipulation result has the same dimesions as the original. -- ``_constructor_sliced``: Used when a manipulation result has one lower dimension(s) as the original, such as ``DataFrame`` single columns slicing. -- ``_constructor_expanddim``: Used when a manipulation result has one higher dimension as the original, such as ``Series.to_frame()`` and ``DataFrame.to_panel()``. - -Following table shows how ``pandas`` data structures define constructor properties by default. - -=========================== ======================= =================== ======================= -Property Attributes ``Series`` ``DataFrame`` ``Panel`` -=========================== ======================= =================== ======================= -``_constructor`` ``Series`` ``DataFrame`` ``Panel`` -``_constructor_sliced`` ``NotImplementedError`` ``Series`` ``DataFrame`` -``_constructor_expanddim`` ``DataFrame`` ``Panel`` ``NotImplementedError`` -=========================== ======================= =================== ======================= - -Below example shows how to define ``SubclassedSeries`` and ``SubclassedDataFrame`` overriding constructor properties. - -.. code-block:: python - - class SubclassedSeries(Series): - - @property - def _constructor(self): - return SubclassedSeries - - @property - def _constructor_expanddim(self): - return SubclassedDataFrame - - class SubclassedDataFrame(DataFrame): - - @property - def _constructor(self): - return SubclassedDataFrame - - @property - def _constructor_sliced(self): - return SubclassedSeries - -.. code-block:: python - - >>> s = SubclassedSeries([1, 2, 3]) - >>> type(s) - - - >>> to_framed = s.to_frame() - >>> type(to_framed) - - - >>> df = SubclassedDataFrame({'A', [1, 2, 3], 'B': [4, 5, 6], 'C': [7, 8, 9]}) - >>> df - A B C - 0 1 4 7 - 1 2 5 8 - 2 3 6 9 - - >>> type(df) - - - >>> sliced1 = df[['A', 'B']] - >>> sliced1 - A B - 0 1 4 - 1 2 5 - 2 3 6 - >>> type(sliced1) - - - >>> sliced2 = df['A'] - >>> sliced2 - 0 1 - 1 2 - 2 3 - Name: A, dtype: int64 - >>> type(sliced2) - - -Define Original Properties -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -To let original data structures have additional properties, you should let ``pandas`` know what properties are added. ``pandas`` maps unknown properties to data names overriding ``__getattribute__``. Defining original properties can be done in one of 2 ways: - -1. Define ``_internal_names`` and ``_internal_names_set`` for temporary properties which WILL NOT be passed to manipulation results. -2. Define ``_metadata`` for normal properties which will be passed to manipulation results. - -Below is an example to define 2 original properties, "internal_cache" as a temporary property and "added_property" as a normal property - -.. code-block:: python - - class SubclassedDataFrame2(DataFrame): - - # temporary properties - _internal_names = pd.DataFrame._internal_names + ['internal_cache'] - _internal_names_set = set(_internal_names) - - # normal properties - _metadata = ['added_property'] - - @property - def _constructor(self): - return SubclassedDataFrame2 - -.. code-block:: python - - >>> df = SubclassedDataFrame2({'A': [1, 2, 3], 'B': [4, 5, 6], 'C': [7, 8, 9]}) - >>> df - A B C - 0 1 4 7 - 1 2 5 8 - 2 3 6 9 - - >>> df.internal_cache = 'cached' - >>> df.added_property = 'property' - - >>> df.internal_cache - cached - >>> df.added_property - property - - # properties defined in _internal_names is reset after manipulation - >>> df[['A', 'B']].internal_cache - AttributeError: 'SubclassedDataFrame2' object has no attribute 'internal_cache' - - # properties defined in _metadata are retained - >>> df[['A', 'B']].added_property - property From 0842fcf12b07065ca20743927b0cb81506f9e585 Mon Sep 17 00:00:00 2001 From: Tom Augspurger Date: Wed, 28 Feb 2018 06:47:51 -0600 Subject: [PATCH 2/8] Updated docs --- doc/source/ecosystem.rst | 27 +++++++++++++++++++++++++++ doc/source/extending.rst | 32 ++++++++++++++++++++++++++------ doc/source/internals.rst | 5 +++++ 3 files changed, 58 insertions(+), 6 deletions(-) diff --git a/doc/source/ecosystem.rst b/doc/source/ecosystem.rst index c770bf2851643..88981a5a33818 100644 --- a/doc/source/ecosystem.rst +++ b/doc/source/ecosystem.rst @@ -262,3 +262,30 @@ Data validation Engarde is a lightweight library used to explicitly state your assumptions abour your datasets and check that they're *actually* true. + +.. _ecosystem.extensions: + +Extension Data Types +-------------------- + +`cyberpandas`_ +~~~~~~~~~~~~~~ + +Cyberpandas provides an extension type for storing arrays of IP Addresses. These +arrays can be stored inside pandas' Series and DataFrame. + +.. _ecosystem.accessors: +------------------------ + +A directory of projects providing extension accessors. This is for users to +discover new accessors and for libraries authors to coordinate on the namespace. + +============== ========== ================= +Library Accessor Classes +============== ========== ================= +`cyberpandas`_ ``ip`` Series +`pdvega`_ ``vgplot`` Series, DataFrame +============== ========== ================= + +.. _cyberpandas: https://cyberpandas.readthedocs.io/en/latest +.. _pdvega: https://jakevdp.github.io/pdvega/ diff --git a/doc/source/extending.rst b/doc/source/extending.rst index 2778ea9867a07..c42b32fe33f95 100644 --- a/doc/source/extending.rst +++ b/doc/source/extending.rst @@ -52,18 +52,24 @@ This can be a convenient way to extend pandas objects without subclassing them. If you write a custom accessor, make a pull request adding it to our :ref:`ecosystem` page. -Extension Arrays ----------------- +.. _extending.extension-types: + +Extension Types +--------------- Pandas defines an interface for implementing data types and arrays that *extend* -NumPy's type system. Pandas uses itself for some types that aren't built into -NumPy (categorical, period, interval, datetime with timezone). +NumPy's type system. Pandas iteself uses the extension system for some types +that aren't built into NumPy (categorical, period, interval, datetime with +timezone). Libraries can define an custom array and data type. When pandas encounters these objects, they will be handled properly (i.e. not converted to an ndarray of objects). Many methods like :func:`pandas.isna` will dispatch to the extension type's implementation. +If you're building a library that implements the interface, please publicize it +on :ref:`ecosystem.extensions`. + The interface consists of two classes. ``ExtensionDtype`` @@ -81,10 +87,24 @@ See ``pandas/core/dtypes/base.py`` for interface definition. ``ExtensionArray`` """""""""""""""""" -This is the main object. +This class provides all the array-like functionality. ExtensionArrays are +limited to 1 dimension. An ExtensionArray is linked to an ExtensionDtype via the +``dtype`` attribute. + +Pandas makes no restrictions on how an extension array is created via its +``__new__`` or ``__init__``, and puts no restrictions on how you store your +data. We do require that your array be convertible to a NumPy array, even if +this is relatively expensive (as it is for ``Categorical``). +They may be backed by none, one, or many NumPy ararys. For example, +``pandas.Categorical`` is an extension array backed by two arrays, +one for codes and one for categories. An array of IPv6 address may +be backed by a NumPy structured array with two fields, one for the +lower 64 bits and one for the upper 64 bits. Or they may be backed +by some other storage type, like Python lists. -See ``pandas/core/arrays/base.py`` for the interface definition. +See ``pandas/core/arrays/base.py`` for the interface definition. The docstrings +and comments contain guidance for properly implementing the interface. .. _ref-subclassing-pandas: diff --git a/doc/source/internals.rst b/doc/source/internals.rst index d398a72ccf357..e527f6f4544a1 100644 --- a/doc/source/internals.rst +++ b/doc/source/internals.rst @@ -107,3 +107,8 @@ containers (``Index`` classes and ``Series``) we have the following convention: So, for example, ``Series[category]._values`` is a ``Categorical``, while ``Series[category]._ndarray_values`` is the underlying codes. + +Subclassing pandas Data Structures +---------------------------------- + +This section has been moved to :ref:`ref-subclassing-pandas`. From e1f3f6022682d6cad38f3a4061f7e6afd3901859 Mon Sep 17 00:00:00 2001 From: Tom Augspurger Date: Wed, 28 Feb 2018 07:04:55 -0600 Subject: [PATCH 3/8] More text, fixed header --- doc/source/ecosystem.rst | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/doc/source/ecosystem.rst b/doc/source/ecosystem.rst index 88981a5a33818..9cd320ddf3640 100644 --- a/doc/source/ecosystem.rst +++ b/doc/source/ecosystem.rst @@ -268,6 +268,11 @@ and check that they're *actually* true. Extension Data Types -------------------- +Pandas provides an interface for defining +:ref:`extending.extension-types ` to extend NumPy's type +system. The following libraries implement that interface to provide types not +found in NumPy or pandas, which work well with pandas' data containers. + `cyberpandas`_ ~~~~~~~~~~~~~~ @@ -275,9 +280,12 @@ Cyberpandas provides an extension type for storing arrays of IP Addresses. These arrays can be stored inside pandas' Series and DataFrame. .. _ecosystem.accessors: ------------------------- -A directory of projects providing extension accessors. This is for users to +Accessors +--------- + +A directory of projects providing +:ref:`extending.register-accessors `. This is for users to discover new accessors and for libraries authors to coordinate on the namespace. ============== ========== ================= From 057a4f95530e88b9f1135fa2a2b756b128a4a8ef Mon Sep 17 00:00:00 2001 From: Tom Augspurger Date: Wed, 28 Feb 2018 17:05:01 -0600 Subject: [PATCH 4/8] Fixed links --- doc/source/ecosystem.rst | 4 ++-- doc/source/extending.rst | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/source/ecosystem.rst b/doc/source/ecosystem.rst index 9cd320ddf3640..dc3cc286a9288 100644 --- a/doc/source/ecosystem.rst +++ b/doc/source/ecosystem.rst @@ -269,7 +269,7 @@ Extension Data Types -------------------- Pandas provides an interface for defining -:ref:`extending.extension-types ` to extend NumPy's type +:ref:`extension types ` to extend NumPy's type system. The following libraries implement that interface to provide types not found in NumPy or pandas, which work well with pandas' data containers. @@ -285,7 +285,7 @@ Accessors --------- A directory of projects providing -:ref:`extending.register-accessors `. This is for users to +:ref:`extension accessors `. This is for users to discover new accessors and for libraries authors to coordinate on the namespace. ============== ========== ================= diff --git a/doc/source/extending.rst b/doc/source/extending.rst index c42b32fe33f95..545c4bb86a2f1 100644 --- a/doc/source/extending.rst +++ b/doc/source/extending.rst @@ -117,7 +117,7 @@ Subclassing pandas Data Structures 2. Use *composition*. See `here `_. - 3. Extending by :ref:`registering an accessor ` + 3. Extending by :ref:`registering an accessor ` This section describes how to subclass ``pandas`` data structures to meet more specific needs. There are 2 points which need attention: From 51097c625d6d03648e3268f9f6534ad02e01b87e Mon Sep 17 00:00:00 2001 From: Tom Augspurger Date: Thu, 1 Mar 2018 07:05:19 -0600 Subject: [PATCH 5/8] Updates --- doc/source/ecosystem.rst | 12 +++++----- doc/source/extending.rst | 52 +++++++++++++++++++++++----------------- pandas/core/accessor.py | 6 ++--- 3 files changed, 39 insertions(+), 31 deletions(-) diff --git a/doc/source/ecosystem.rst b/doc/source/ecosystem.rst index dc3cc286a9288..30cdb06b28487 100644 --- a/doc/source/ecosystem.rst +++ b/doc/source/ecosystem.rst @@ -286,14 +286,14 @@ Accessors A directory of projects providing :ref:`extension accessors `. This is for users to -discover new accessors and for libraries authors to coordinate on the namespace. +discover new accessors and for library authors to coordinate on the namespace. -============== ========== ================= +============== ========== ========================= Library Accessor Classes -============== ========== ================= -`cyberpandas`_ ``ip`` Series -`pdvega`_ ``vgplot`` Series, DataFrame -============== ========== ================= +============== ========== ========================= +`cyberpandas`_ ``ip`` ``Series`` +`pdvega`_ ``vgplot`` ``Series``, ``DataFrame`` +============== ========== ========================= .. _cyberpandas: https://cyberpandas.readthedocs.io/en/latest .. _pdvega: https://jakevdp.github.io/pdvega/ diff --git a/doc/source/extending.rst b/doc/source/extending.rst index 545c4bb86a2f1..7f771f101e72e 100644 --- a/doc/source/extending.rst +++ b/doc/source/extending.rst @@ -18,8 +18,8 @@ Libraries can use the decorators :func:`pandas.api.extensions.register_series_accessor`, and :func:`pandas.api.extensions.register_index_accessor`, to add additional "namespaces" to pandas objects. All of these follow a similar convention: you -decorate a class, providing the name of attribute to add. The class's `__init__` -method gets the object being decorated. For example: +decorate a class, providing the name of attribute to add. The class's +``__init__`` method gets the object being decorated. For example: .. code-block:: python @@ -30,16 +30,16 @@ method gets the object being decorated. For example: @property def center(self): - # return the geographic center point of this DataFarme - lon = self._obj.latitude - lat = self._obj.longitude + # return the geographic center point of this DataFrame + lat = self._obj.latitude + lon = self._obj.longitude return (float(lon.mean()), float(lat.mean())) def plot(self): # plot this array's data on a map, e.g., using Cartopy pass -Now users can access your methods using the `geo` namespace: +Now users can access your methods using the ``geo`` namespace: >>> ds = pd.DataFrame({'longitude': np.linspace(0, 10), ... 'latitude': np.linspace(0, 20)}) @@ -58,7 +58,7 @@ Extension Types --------------- Pandas defines an interface for implementing data types and arrays that *extend* -NumPy's type system. Pandas iteself uses the extension system for some types +NumPy's type system. Pandas itself uses the extension system for some types that aren't built into NumPy (categorical, period, interval, datetime with timezone). @@ -82,7 +82,7 @@ One particularly important item is the ``type`` property. This should be the class that is the scalar type for your data. For example, if you were writing an extension array for IP Address data, this might be ``ipaddress.IPv4Address``. -See ``pandas/core/dtypes/base.py`` for interface definition. +See the `extension dtype source`_ for interface definition. ``ExtensionArray`` """""""""""""""""" @@ -96,16 +96,20 @@ Pandas makes no restrictions on how an extension array is created via its data. We do require that your array be convertible to a NumPy array, even if this is relatively expensive (as it is for ``Categorical``). -They may be backed by none, one, or many NumPy ararys. For example, +They may be backed by none, one, or many NumPy arrays. For example, ``pandas.Categorical`` is an extension array backed by two arrays, -one for codes and one for categories. An array of IPv6 address may +one for codes and one for categories. An array of IPv6 addresses may be backed by a NumPy structured array with two fields, one for the lower 64 bits and one for the upper 64 bits. Or they may be backed by some other storage type, like Python lists. -See ``pandas/core/arrays/base.py`` for the interface definition. The docstrings +See the `extension array source`_ for the interface definition. The docstrings and comments contain guidance for properly implementing the interface. + +.. _extension dtype source: https://github.com/pandas-dev/pandas/blob/master/pandas/core/dtypes/base.py +.. _extension array source: https://github.com/pandas-dev/pandas/blob/master/pandas/core/arrays/base.py + .. _ref-subclassing-pandas: Subclassing pandas Data Structures @@ -119,7 +123,9 @@ Subclassing pandas Data Structures 3. Extending by :ref:`registering an accessor ` -This section describes how to subclass ``pandas`` data structures to meet more specific needs. There are 2 points which need attention: + 4. Extending by :ref:`extension type ` + +This section describes how to subclass ``pandas`` data structures to meet more specific needs. There are two points that need attention: 1. Override constructor properties. 2. Define original properties @@ -129,9 +135,11 @@ This section describes how to subclass ``pandas`` data structures to meet more s Override Constructor Properties ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Each data structure has constructor properties to specifying data constructors. By overriding these properties, you can retain defined-classes through ``pandas`` data manipulations. +Each data structure has several *constructor properties* for returning a new +data structure as the result of an operation. By overriding these properties, +you can retain subclasses through ``pandas`` data manipulations. -There are 3 constructors to be defined: +There are 3 constructor properties to be defined: - ``_constructor``: Used when a manipulation result has the same dimesions as the original. - ``_constructor_sliced``: Used when a manipulation result has one lower dimension(s) as the original, such as ``DataFrame`` single columns slicing. @@ -139,13 +147,13 @@ There are 3 constructors to be defined: Following table shows how ``pandas`` data structures define constructor properties by default. -=========================== ======================= =================== ======================= -Property Attributes ``Series`` ``DataFrame`` ``Panel`` -=========================== ======================= =================== ======================= -``_constructor`` ``Series`` ``DataFrame`` ``Panel`` -``_constructor_sliced`` ``NotImplementedError`` ``Series`` ``DataFrame`` -``_constructor_expanddim`` ``DataFrame`` ``Panel`` ``NotImplementedError`` -=========================== ======================= =================== ======================= +=========================== ======================= ============= +Property Attributes ``Series`` ``DataFrame`` +=========================== ======================= ============= +``_constructor`` ``Series`` ``DataFrame`` +``_constructor_sliced`` ``NotImplementedError`` ``Series`` +``_constructor_expanddim`` ``DataFrame`` ``Panel`` +=========================== ======================= ============= Below example shows how to define ``SubclassedSeries`` and ``SubclassedDataFrame`` overriding constructor properties. @@ -217,7 +225,7 @@ To let original data structures have additional properties, you should let ``pan 1. Define ``_internal_names`` and ``_internal_names_set`` for temporary properties which WILL NOT be passed to manipulation results. 2. Define ``_metadata`` for normal properties which will be passed to manipulation results. -Below is an example to define 2 original properties, "internal_cache" as a temporary property and "added_property" as a normal property +Below is an example to define two original properties, "internal_cache" as a temporary property and "added_property" as a normal property .. code-block:: python diff --git a/pandas/core/accessor.py b/pandas/core/accessor.py index 96bf628c8d7ff..06c4068f86bfe 100644 --- a/pandas/core/accessor.py +++ b/pandas/core/accessor.py @@ -191,9 +191,9 @@ def __init__(self, pandas_obj): @property def center(self): - # return the geographic center point of this DataFarme - lon = self._obj.latitude - lat = self._obj.longitude + # return the geographic center point of this DataFrame + lat = self._obj.latitude + lon = self._obj.longitude return (float(lon.mean()), float(lat.mean())) def plot(self): From 10a25cfa7e6ac858239fea9a1ecdb94f02ab0f58 Mon Sep 17 00:00:00 2001 From: Tom Augspurger Date: Thu, 1 Mar 2018 11:37:07 -0600 Subject: [PATCH 6/8] Updated link labels --- doc/source/extending.rst | 5 ++--- doc/source/internals.rst | 4 +++- doc/source/whatsnew/v0.16.1.txt | 2 +- 3 files changed, 6 insertions(+), 5 deletions(-) diff --git a/doc/source/extending.rst b/doc/source/extending.rst index 7f771f101e72e..a7d7f5a289df8 100644 --- a/doc/source/extending.rst +++ b/doc/source/extending.rst @@ -62,7 +62,7 @@ NumPy's type system. Pandas itself uses the extension system for some types that aren't built into NumPy (categorical, period, interval, datetime with timezone). -Libraries can define an custom array and data type. When pandas encounters these +Libraries can define a custom array and data type. When pandas encounters these objects, they will be handled properly (i.e. not converted to an ndarray of objects). Many methods like :func:`pandas.isna` will dispatch to the extension type's implementation. @@ -106,11 +106,10 @@ by some other storage type, like Python lists. See the `extension array source`_ for the interface definition. The docstrings and comments contain guidance for properly implementing the interface. - .. _extension dtype source: https://github.com/pandas-dev/pandas/blob/master/pandas/core/dtypes/base.py .. _extension array source: https://github.com/pandas-dev/pandas/blob/master/pandas/core/arrays/base.py -.. _ref-subclassing-pandas: +.. _extending.subclassing-pandas: Subclassing pandas Data Structures ---------------------------------- diff --git a/doc/source/internals.rst b/doc/source/internals.rst index e527f6f4544a1..b120e3a98db7f 100644 --- a/doc/source/internals.rst +++ b/doc/source/internals.rst @@ -108,7 +108,9 @@ containers (``Index`` classes and ``Series``) we have the following convention: So, for example, ``Series[category]._values`` is a ``Categorical``, while ``Series[category]._ndarray_values`` is the underlying codes. +.. _ref-subclassing-pandas: + Subclassing pandas Data Structures ---------------------------------- -This section has been moved to :ref:`ref-subclassing-pandas`. +This section has been moved to :ref:`extending.subclassing-pandas`. diff --git a/doc/source/whatsnew/v0.16.1.txt b/doc/source/whatsnew/v0.16.1.txt index b1e8aa10457f8..9e1dc391d7ace 100644 --- a/doc/source/whatsnew/v0.16.1.txt +++ b/doc/source/whatsnew/v0.16.1.txt @@ -313,7 +313,7 @@ Other Enhancements - Add/delete ``str/dt/cat`` accessors dynamically from ``__dir__``. (:issue:`9910`) - Add ``normalize`` as a ``dt`` accessor method. (:issue:`10047`) -- ``DataFrame`` and ``Series`` now have ``_constructor_expanddim`` property as overridable constructor for one higher dimensionality data. This should be used only when it is really needed, see :ref:`here ` +- ``DataFrame`` and ``Series`` now have ``_constructor_expanddim`` property as overridable constructor for one higher dimensionality data. This should be used only when it is really needed, see :ref:`here ` - ``pd.lib.infer_dtype`` now returns ``'bytes'`` in Python 3 where appropriate. (:issue:`10032`) From 59f83ce8f610512af78facfc2b07e42f073abc74 Mon Sep 17 00:00:00 2001 From: Tom Augspurger Date: Thu, 1 Mar 2018 13:30:29 -0600 Subject: [PATCH 7/8] Title level --- doc/source/extending.rst | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/doc/source/extending.rst b/doc/source/extending.rst index a7d7f5a289df8..b4c0164c596e6 100644 --- a/doc/source/extending.rst +++ b/doc/source/extending.rst @@ -129,10 +129,12 @@ This section describes how to subclass ``pandas`` data structures to meet more s 1. Override constructor properties. 2. Define original properties -.. note:: You can find a nice example in `geopandas `_ project. +.. note:: + + You can find a nice example in `geopandas `_ project. Override Constructor Properties -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Each data structure has several *constructor properties* for returning a new data structure as the result of an operation. By overriding these properties, @@ -217,7 +219,7 @@ Below example shows how to define ``SubclassedSeries`` and ``SubclassedDataFrame Define Original Properties -~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^ To let original data structures have additional properties, you should let ``pandas`` know what properties are added. ``pandas`` maps unknown properties to data names overriding ``__getattribute__``. Defining original properties can be done in one of 2 ways: From 87aca055ea8111ad304e921e49143734a2d71c3e Mon Sep 17 00:00:00 2001 From: Tom Augspurger Date: Fri, 2 Mar 2018 06:41:24 -0600 Subject: [PATCH 8/8] DOC: Fix levels --- doc/source/extending.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/source/extending.rst b/doc/source/extending.rst index b4c0164c596e6..25c4ba4a4a2a3 100644 --- a/doc/source/extending.rst +++ b/doc/source/extending.rst @@ -73,7 +73,7 @@ on :ref:`ecosystem.extensions`. The interface consists of two classes. ``ExtensionDtype`` -"""""""""""""""""" +^^^^^^^^^^^^^^^^^^ An ``ExtensionDtype`` is similar to a ``numpy.dtype`` object. It describes the data type. Implementors are responsible for a few unique items like the name. @@ -85,7 +85,7 @@ extension array for IP Address data, this might be ``ipaddress.IPv4Address``. See the `extension dtype source`_ for interface definition. ``ExtensionArray`` -"""""""""""""""""" +^^^^^^^^^^^^^^^^^^ This class provides all the array-like functionality. ExtensionArrays are limited to 1 dimension. An ExtensionArray is linked to an ExtensionDtype via the