DEPR: let's deprecate

[jreback]

xref #18202

We have some cruft, let's deprecate it (I have noted some which already have an issue associated).

From ndarray

• Series.compress() (DEPR: Series.compress #21930)
• Series.imag / Series.real (DEPR: Series.put, Series.real, Series.imag, Index.dtype_str #27106)
• Series.nonzero() (Deprecate series.nonzero (GH18262) #24048)
• Series.put() (DEPR: Series.put, Series.real, Series.imag, Index.dtype_str #27106)
• Series.itemsize
• Series.flags
• Series.strides

timeseries specific

• Series.first()
  let head / tail take a timedelta
• Series.last()

other

• Series.swapaxes()
  • No reason once panel is gone.

non-controversial

• MultiIndex.to_hierarchical (only used for Panel) (DEPR: MultiIndex.to_hierarchical #21613)
• Series/DataFrame.compound() (DEPR: deprecate Series/DataFrame.compound #26405)
• Series.ptp() (DEPR: Series.ptp() #21614)
• Series.from_array (Deprecate Series.from_array ? #18213)
• Series.valid() (DEPR: Deprecate Series.valid #18800)
• Series/DataFrame.slice_shift() (DEPR: DataFrame/Series.slice_shift #37601)
• Series/DataFrame.tshift() (DEPR: Deprecate tshift and integrate it to shift #34545)
• Series/DataFrame.get_values() (DEPR: .get_values() #19617)
• Index.dtype_str (DEPR: Series.put, Series.real, Series.imag, Index.dtype_str #27106)
• Index.summary() (Deprecate Index.summary #18217)
• .get_ftype_counts (DEPR: deprecate get_ftype_counts #18243) (DEPR: deprecate get_ftype_counts (GH18243) #20404)
• .get_dtype_counts DEPR: DataFrame.get_dtype_counts #27145
• Index/Series.asobject (Deprecate Series.asobject, Index.asobject, rename to _asobject #18237) (DEPR: deprecate .asobject property #18572)
• Index.to_native_types() (make private) (DEPR: Index.to_native_types #36418)
• DataFrame/Series.as_matrix (DEPR: Deprecate NDFrame.as_matrix #18458)
• .clip_upper/.clip_lower (replace by .clip) (DEPR: Deprecating .clip_lower and .clip_upper #24203)

Potentially

• .ftypes (DEPR: deprecate get_ftype_counts #18243) (Deprecated Series.ftype, Series.ftypes and DataFrame.ftypes features #26744)
• .xs() (CLN: implement xs in terms of loc #6249)
• .iat/.at
• .take
• .lookup (DEPR: DataFrame.lookup #35224)
• DataFrame.from_items
• Series/DataFrame.add_prefix/add_suffix (DEPR: deprecate .add_prefix and .add_suffix #18347)
  • Maybe add suffix / prefix to concat?
• NDFrame.filter

[jreback]

cc @jorisvandenbossche @TomAugspurger if any comments / objections pls note and I will update the top section.

[TomAugspurger]

I think I'm -1 on deprecating xs.

-0 on deprecating ptp

What's the alternative to tshift? I think that's sometimes useful when a shift won't quite work.

[jorisvandenbossche]

I was just writing up a similar issue :-) (but only for Series, as by fixing the api docs and docstrings, I bumped into quite some methods unknown to me)

Overview of methods that could be considered for removal (note that this list is very long, and are methods that I would not miss if they are gone, which does not mean that they are not useful to others, it's just for stirring discussion):

• Related to the original ndarray subclassing:

  • Series.compress()
  • Series.flags
  • Series.imag / Series.real
  • Series.item()
  • Series.itemsize
  • Series.nonzero()
  • Series.put()
  • Series.strides
  • Series.ptp

• Time series specific ones (the question here is if they are all worth it as method, while very specific in application):

  • Series.at_time()
  • Series.first()
  • Series.last()
  • Series.between_time()
  • Series.tshift()

• Finance? specific

  • Series.compound()

• Other:

  • Series.asobject
  • Series.as_matrix
  • Series.between()
  • Series.first_valid_index() / Series.last_valid_index()
  • Series.from_array()
  • Series.slice_shift()
  • Series.swapaxes()
  • Series.truncate()
  • Series.valid()

[jorisvandenbossche]

What's the alternative to tshift? I think that's sometimes useful when a shift won't quite work.

shift already seems to have a freq keyword as well, and it dispatches to tshift if freq is specified

Note that my above list is very long. The more obvious ones to me that are not yet in the list in the top post are: as_matrix (and maybe swapaxes ?)

[topper-123]

Could .add_prefix and .add_suffix be added to the deprecation list?

The dataframe/Series namespace is huge and cutting down can make the API easier to grasp. I would also think it more logical and idiomatic to operate directly on the columns, rather than on the dataframe.

[jreback]

@topper-123 added, feel free to submit PR's for any of these!

[manrajgrover]

@jorisvandenbossche @jreback I would love to submit PRs for this issue. How can I go about in deprecating these APIs?

[jreback]

see for example #18258

[jorisvandenbossche]

@manrajgrover Best first post a comment here with which one you would start doing, as I think not all those listed above are uncontroversial.

[topper-123]

I think .compound, while not very useful ATM, could be more useful if it was cumulative, ie. just use .cumprod instead of .prod and return a series:

>>> s = pd.Series([0.2, 0.2, 0.2])
>>> s.compound()  # essentialy the same as (s+1).cumprod() - 1
0    0.2000
1    0.440
2    0.728
dtype: float64

The above would play excellently together with .pct_change, so data.pct_change().compound() would read really well and be very useful in many use cases.

any opinions if .compound could be changed like above rather than deprecated? If .compound returns a scalar as today, I agree it should be deprecated.

[topper-123]

I've started a PR for add_prefix and add_postfix.

I will take on Series.asobject and NDFrame.as_matrix next, unless @manrajgrover wants to to them, in which case you'll be welcome.

And yes, I like to remove superfluous methods that start with a, as these are so visible then tab-completing in the REPL :-)

[jorisvandenbossche]

Although I never use add_prefix / add_suffix, I think they are quite used a bit (looking at the number of stackoverflow questions), so I am not yet fully convinced they are ok to deprecate.
So I would rather already start with the others like asobject, as_matrix, valid, tshift, ..

as_matrix is also used a bit (more than the others mentioned in the list above), but because it is a very confusing name for what it does, I think it would be good to deprecate.

[manrajgrover]

@jorisvandenbossche I can start with Index.summary() for now and pick the next one from the following list:

• Index.dtype_str
• .ftypes/.get_ftype_counts (DEPR: deprecate get_ftype_counts #18243)
• Index/Series.asobject (Deprecate Series.asobject, Index.asobject, rename to _asobject #18237)
• Index.to_native_types() (make private)

[topper-123]

@jorisvandenbossche, what about removing a prefix/suffix or doing any other transformation you'd want to do on a index? My point is that .add_prefix/.add_suffix are way too specialized methods, and pandas should have methods that are more generally useful. .rename is great in that respect, and should be the canonical method for changing axis values.

I've already made a proposal for .add_prefix/.add_suffix (#18347), so that can wait to see what the agreement will be on that. I would appreciate input though, if you see anything obvoíous, as that is the first deprerecation PR I've made.

In the same vein, the difference between .as_matrix and .values is miniscule to the point where df[columns].values is the same is df.as_matrix(columns). pandas will be cleaner and leaner with only one way to archieve such a common result (obviously so, IMO...).

I'll make a PR for as_matrix, as there seems to be agreement on that.

[jorisvandenbossche]

My point is that .add_prefix/.add_suffix are way too specialized methods, and pandas should have methods that are more generally useful. .rename is great in that respect, and should be the canonical method for changing axis values.

I completely agree with this. But, you also have the fact that people are using it and thus a removal will cause inconvenience / break code. So it is always a balance between both.

I am certainly +1 on deprecating as_matrix. As you say this is almost exactly the same as .values, and although I think this method is also used quite a bit (the argument I use for add_prefix ..), it's an awfully confusing name, so that's for me an extra reason to deprecate it.

[tdpetrou]

Here are my deprecation suggestions:

• I'd like to see read_table deprecated. Its the exact same as read_csv with tab delimiter
• remove get_dtype_counts/get_ftype_counts - these are just convenience for DataFrame.dtypes.value_counts
• remove the indexers iat/at. They give a small performance boost for an increase in API complexity
• Use one of iterrows/itertuples to iterate over rows
• remove lookup and take - other indexers do the same thing
• remove combine - never used it and almost no use on SO. Looks to do nothing more than DataFrame.add
• Probably get rid of applymap - should do the same thing with apply and then map inside of it
• Use only agg not aggregate
• Remove clip_upper and clip_lower and keep DataFrame.clip for both
• Combine add_prefix/add_suffix into one method
• One of the biggest issues are the methods that work only with DataFrames with a DatetimeIndex - first, last, truncate, at_time, between_time, to_period, to_timestamp - These could be removed or put in an accessor
• Remove reindex_axis and reindex_like in favor of just reindex
• remove isna - its an alias to isnull

[jreback]

remove the indexers iat/at. They give a small performance boost for an increase in API complexity

these are convenience methods, not sure they add much to API burden

take

this is a very common notion and is a very array-like method

remove isna - its an alias to isnull

this was just added for compat with dropna, fillna, see the pattern :>, so if anything we would remove isnull, but that has been in the API so long that it may well nigh be impossible to actually remove (and more to the point very annoying).

[jorisvandenbossche]

Remove reindex_axis and reindex_like in favor of just reindex

reindex_axis is already deprecated.

One of the biggest issues are the methods that work only with DataFrames with a DatetimeIndex - first, last, truncate, at_time, between_time, to_period, to_timestamp - These could be removed or put in an accessor

to_period and to_timestamp on a series do something else than the methods in the .dt accessor. The former work on the index, the latter on the values. So it's not possible to just move them.
But on the other datetime-related I agree, I also find it a bit unfortunate that those exist (certainly first and last are very confusing in naming)

[tdpetrou]

@jreback There are only a total of 13 occurrences of df.take in all of Stack Overflow and in my opinion should never be used.

df.iat/.at are probably too entrenched in legacy code to remove but they provide no extra functionality. Indexing is the most confusing aspect to pandas and the less the better. Maybe a better design would have been to do df.loc(type='scalar')['row', 'col']

I guess there is no going back on isna/isnull but I really dislike having methods that are aliases of one another.

@jorisvandenbossche I wasn't being clear, but all those DataFrame/Series methods that only work on DateTimeIndexes could be put in their own accesor (not .dt) but I don't think even that would be a good idea. Perhaps just deprecating all of them would be best.

[jreback]

@jreback There are only a total of 13 occurrences of df.take in all of Stack Overflow and in my opinion should never be used.

.take() is a common name for array-like things.

df.iat/.at are probably too entrenched in legacy code to remove but they provide no extra functionality. Indexing is the most confusing aspect to pandas and the less the better. Maybe a better design would have been to do df.loc(type='scalar')['row', 'col']

your suggestion is much less readable

I guess there is no going back on isna/isnull but I really dislike having methods that are aliases of one another.

sure, but isnull is even more entrenched than anything else.

[max-sixty]

df.iat/.at are probably too entrenched in legacy code to remove but they provide no extra functionality. Indexing is the most confusing aspect to pandas and the less the better. Maybe a better design would have been to do df.loc(type='scalar')['row', 'col']

As indexing is simplified & improved, the speed diff between .loc and .at should fall (a lot of the time it's a very similar function). Then deprecating .at will cause less strife

[topper-123]

DataFrames have a method .boxplot. I would assume this should be deprecated and people should use .plot.box instead?

[jreback]

yes i think there is an issue about this

[TomAugspurger]

The two methods aren’t quite equivalent iirc.

…

________________________________ From: Jeff Reback <notifications@github.com> Sent: Saturday, November 25, 2017 10:23:01 PM To: pandas-dev/pandas Cc: Tom Augspurger; Mention Subject: Re: [pandas-dev/pandas] DEPR: let's deprecate (#18262) yes i think there is an issue about this — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#18262 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABQHItMklrTp28DbdKXtQmYS9WIL_7sKks5s6NmVgaJpZM4Qb0GL>.

[jimmywan]

Why on earth would you deprecate read_table? That makes no damn sense.
The suggested change is to call read_csv to read things that are not comma-separated? This is 100% backwards.

[st-bender]

One could argue why not deprecate read_csv() instead of read_table() since table sounds more flexible.

Edit:
I have to agree with @jimmywan here, and if they are basically the same, why not at least keep it as an alias? One could always wrap it, but people would not be confused or avoid updating.

[ghost]

DataFrame.where and DataFrame.mask are duals, but their names don't indicate that. perhaps deprecate mask? since mask is just where(~cond), IIUC. alternatively rename to where_not.

[jreback]

cc @jbrockmendel here is my original list :->

[alexiswl]

Would it be possible to get a rationale to why .item() has been deprecated or a suggested alternative? I really appreciated this feature as it would assume one and only one match to a query/column combo. i.e pd.query("Country=='%s' % country)['CapitalCity'].unique().item().

Using the suggest next(iter, None)) feature as suggested in the link below is very readable, but breaks the assumption that there are not multiple matches to a query and returns only the first value, meaning I'd need to do add a check-query-length==1 prior to extracting the value.

https://stackoverflow.com/questions/57390363/pandas-item-has-been-deprecated

[jreback]

.unique() currently returns an ndarray so .item() would still be valid

this is for .item() on Series (and Index)

[alexiswl]

Thanks! I'll put in unique() to all pd.query("row_name=='var')[col_name] as a midfix to resolve the deprecation warning.

[erfannariman]

Do we want to deprecate iat / at? These are under "potentially" now. I would be +1, iloc and loc can be used.

[mroeschke]

Appears that most of the functions described in the OP have been tackled. Further discussion on other specific APIs that should be deprecated would be better suited to individual issues. Closing for now.