[SPARK-22313][PYTHON] Mark/print deprecation warnings as DeprecationWarning for deprecated APIs

[HyukjinKwon]

What changes were proposed in this pull request?

This PR proposes to mark the existing warnings as DeprecationWarning and print out warnings for deprecated functions.

This could be actually useful for Spark app developers. I use (old) PyCharm and this IDE can detect this specific DeprecationWarning in some cases:

Before

After

For console usage, DeprecationWarning is usually disabled (see https://docs.python.org/2/library/warnings.html#warning-categories and https://docs.python.org/3/library/warnings.html#warning-categories):

>>> import warnings
>>> filter(lambda f: f[2] == DeprecationWarning, warnings.filters)
[('ignore', <_sre.SRE_Pattern object at 0x10ba58c00>, <type 'exceptions.DeprecationWarning'>, <_sre.SRE_Pattern object at 0x10bb04138>, 0), ('ignore', None, <type 'exceptions.DeprecationWarning'>, None, 0)]

so, it won't actually mess up the terminal much unless it is intended.

If this is intendedly enabled, it'd should as below:

>>> import warnings
>>> warnings.simplefilter('always', DeprecationWarning)
>>>
>>> from pyspark.sql import functions
>>> functions.approxCountDistinct("a")
.../spark/python/pyspark/sql/functions.py:232: DeprecationWarning: Deprecated in 2.1, use approx_count_distinct instead.
  "Deprecated in 2.1, use approx_count_distinct instead.", DeprecationWarning)
...

These instances were found by:

cd python/pyspark
grep -r "Deprecated" .
grep -r "deprecated" .
grep -r "deprecate" .

How was this patch tested?

Manually tested.

[HyukjinKwon]

@holdenk and @rxin, I didn't open a JIRA yet and I want to have your opinions first. Do you guys like this idea?

[srowen]

Seems OK to me if it's done consistently and actively helps developers see this info in Python

[HyukjinKwon]

Let me probably open up a JIRA and link it.

[HyukjinKwon]

Another example:

[HyukjinKwon]

Here, I intendedly avoided *args and **kwargs to keep the argument signature printed in pydoc, help(...).

[SparkQA]

Test build #82906 has finished for PR 19535 at commit 376162a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

Looks good at high level.

[felixcheung]

LGTM, minor comment

[felixcheung]

for these, could you provide more information? link to the doc on deprecating DStream in python?

[felixcheung]

ditto here

[HyukjinKwon]

Thanks @felixcheung. Will update soon.

[SparkQA]

Test build #82956 has finished for PR 19535 at commit f8a7d2b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #82957 has finished for PR 19535 at commit 5ac0697.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

Thanks @srowen, @rxin and @felixcheung.

[HyukjinKwon]

Merged to master

[yhuai]

Seems warnings is not imported in this file?

[HyukjinKwon]

Yes, it is not. Will make a followup after double checking other files too. Thank you.

[yhuai]

thank you :) It will be good to also check why master build does not fail since python should complain about it.

[HyukjinKwon]

Sure, I took a quick look and I think this one is actually not being tested and seems that's why .. will double check and take a closer look tonight (KST).

I have seen few mistakes about this so far and .. I am working on Python coverage BTW - https://issues.apache.org/jira/browse/SPARK-7721

Anyway, it was my stupid mistake. Thanks ..