[SPARK-10162] [SQL] Fix the timezone omitting for PySpark Dataframe filter function

[0x0FFF]

This PR addresses SPARK-10162
The issue is with DataFrame filter() function, if datetime.datetime is passed to it:

• Timezone information of this datetime is ignored
• This datetime is assumed to be in local timezone, which depends on the OS timezone setting

Fix includes both code change and regression test. Problem reproduction code on master:

import pytz
from datetime import datetime
from pyspark.sql import *
from pyspark.sql.types import *
sqc = SQLContext(sc)
df = sqc.createDataFrame([], StructType([StructField("dt", TimestampType())]))

m1 = pytz.timezone('UTC')
m2 = pytz.timezone('Etc/GMT+3')

df.filter(df.dt > datetime(2000, 01, 01, tzinfo=m1)).explain()
df.filter(df.dt > datetime(2000, 01, 01, tzinfo=m2)).explain()

It gives the same timestamp ignoring time zone:

>>> df.filter(df.dt > datetime(2000, 01, 01, tzinfo=m1)).explain()
Filter (dt#0 > 946713600000000)
 Scan PhysicalRDD[dt#0]

>>> df.filter(df.dt > datetime(2000, 01, 01, tzinfo=m2)).explain()
Filter (dt#0 > 946713600000000)
 Scan PhysicalRDD[dt#0]

After the fix:

>>> df.filter(df.dt > datetime(2000, 01, 01, tzinfo=m1)).explain()
Filter (dt#0 > 946684800000000)
 Scan PhysicalRDD[dt#0]

>>> df.filter(df.dt > datetime(2000, 01, 01, tzinfo=m2)).explain()
Filter (dt#0 > 946695600000000)
 Scan PhysicalRDD[dt#0]

PR 8536 was occasionally closed by me dropping the repo

[JoshRosen]

Jenkins, this is ok to test.

[SparkQA]

Test build #41876 has finished for PR 8555 at commit 610cb3f.

• This patch fails Python style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

@davies

[davies]

Existing: We should include microseconds by

t = Timestamp(int(seconds) * 1000)
t.setNanos(obj.microsecond * 1000)
return t

[0x0FFF]

Fixed in commit 2acd285

[SparkQA]

Test build #41880 has finished for PR 8555 at commit cd63eb0.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #41885 has finished for PR 8555 at commit 2acd285.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[davies]

LGTM, merging this into master.