Skip to content

Commit 2aad2d3

Browse files
HyukjinKwonrxin
authored andcommitted
[SPARK-12315][SQL] isnotnull operator not pushed down for JDBC datasource.
https://issues.apache.org/jira/browse/SPARK-12315 `IsNotNull` filter is not being pushed down for JDBC datasource. It looks it is SQL standard according to [SQL-92](http://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt), SQL:1999, [SQL:2003](http://www.wiscorp.com/sql_2003_standard.zip) and [SQL:201x](http://www.wiscorp.com/sql20nn.zip) and I believe most databases support this. In this PR, I simply added the case for `IsNotNull` filter to produce a proper filter string. Author: hyukjinkwon <gurwls223@gmail.com> This patch had conflicts when merged, resolved by Committer: Reynold Xin <rxin@databricks.com> Closes #10287 from HyukjinKwon/SPARK-12315.
1 parent 7f443a6 commit 2aad2d3

File tree

2 files changed

+3
-0
lines changed

2 files changed

+3
-0
lines changed

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -287,6 +287,7 @@ private[sql] class JDBCRDD(
287287
case LessThanOrEqual(attr, value) => s"$attr <= ${compileValue(value)}"
288288
case GreaterThanOrEqual(attr, value) => s"$attr >= ${compileValue(value)}"
289289
case IsNull(attr) => s"$attr IS NULL"
290+
case IsNotNull(attr) => s"$attr IS NOT NULL"
290291
case _ => null
291292
}
292293

sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -183,6 +183,8 @@ class JDBCSuite extends SparkFunSuite with BeforeAndAfter with SharedSQLContext
183183
assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME > 'fred'")).collect().size === 2)
184184
assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME != 'fred'")).collect().size === 2)
185185
assert(stripSparkFilter(sql("SELECT * FROM nulltypes WHERE A IS NULL")).collect().size === 1)
186+
assert(stripSparkFilter(
187+
sql("SELECT * FROM nulltypes WHERE A IS NOT NULL")).collect().size === 0)
186188
}
187189

188190
test("SELECT * WHERE (quoted strings)") {

0 commit comments

Comments
 (0)