Fix: array contains null handling#3372
Conversation
|
Thanks @Shekharrajak! Can we add/migrate the tests to the new framework? https://datafusion.apache.org/comet/contributor-guide/sql-file-tests.html |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3372 +/- ##
============================================
+ Coverage 56.12% 60.01% +3.89%
- Complexity 976 1475 +499
============================================
Files 119 175 +56
Lines 11743 16185 +4442
Branches 2251 2684 +433
============================================
+ Hits 6591 9714 +3123
- Misses 4012 5116 +1104
- Partials 1140 1355 +215 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Updated b7929c0 Please trigger the checks . |
| } | ||
| } | ||
|
|
||
| test("array_contains - NULL array returns NULL") { |
There was a problem hiding this comment.
Unless this test is testing functionality that isn't already tested in the SQL-based tests, I think we can just remove this.
|
It seems build is failing at create array https://github.com/apache/datafusion-comet/blob/main/spark/src/main/scala/org/apache/comet/serde/arrays.scala#L401 |
|
All checks looks fine now. |
Which issue does this PR close?
Closes #3345
Rationale for this change
When array_contains() is called with a literal cast(NULL as array) argument, Comet returns a different result than Spark. Spark follows SQL
three-valued logic where array_contains(NULL, value) should return NULL, but Comet was delegating directly to DataFusion's array_has without proper NULL
array handling.
Other array functions like CometArrayRemove and CometArrayAppend already handle NULL arrays correctly using a CASE WHEN wrapper, but CometArrayContains
was missing this handling.
What changes are included in this PR?
How are these changes tested?
Added new test "array_contains - NULL array returns NULL" in CometArrayExpressionSuite