Commit bdfa6ef
[SPARK-50870][SQL] Add the timezone when casting to timestamp in V2ScanRelationPushDown
### What changes were proposed in this pull request?
Add the timezone information to a cast expression when the destination type requires it.
### Why are the changes needed?
When current_timestamp() is materialized as a string, the timezone information is gone (e.g., 2024-12-27 10:26:27.684158) which prohibits further optimization rules from being applied to the affected data source.
For example,
```
Project [1735900357973433#10 AS current_timestamp()#6]
+- 'Project [cast(2025-01-03 10:32:37.973433#11 as timestamp) AS 1735900357973433#10]
+- RelationV2[2025-01-03 10:32:37.973433#11] xxx
```
-> This query fails to execute because the injected cast expression lacks the timezone information.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing tests.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #49549 from changgyoopark-db/SPARK-50870.
Authored-by: changgyoopark-db <changgyoo.park@databricks.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
(cherry picked from commit 24abb0f)
Signed-off-by: Wenchen Fan <wenchen@databricks.com>1 parent 5de80b7 commit bdfa6ef
File tree
1 file changed
+6
-1
lines changed- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2
1 file changed
+6
-1
lines changedsql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala
Lines changed: 6 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
322 | 322 | | |
323 | 323 | | |
324 | 324 | | |
325 | | - | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
326 | 331 | | |
327 | 332 | | |
328 | 333 | | |
| |||
0 commit comments