Commit 908c472
committed
[SPARK-46396][SQL] Timestamp inference should not throw exception
### What changes were proposed in this pull request?
When setting `spark.sql.legacy.timeParserPolicy=LEGACY`, Spark will use the LegacyFastTimestampFormatter to infer potential timestamp columns. The inference shouldn't throw exception.
However, when the input is 23012150952, there is exception:
```
For input string: "23012150952"
java.lang.NumberFormatException: For input string: "23012150952"
at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67)
at java.base/java.lang.Integer.parseInt(Integer.java:668)
at java.base/java.lang.Integer.parseInt(Integer.java:786)
at org.apache.commons.lang3.time.FastDateParser$NumberStrategy.parse(FastDateParser.java:304)
at org.apache.commons.lang3.time.FastDateParser.parse(FastDateParser.java:1045)
at org.apache.commons.lang3.time.FastDateFormat.parse(FastDateFormat.java:651)
at org.apache.spark.sql.catalyst.util.LegacyFastTimestampFormatter.parseOptional(TimestampFormatter.scala:418)
```
This PR is to fix the issue.
### Why are the changes needed?
Bug fix, Timestamp inference should not throw exception
### Does this PR introduce _any_ user-facing change?
NO
### How was this patch tested?
New test case + existing tests
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #44338 from gengliangwang/fixParseOptional.
Authored-by: Gengliang Wang <gengliang@apache.org>
Signed-off-by: Gengliang Wang <gengliang@apache.org>
(cherry picked from commit 4a79ae9)
Signed-off-by: Gengliang Wang <gengliang@apache.org>1 parent eb1e6ad commit 908c472
File tree
2 files changed
+10
-5
lines changed- sql
- api/src/main/scala/org/apache/spark/sql/catalyst/util
- catalyst/src/test/scala/org/apache/spark/sql/catalyst/util
2 files changed
+10
-5
lines changedLines changed: 8 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
414 | 414 | | |
415 | 415 | | |
416 | 416 | | |
417 | | - | |
418 | | - | |
419 | | - | |
420 | | - | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
421 | 425 | | |
422 | 426 | | |
423 | 427 | | |
| |||
Lines changed: 2 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
502 | 502 | | |
503 | 503 | | |
504 | 504 | | |
| 505 | + | |
505 | 506 | | |
506 | 507 | | |
507 | 508 | | |
508 | | - | |
| 509 | + | |
509 | 510 | | |
510 | 511 | | |
511 | 512 | | |
| |||
0 commit comments