-
Notifications
You must be signed in to change notification settings - Fork 6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SDENT 180] support for fail fast type casting #201
base: snappy/branch-2.1
Are you sure you want to change the base?
Conversation
conversions based on configuration property `snappydata.failFastTypeCasting`.
- string to date - string to decimal - NaN, Infinite value of fractional numbers to timestamp - string to boolean - decimal precision overflow - date to decimal Enhancing string to numeric type casting failure to include the data value for which casting failed. Also unifying the exception of string to numeric conversion failure to match the exception of other casting failures.
instead of using generic RuntimeException - Adding tests for more scenarios
code flow is not reaching there due to analysis check performed by `checkInputDataTypes` method.
This set of changes is large and will be difficult to port to later 2.x versions. Moreover, this will be handled cleanly in Spark 3.0 (https://issues.apache.org/jira/browse/SPARK-28989) so I do not think we should make this change as of now. If really required for 2.x series, then
|
To clarify, the CAST operator's child will also need to be changed to a new expression that simply |
Another advantage of the mentioned approach is that it can be done entirely in SD layer without any Spark layer changes so will work for smart connector too. The only disadvantage will be that the exception message will have to be a generic one like "CAST operation from '...' to type ... failed" rather than specific ones but that is a reasonable compromise. |
4ffe404
to
79ddd88
Compare
52a9a47
to
763e1bb
Compare
1456d1f
to
3d73189
Compare
d35dbba
to
e76c62d
Compare
1feaa36
to
cdd6d29
Compare
cba053b
to
3527357
Compare
0169c3d
to
1fb3673
Compare
4a3161a
to
efa4458
Compare
Changes proposed in this pull request
Apache Spark handles type casting failures in a fail safe manner. Instead of
failing the entire query, Spark populates
null
values whenever type castingfailure occurs. Snappydata inherits the same behaviour from Spark.
With this changes we are exposing a session level configuration to enable fail
fast behaviour for casting errors. i.e. instead of returning
null
values forcasting failures, the query will be failed altogether whenever first type cast
error occurs.
The configuration property is
snappydata.failOnCastError
which defaultsto
false
. Setting this property true will turn on fail fast casting behaviour.Patch testing
Other PRs
TIBCOSoftware/snappydata#1542