You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This patch adds support for reading unloaded decimal columns. Previously, we would fail to parse the string into a decimal, leading to ClassCastExceptions.
Note that there are still problems when loading decimals into Redshift; they appear to be truncated to longs. This will be fixed separately (see #61), but may be blocked by adding proper decimal support to spark-avro.
Note that there are two more things that I need to do before this is merge-worthy:
Fix / extend support for decimals in the schema-parsing code for the redshiftFile API.
Add tests for reading the largest / smallest possible decimal values and tests for varying precision and scale. I may be able to adapt some test cases from Spark's own decimal tests.
Oh no! It looks like stopping and starting new HiveContexts is somehow leaking PermGen space or something. I'm going to try enabling forking in tests and will bump up the permgen limit to see if that quickly fixes the problem.
I think that we're going to deprecate redshiftFile in 0.5.0 (and remove it in the next release if nobody complains; see #65), so I'm going to skip fixing the decimal support there. Given this, I'm going to merge this now in order to minimize merge conflicts with a bunch of other in-flight patches. I might loop back later to add additional tests and solicit post-hoc review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This patch adds support for reading unloaded decimal columns. Previously, we would fail to parse the string into a decimal, leading to ClassCastExceptions.