This repository was archived by the owner on Mar 24, 2025. It is now read-only.
Support for PERMISSIVE/DROPMALFORMED mode and corrupt record option. #107
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Current coverage is 90.53% (diff: 88.88%)@@ master #107 diff @@
==========================================
Files 14 15 +1
Lines 682 697 +15
Methods 621 638 +17
Messages 0 0
Branches 61 59 -2
==========================================
+ Hits 617 631 +14
- Misses 65 66 +1
Partials 0 0
|
HyukjinKwon
changed the title
Member
Author
|
@cloud-fan Do you mind if I ask a quick look for this please? I rememeber JSON parse modes were reviewed by you. |
|
Haven't we absorbed it into Spark already? |
Member
Author
|
@cloud-fan Ah, no CSV was ported into Spark but XML one has not been. I have been managing this and confused of who I should cc. |
|
cc @rxin |
Closed
Member
Author
|
Sorry for cc here and there @cloud-fan. Could you please give a quick look please? |
Member
Author
|
Maybe I will merge this after double-checking this by myself if you don't have time to review this PR (but will wait more just in case) - @cloud-fan |
Member
Author
|
I am going to merge this as soon as the tests pass. |
Closed
HyukjinKwon
added a commit
that referenced
this pull request
Sep 10, 2016
This PR prepares the release for 0.4.0. This will include the changes below: - Support for PERMISSIVE/DROPMALFORMED mode and corrupt record option. #107 - Change default values for valueTag and attributePrefix to avoid always require field escape for some apis #142 - deprecates saveAsXmlFile and promote the usage of write(). #150 - deprecates xmlFile and promote the usage of read(). #150 - drops 1.x compatibility from 0.4.0. #150 - makes not supporting UserDefinedType as it became private. #150 Author: hyukjinkwon <gurwls223@gmail.com> Closes #176 from HyukjinKwon/version-0.4.0.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
#105
Currently, this library does not support
PERMISSIVEparse mode. Similar with JSON data source, this also can be done in the same way with_corrupt_record.This PR adds the support for
PERMISSIVEmode and make this behaviour consistent with the other data sources supporting parse modes (JSON and CSV data sources.)Also, this PR adds the support for
_corrupt_record.This PR is similar with apache/spark#11756 and apache/spark#11881.