Skip to content

Conversation

@HyukjinKwon
Copy link
Member

@HyukjinKwon HyukjinKwon commented May 5, 2016

What changes were proposed in this pull request?

https://issues.apache.org/jira/browse/SPARK-15148

Mainly it improves the performance roughtly about 30%-40% according to the release note. For the details of the purpose is described in the JIRA.

This PR upgrades Univocity library from 2.0.2 to 2.1.0.

How was this patch tested?

Existing tests should cover this.

@HyukjinKwon
Copy link
Member Author

cc @rxin and @jbax (who is the author of Univocity library and suggested this change).

@rxin
Copy link
Contributor

rxin commented May 5, 2016

LGTM pending Jenkins.

@holdenk
Copy link
Contributor

holdenk commented May 5, 2016

So doing a quick grep shows that we are using the newly deprecated method setParseUnescapedQuotes and setParseUnescapedQuotesUntilDelimiter, we should update to setUnescapedQuoteHandling as per the release notes ( https://github.com/uniVocity/univocity-parsers/releases )

If its really urgent to get this upgrade in quickly, lets make a follow up JIRA to move away from these deprecated functions for the next release.

@HyukjinKwon
Copy link
Member Author

Thanks, @holdenk! It is not urgent. I can do this in this PR.

@holdenk
Copy link
Contributor

holdenk commented May 5, 2016

Great!

@SparkQA
Copy link

SparkQA commented May 5, 2016

Test build #2975 has finished for PR 12923 at commit e71c1ee.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented May 5, 2016

Test build #57853 has finished for PR 12923 at commit e71c1ee.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented May 5, 2016

Test build #57861 has finished for PR 12923 at commit e1e8063.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@srowen
Copy link
Member

srowen commented May 5, 2016

LGTM, except that it needs a rebase now

@SparkQA
Copy link

SparkQA commented May 5, 2016

Test build #57875 has finished for PR 12923 at commit bf7262f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@rxin
Copy link
Contributor

rxin commented May 5, 2016

Merging in master/branch-2.0.

@asfgit asfgit closed this in ac12b35 May 5, 2016
asfgit pushed a commit that referenced this pull request May 5, 2016
## What changes were proposed in this pull request?

https://issues.apache.org/jira/browse/SPARK-15148

Mainly it improves the performance roughtly about 30%-40% according to the [release note](https://github.com/uniVocity/univocity-parsers/releases/tag/v2.1.0). For the details of the purpose is described in the JIRA.

This PR upgrades Univocity library from 2.0.2 to 2.1.0.

## How was this patch tested?

Existing tests should cover this.

Author: hyukjinkwon <gurwls223@gmail.com>

Closes #12923 from HyukjinKwon/SPARK-15148.

(cherry picked from commit ac12b35)
Signed-off-by: Reynold Xin <rxin@databricks.com>
@HyukjinKwon HyukjinKwon deleted the SPARK-15148 branch January 2, 2018 03:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants