Skip to content

[SPARK-13638][SQL] Add quoteAll option to CSV DataFrameWriter #13374

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
jurriaan wants to merge 1 commit into from
Closed

[SPARK-13638][SQL] Add quoteAll option to CSV DataFrameWriter #13374

jurriaan wants to merge 1 commit into from

Conversation

@jurriaan
Copy link
Contributor

@jurriaan jurriaan commented May 28, 2016

What changes were proposed in this pull request?

Adds an quoteAll option for writing CSV which will quote all fields.
See https://issues.apache.org/jira/browse/SPARK-13638

How was this patch tested?

Added a test to verify the output columns are quoted for all fields in the Dataframe

@jurriaan
Copy link
Contributor Author

jurriaan commented May 28, 2016

cc @rxin @HyukjinKwon

@SparkQA
Copy link

SparkQA commented May 28, 2016

Test build #3031 has finished for PR 13374 at commit 5a64a3a.

  • This patch fails RAT tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

hvanhovell
hvanhovell reviewed May 28, 2016
View reviewed changes
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVParser.scala Outdated
Copy link
Contributor

@hvanhovell hvanhovell May 28, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we remove this one?

Copy link
Contributor Author

@jurriaan jurriaan May 28, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed :)

@jurriaan
Copy link
Contributor Author

jurriaan commented Jun 28, 2016

Rebased, ping @rxin @HyukjinKwon

HyukjinKwon
HyukjinKwon reviewed Jul 1, 2016
View reviewed changes
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala Outdated
Copy link
Member

@HyukjinKwon HyukjinKwon Jul 1, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems escapeAll and quoteAll are mixed (across the updated files)..

Copy link
Contributor Author

@jurriaan jurriaan Jul 1, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed :)

@jurriaan jurriaan force-pushed the csv-quote-all branch 2 times, most recently from fd07fdf to 492391d Compare July 1, 2016 11:01
@jurriaan jurriaan changed the title [SPARK-13638][SQL] Add quoteAll option to CSV DataFrameWriter [SPARK-13638][SQL] Add escapeAll option to CSV DataFrameWriter Jul 1, 2016
@rxin
Copy link
Contributor

rxin commented Jul 8, 2016

@jurriaan should this be called quoteAll rather than escapeAll?

@jurriaan
Copy link
Contributor Author

jurriaan commented Jul 8, 2016

I thought it should be named in line with the escapeQuotes method, but what it's doing is more like quoting all values then escaping all. So i guess that name could make sense after all

@rxin
Copy link
Contributor

rxin commented Jul 8, 2016

Yup... would be great if you can update this. Otherwise LGTM.

@jurriaan
Copy link
Contributor Author

jurriaan commented Jul 8, 2016

@rxin Sorry for the confusion, fixed it :)

@jurriaan jurriaan changed the title [SPARK-13638][SQL] Add escapeAll option to CSV DataFrameWriter [SPARK-13638][SQL] Add quoteAll option to CSV DataFrameWriter Jul 8, 2016
@SparkQA
Copy link

SparkQA commented Jul 8, 2016

Test build #3171 has finished for PR 13374 at commit e42ab25.

  • This patch passes all tests.
  • This patch does not merge cleanly.
  • This patch adds no public classes.

@rxin
Copy link
Contributor

rxin commented Jul 8, 2016 •
edited
Loading

Merging in master/2.0. Thanks.

asfgit pushed a commit that referenced this pull request Jul 8, 2016
@jurriaan @rxin
[SPARK-13638][SQL] Add quoteAll option to CSV DataFrameWriter
8dee2ec 
## What changes were proposed in this pull request?

Adds an quoteAll option for writing CSV which will quote all fields.
See https://issues.apache.org/jira/browse/SPARK-13638

## How was this patch tested?

Added a test to verify the output columns are quoted for all fields in the Dataframe

Author: Jurriaan Pruis <email@jurriaanpruis.nl>

Closes #13374 from jurriaan/csv-quote-all.

(cherry picked from commit 38cf8f2)
Signed-off-by: Reynold Xin <rxin@databricks.com>
@asfgit asfgit closed this in 38cf8f2 Jul 8, 2016
@SparkQA
Copy link

SparkQA commented Jul 8, 2016

Test build #3172 has finished for PR 13374 at commit e42ab25.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@jurriaan @SparkQA @rxin @HyukjinKwon @hvanhovell