Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[file-based cdk] Remove CSV quoting_behavior config option #28330

Closed
3 tasks
brianjlai opened this issue Jul 14, 2023 · 0 comments · Fixed by #29388
Closed
3 tasks

[file-based cdk] Remove CSV quoting_behavior config option #28330

brianjlai opened this issue Jul 14, 2023 · 0 comments · Fixed by #29388
Assignees
@brianjlai
Labels
area/file-cdk team/extensibility

Comments

@brianjlai
Copy link
Contributor

What

We had initially added quoting_behavior to support the newlines_in_values field from the legacy S3 destination that uses pyarrow. After some testing of various CSV inputs with newlines, we should be able to support newlines as long as we pass csv.QUOTE_MINIMAL to the parser which is also the default.

Acceptance Criteria

  • The quoting_behavior config is removed from the CsvFormat
  • When creating the new dialect incsv_parser.py, the csv.QUOTE_MINIMAL is explicitly passed. We want to explicitly use this to make it obvious which behavior we use in case the default behavior changes
  • The resulting spec message does not have quoting_behavior and test configs have it removed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@brianjlai brianjlai

Labels
area/file-cdk team/extensibility
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[file-based cdk] Remove CSV quoting_behavior config option airbytehq/airbyte
2 participants
@clnoll @brianjlai