🐛 Source S3: work-around for format.delimiter change '\\t' -> '\t' #9163

grubberr · 2021-12-28T18:26:20Z

Signed-off-by: Sergey Chvalyuk grubberr@gmail.com

What

It's not easy task to pass [tab] symbol from UI to config.json

If we enter [tab] symbol in HTML form field it just lost and converted to empty string "" in javascript level
If we enter [backslash]-[t] '\t' in HTML form field it's passed as 2 symbols respectively => [backslash]-[t]

Both variants is not what we need if we want "tab" delimiter

How

This work-around tranformat '\\t' to '\t' for field format.delimiter

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

antixar

could you add tests for similar cases?

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

This reverts commit 8177d41.

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2022-01-06T11:06:43Z

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1662715943
❌ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1662715943
🐛 https://gradle.com/s/aya7pzgoj5xoa
Python short test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestSpec::test_match_expected[inputs0] - AssertionError:...
=================== 1 failed, 31 passed in 112.12s (0:01:52) ===================

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2022-01-06T11:52:13Z

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1662863917
✅ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1662863917
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      6    92%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              242     96    60%
	 source_acceptance_test/tests/test_full_refresh.py       38      0   100%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     17    69%
	 source_acceptance_test/utils/compare.py                 62     23    63%
	 source_acceptance_test/utils/connector_runner.py       110     48    56%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  979    404    59%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 source_s3/__init__.py                                                 2      0   100%
	 source_s3/s3_utils.py                                                20      3    85%
	 source_s3/s3file.py                                                  49      3    94%
	 source_s3/source.py                                                  28      4    86%
	 source_s3/source_files_abstract/__init__.py                           0      0   100%
	 source_s3/source_files_abstract/formats/abstract_file_parser.py      34      2    94%
	 source_s3/source_files_abstract/formats/csv_parser.py                68     18    74%
	 source_s3/source_files_abstract/formats/csv_spec.py                  16      0   100%
	 source_s3/source_files_abstract/formats/parquet_parser.py            61     44    28%
	 source_s3/source_files_abstract/formats/parquet_spec.py               9      0   100%
	 source_s3/source_files_abstract/source.py                            40     18    55%
	 source_s3/source_files_abstract/spec.py                              42     22    48%
	 source_s3/source_files_abstract/storagefile.py                       16      0   100%
	 source_s3/source_files_abstract/stream.py                           193     10    95%
	 source_s3/stream.py                                                  42      3    93%
	 source_s3/utils.py                                                   27     10    63%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                               647    137    79%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 source_s3/__init__.py                                                 2      0   100%
	 source_s3/s3_utils.py                                                20     13    35%
	 source_s3/s3file.py                                                  49     26    47%
	 source_s3/source.py                                                  28      0   100%
	 source_s3/source_files_abstract/__init__.py                           0      0   100%
	 source_s3/source_files_abstract/formats/abstract_file_parser.py      34      0   100%
	 source_s3/source_files_abstract/formats/csv_parser.py                68     11    84%
	 source_s3/source_files_abstract/formats/csv_spec.py                  16      0   100%
	 source_s3/source_files_abstract/formats/parquet_parser.py            61      3    95%
	 source_s3/source_files_abstract/formats/parquet_spec.py               9      0   100%
	 source_s3/source_files_abstract/source.py                            40     18    55%
	 source_s3/source_files_abstract/spec.py                              42     22    48%
	 source_s3/source_files_abstract/storagefile.py                       16      3    81%
	 source_s3/source_files_abstract/stream.py                           193    101    48%
	 source_s3/stream.py                                                  42     30    29%
	 source_s3/utils.py                                                   27      8    70%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                               647    235    64%

antixar

LGTM
Please publish your changes

grubberr · 2022-01-06T17:32:53Z

/publish connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1664122059
✅ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1664122059

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

overrite config if sequence of 2 characters '\' an 't' is found in the delimeter config, then convert it into a \t [TAB] character. Then, proceed with standard validation

work-around for format.delimiter '\\t' -> '\t'

ca2755e

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr changed the title ~~🐛 Source S3: work-around for format.delimiter '\\t' -> '\t'~~ 🐛 Source S3: work-around for format.delimiter change '\\t' -> '\t' Dec 28, 2021

github-actions bot added the area/connectors Connector related issues label Dec 28, 2021

grubberr temporarily deployed to more-secrets December 28, 2021 18:27 Inactive

grubberr linked an issue Dec 28, 2021 that may be closed by this pull request

🐛 Source S3: tab delimiter is not parsed properly #8789

Closed

grubberr self-assigned this Dec 28, 2021

grubberr requested a review from antixar December 28, 2021 18:39

antixar reviewed Jan 4, 2022

View reviewed changes

Merge branch 'master' into grubberr/8789-source-s3

96283fd

grubberr temporarily deployed to more-secrets January 4, 2022 15:57 Inactive

check disk space

8177d41

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr temporarily deployed to more-secrets January 4, 2022 17:08 Inactive

Revert "check disk space"

6769362

This reverts commit 8177d41.

grubberr temporarily deployed to more-secrets January 4, 2022 17:18 Inactive

add some debug info

c49a5e2

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr temporarily deployed to more-secrets January 4, 2022 19:30 Inactive

grubberr temporarily deployed to more-secrets January 4, 2022 20:46 Inactive

build airbyteDocker

8de2b19

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr temporarily deployed to more-secrets January 4, 2022 21:28 Inactive

grubberr added 4 commits January 5, 2022 13:20

Merge branch 'master' into grubberr/8789-source-s3

67cb8c5

remove debug

03d0b82

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

Merge branch 'master' into grubberr/8789-source-s3

9b84368

test_transform_backslash_t_to_tab added

b53f7a5

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr temporarily deployed to more-secrets January 6, 2022 10:48 Inactive

bump version 0.1.8 -> 0.1.9

af80ed6

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr temporarily deployed to more-secrets January 6, 2022 11:02 Inactive

s3.md updated

30a0924

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

github-actions bot added the area/documentation Improvements or additions to documentation label Jan 6, 2022

grubberr temporarily deployed to more-secrets January 6, 2022 11:07 Inactive

fix 2021 -> 2022

145ccf3

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr temporarily deployed to more-secrets January 6, 2022 11:09 Inactive

grubberr added 2 commits January 6, 2022 13:10

s3.md fixed

3fe044e

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

s3.md updated 2-nd time

d76302d

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr temporarily deployed to more-secrets January 6, 2022 11:15 Inactive

fix SAT tests

fa3bfd6

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr temporarily deployed to more-secrets January 6, 2022 11:53 Inactive

jrhizor temporarily deployed to more-secrets January 6, 2022 11:54 Inactive

grubberr requested a review from antixar January 6, 2022 12:13

antixar approved these changes Jan 6, 2022

View reviewed changes

Merge branch 'master' into grubberr/8789-source-s3

48397eb

grubberr temporarily deployed to more-secrets January 6, 2022 17:34 Inactive

jrhizor temporarily deployed to more-secrets January 6, 2022 17:35 Inactive

grubberr added 2 commits January 6, 2022 20:47

Merge branch 'master' into grubberr/8789-source-s3

897a0b1

source_specs.yaml updated

a1b546b

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr merged commit c7021e6 into master Jan 6, 2022

grubberr deleted the grubberr/8789-source-s3 branch January 6, 2022 18:49

grubberr temporarily deployed to more-secrets January 6, 2022 18:51 Inactive

jrhizor mentioned this pull request Jan 7, 2022

Bump Airbyte version from 0.35.3-alpha to 0.35.4-alpha #9353

Merged

dashirov mentioned this pull request Nov 10, 2023

Source S3: no longer supports tab-delimited (TSV) configurations #32447

Closed

1 task

brianjlai mentioned this pull request Mar 12, 2024

[airbyte-cdk] Fix tab delimiter configuration in CSV file type #35901

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Source S3: work-around for format.delimiter change '\\t' -> '\t' #9163

🐛 Source S3: work-around for format.delimiter change '\\t' -> '\t' #9163

grubberr commented Dec 28, 2021 •

edited

Loading

antixar left a comment

grubberr commented Jan 6, 2022 •

edited by github-actions bot

Loading

grubberr commented Jan 6, 2022 •

edited by github-actions bot

Loading

antixar left a comment

grubberr commented Jan 6, 2022 •

edited by github-actions bot

Loading

🐛 Source S3: work-around for format.delimiter change '\\t' -> '\t' #9163

🐛 Source S3: work-around for format.delimiter change '\\t' -> '\t' #9163

Conversation

grubberr commented Dec 28, 2021 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

antixar left a comment

Choose a reason for hiding this comment

grubberr commented Jan 6, 2022 • edited by github-actions bot Loading

grubberr commented Jan 6, 2022 • edited by github-actions bot Loading

antixar left a comment

Choose a reason for hiding this comment

grubberr commented Jan 6, 2022 • edited by github-actions bot Loading

grubberr commented Dec 28, 2021 •

edited

Loading

grubberr commented Jan 6, 2022 •

edited by github-actions bot

Loading

grubberr commented Jan 6, 2022 •

edited by github-actions bot

Loading

grubberr commented Jan 6, 2022 •

edited by github-actions bot

Loading