Source S3: read a single record on check

[davydov-d]

What

https://github.com/airbytehq/oncall/issues/1697

How

When calling check, we make a test read. To save time and memory, we need to read a single record instead of a whole slice. Since the block size in a config may be huge, we lower its value for the check command.

[davydov-d]

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/4505469506
❌ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/4505469506
🐛 https://gradle.com/s/cgxk3bvjp4exs

Build Failed

Test summary info:

	 =========================== short test summary info ============================
	 FAILED unit_tests/test_stream.py::TestIncrementalFileStream::test_fileformatparser_map
	 �[31m============ �[31m�[1m1 failed�[0m, �[32m165 passed�[0m, �[33m10 warnings�[0m�[31m in 66.33s (0:01:06)�[0m�[31m =============�[0m

[davydov-d]

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/4509189579
✅ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/4509189579
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
source_s3/source_files_abstract/storagefile.py                       23      0   100%
source_s3/source_files_abstract/spec.py                              55      0   100%
source_s3/source_files_abstract/formats/parquet_spec.py               9      0   100%
source_s3/source_files_abstract/formats/jsonl_spec.py                13      0   100%
source_s3/source_files_abstract/formats/csv_spec.py                  16      0   100%
source_s3/source_files_abstract/formats/avro_spec.py                  5      0   100%
source_s3/source.py                                                  27      0   100%
source_s3/exceptions.py                                              10      0   100%
source_s3/__init__.py                                                 2      0   100%
source_s3/source_files_abstract/formats/parquet_parser.py            67      3    96%
source_s3/source_files_abstract/formats/abstract_file_parser.py      44      3    93%
source_s3/stream.py                                                  43      3    93%
source_s3/s3file.py                                                  41      3    93%
source_s3/source_files_abstract/formats/avro_parser.py               39      3    92%
source_s3/source_files_abstract/formats/jsonl_parser.py              56      6    89%
source_s3/source_files_abstract/file_info.py                         26      3    88%
source_s3/source_files_abstract/formats/csv_parser.py               133     16    88%
source_s3/source_files_abstract/stream.py                           209     35    83%
source_s3/source_files_abstract/source.py                            41      7    83%
source_s3/s3_utils.py                                                20      4    80%
source_s3/utils.py                                                   31     10    68%
-------------------------------------------------------------------------------------
TOTAL                                                               910     96    89%
Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
source_s3/source_files_abstract/formats/parquet_spec.py               9      0   100%
source_s3/source_files_abstract/formats/jsonl_spec.py                13      0   100%
source_s3/source_files_abstract/formats/csv_spec.py                  16      0   100%
source_s3/source_files_abstract/formats/avro_spec.py                  5      0   100%
source_s3/s3_utils.py                                                20      0   100%
source_s3/__init__.py                                                 2      0   100%
source_s3/source_files_abstract/storagefile.py                       23      1    96%
source_s3/stream.py                                                  43      3    93%
source_s3/s3file.py                                                  41      3    93%
source_s3/source_files_abstract/stream.py                           209     18    91%
source_s3/source_files_abstract/formats/abstract_file_parser.py      44      5    89%
source_s3/source.py                                                  27      4    85%
source_s3/source_files_abstract/file_info.py                         26      8    69%
source_s3/source_files_abstract/formats/csv_parser.py               133     41    69%
source_s3/utils.py                                                   31     10    68%
source_s3/exceptions.py                                              10      4    60%
source_s3/source_files_abstract/source.py                            41     18    56%
source_s3/source_files_abstract/spec.py                              55     31    44%
source_s3/source_files_abstract/formats/jsonl_parser.py              56     33    41%
source_s3/source_files_abstract/formats/avro_parser.py               39     25    36%
source_s3/source_files_abstract/formats/parquet_parser.py            67     45    33%
-------------------------------------------------------------------------------------
TOTAL                                                               910    249    73%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [5] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:578: The previous and actual discovered catalogs are identical.
================== 98 passed, 6 skipped in 271.22s (0:04:31) ===================

[wmikemoon]

@davydov-d any update here?

[davydov-d]

@wmikemoon still waiting for the review

[lazebnyi]

Nice!

[davydov-d]

/publish connector=connectors/source-s3

🕑 Publishing the following connectors:
connectors/source-s3
https://github.com/airbytehq/airbyte/actions/runs/4529833611

Connector	Did it publish?	Were definitions generated?
connectors/source-s3	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️