Add connector acceptance test for mysql #22317

flash1293 · 2023-02-02T15:48:01Z

This PR adds a connector acceptance test for the spec to the mysql destination. This ensures the spec is respecting all limitations which is important for providing a functional configuration form to the user in the UI.

Also had to do a minor adjustment to the gradle plugin to make it work (it probably works for other cases related to some implicit dependies between tasks that don't work for the tests of this connector): https://github.com/airbytehq/airbyte/pull/22317/files#diff-6a042a48a0e852de076678fb1b2bd5ee79f3efad8f7e35c8267476c8185e88b4L22

flash1293 · 2023-02-02T15:48:15Z

/test connector=connectors/destination-mysql

github-actions · 2023-02-02T15:50:13Z

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

Run integration tests
Bump connector or module version
Add changelog
Publish the new version

✅ Sources (0)

Connector	Version	Changelog	Publish

See "Actionable Items" below for how to resolve warnings and errors.

❌ Destinations (2)

Connector	Version	Changelog	Publish
`destination-mysql`	`0.1.20`	✅	✅
`destination-mysql-strict-encrypt`	❌ `0.1.21` (mismatch: `0.1.20`)	🔵 (ignored)	🔵 (ignored)

See "Actionable Items" below for how to resolve warnings and errors.

👀 Other Modules (1)

base-normalization

Actionable Items

(click to expand)

Category	Status	Actionable Item
Version	❌ mismatch	The version of the connector is different from its normal variant. Please bump the version of the connector.
	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
Changelog	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
	❌ changelog missing	There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish	⚠ not in seed	The connector is not in the seed file (e.g. `source_definitions.yaml`), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug.
	❌ diff seed version	The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the `/publish` command to publish the latest version.

flash1293 · 2023-02-02T16:32:27Z

/test connector=connectors/destination-mysql

🕑 connectors/destination-mysql https://github.com/airbytehq/airbyte/actions/runs/4076569527
❌ connectors/destination-mysql https://github.com/airbytehq/airbyte/actions/runs/4076569527
🐛 https://gradle.com/s/epxgpu4jbjxn2

Build Failed

Test summary info:

Could not find result summary

flash1293 · 2023-02-02T18:45:19Z

/test connector=connectors/destination-mysql

🕑 connectors/destination-mysql https://github.com/airbytehq/airbyte/actions/runs/4077637030
❌ connectors/destination-mysql https://github.com/airbytehq/airbyte/actions/runs/4077637030
🐛 https://gradle.com/s/fcjpod5uxzeec

Build Failed

Test summary info:

Could not find result summary

flash1293 · 2023-02-02T18:45:25Z

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/4077637624
✅ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/4077637624
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
source_s3/source_files_abstract/storagefile.py                       23      0   100%
source_s3/source_files_abstract/spec.py                              55      0   100%
source_s3/source_files_abstract/formats/parquet_spec.py               9      0   100%
source_s3/source_files_abstract/formats/jsonl_spec.py                13      0   100%
source_s3/source_files_abstract/formats/csv_spec.py                  16      0   100%
source_s3/source_files_abstract/formats/avro_spec.py                  5      0   100%
source_s3/source.py                                                  27      0   100%
source_s3/exceptions.py                                              10      0   100%
source_s3/__init__.py                                                 2      0   100%
source_s3/source_files_abstract/formats/parquet_parser.py            64      2    97%
source_s3/source_files_abstract/formats/abstract_file_parser.py      41      2    95%
source_s3/stream.py                                                  43      3    93%
source_s3/s3file.py                                                  41      3    93%
source_s3/source_files_abstract/formats/avro_parser.py               39      3    92%
source_s3/source_files_abstract/formats/jsonl_parser.py              53      5    91%
source_s3/source_files_abstract/file_info.py                         26      3    88%
source_s3/source_files_abstract/source.py                            39      5    87%
source_s3/source_files_abstract/stream.py                           248     45    82%
source_s3/source_files_abstract/formats/csv_parser.py               102     20    80%
source_s3/s3_utils.py                                                20      4    80%
source_s3/utils.py                                                   31     10    68%
-------------------------------------------------------------------------------------
TOTAL                                                               907    105    88%
	 Name                                                    Stmts   Miss  Cover   Missing
	 -------------------------------------------------------------------------------------
	 connector_acceptance_test/base.py                          12      4    67%   16-19
	 connector_acceptance_test/config.py                       141      5    96%   87, 93, 239, 243-244
	 connector_acceptance_test/conftest.py                     211     95    55%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
	 connector_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 connector_acceptance_test/tests/test_core.py              476    117    75%   53, 58, 97-108, 113-120, 124-125, 129-130, 380, 400, 438, 476-493, 506-517, 521-526, 532, 565-570, 608-615, 658-660, 663, 728-736, 748-751, 756, 812-813, 819, 822, 858-868, 881-906
	 connector_acceptance_test/tests/test_incremental.py       160     14    91%   58-65, 70-83, 246
	 connector_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 connector_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 connector_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 connector_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 connector_acceptance_test/utils/json_schema_helper.py     114     13    89%   31-32, 39, 42, 66-69, 97, 121, 203-205
	 -------------------------------------------------------------------------------------
	 TOTAL                                                    1690    341    80%
Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
source_s3/source_files_abstract/formats/parquet_spec.py               9      0   100%
source_s3/source_files_abstract/formats/jsonl_spec.py                13      0   100%
source_s3/source_files_abstract/formats/csv_spec.py                  16      0   100%
source_s3/source_files_abstract/formats/avro_spec.py                  5      0   100%
source_s3/s3_utils.py                                                20      0   100%
source_s3/__init__.py                                                 2      0   100%
source_s3/source_files_abstract/storagefile.py                       23      1    96%
source_s3/stream.py                                                  43      3    93%
source_s3/s3file.py                                                  41      3    93%
source_s3/source_files_abstract/stream.py                           248     20    92%
source_s3/source_files_abstract/formats/abstract_file_parser.py      41      4    90%
source_s3/source.py                                                  27      4    85%
source_s3/source_files_abstract/formats/csv_parser.py               102     30    71%
source_s3/source_files_abstract/file_info.py                         26      8    69%
source_s3/utils.py                                                   31     10    68%
source_s3/exceptions.py                                              10      4    60%
source_s3/source_files_abstract/source.py                            39     16    59%
source_s3/source_files_abstract/spec.py                              55     31    44%
source_s3/source_files_abstract/formats/jsonl_parser.py              53     32    40%
source_s3/source_files_abstract/formats/avro_parser.py               39     25    36%
source_s3/source_files_abstract/formats/parquet_parser.py            64     44    31%
-------------------------------------------------------------------------------------
TOTAL                                                               907    235    74%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:98: The previous and actual specifications are identical.
SKIPPED [6] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:507: The previous and actual discovered catalogs are identical.
================== 108 passed, 7 skipped in 242.34s (0:04:02) ==================

evantahler · 2023-02-02T18:57:00Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/4077718852
❌ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/4077718852
🐛 https://gradle.com/s/q36gwjs4hadug

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestConnection::test_check[inputs0] - Failed: Timeout >3...
FAILED test_core.py::TestConnection::test_check[inputs1] - Failed: Timeout >3...
FAILED test_core.py::TestDiscovery::test_discover[inputs0] - Failed: Timeout ...
FAILED test_core.py::TestDiscovery::test_discover[inputs1] - Failed: Timeout ...
ERROR test_core.py::TestDiscovery::test_defined_cursors_exist_in_schema[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:98: The previous and actual specifications are identical.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:507: The previous and actual discovered catalogs are identical.
========= 4 failed, 39 passed, 3 skipped, 1 error in 470.67s (0:07:50) =========

evantahler · 2023-02-02T18:57:06Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/4077719495
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/4077719495
Python tests coverage:

	 Name                                                    Stmts   Miss  Cover   Missing
	 -------------------------------------------------------------------------------------
	 connector_acceptance_test/base.py                          12      4    67%   16-19
	 connector_acceptance_test/config.py                       141      5    96%   87, 93, 239, 243-244
	 connector_acceptance_test/conftest.py                     211     95    55%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
	 connector_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 connector_acceptance_test/tests/test_core.py              476    117    75%   53, 58, 97-108, 113-120, 124-125, 129-130, 380, 400, 438, 476-493, 506-517, 521-526, 532, 565-570, 608-615, 658-660, 663, 728-736, 748-751, 756, 812-813, 819, 822, 858-868, 881-906
	 connector_acceptance_test/tests/test_incremental.py       160     14    91%   58-65, 70-83, 246
	 connector_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 connector_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 connector_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 connector_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 connector_acceptance_test/utils/json_schema_helper.py     114     13    89%   31-32, 39, 42, 66-69, 97, 121, 203-205
	 -------------------------------------------------------------------------------------
	 TOTAL                                                    1690    341    80%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:98: The previous and actual specifications are identical.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:507: The previous and actual discovered catalogs are identical.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_incremental.py:22: `future_state` has a bypass reason, skipping.
================== 66 passed, 6 skipped in 757.22s (0:12:37) ===================

add connector acceptance test for mysql

e4a1b12

octavia-squidington-iii added the area/connectors Connector related issues label Feb 2, 2023

octavia-squidington-iii added the connectors/destination/mysql label Feb 2, 2023

fix build

872bb40

flash1293 temporarily deployed to more-secrets February 2, 2023 18:47 — with GitHub Actions Inactive

flash1293 mentioned this pull request Feb 8, 2023

Make connector acceptance test gradle plugin work for destinations #22609

Merged

flash1293 closed this Mar 14, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add connector acceptance test for mysql #22317

Add connector acceptance test for mysql #22317

flash1293 commented Feb 2, 2023 •

edited

Loading

flash1293 commented Feb 2, 2023

github-actions bot commented Feb 2, 2023 •

edited

Loading

✅ Sources (0)

❌ Destinations (2)

👀 Other Modules (1)

Actionable Items

flash1293 commented Feb 2, 2023 •

edited by github-actions bot

Loading

flash1293 commented Feb 2, 2023 •

edited by github-actions bot

Loading

flash1293 commented Feb 2, 2023 •

edited by github-actions bot

Loading

evantahler commented Feb 2, 2023 •

edited by github-actions bot

Loading

evantahler commented Feb 2, 2023 •

edited by github-actions bot

Loading

Add connector acceptance test for mysql #22317

Add connector acceptance test for mysql #22317

Conversation

flash1293 commented Feb 2, 2023 • edited Loading

flash1293 commented Feb 2, 2023

github-actions bot commented Feb 2, 2023 • edited Loading

Affected Connector Report

✅ Sources (0)

❌ Destinations (2)

👀 Other Modules (1)

Actionable Items

flash1293 commented Feb 2, 2023 • edited by github-actions bot Loading

Build Failed

flash1293 commented Feb 2, 2023 • edited by github-actions bot Loading

Build Failed

flash1293 commented Feb 2, 2023 • edited by github-actions bot Loading

Build Passed

evantahler commented Feb 2, 2023 • edited by github-actions bot Loading

Build Failed

evantahler commented Feb 2, 2023 • edited by github-actions bot Loading

Build Passed

flash1293 commented Feb 2, 2023 •

edited

Loading

github-actions bot commented Feb 2, 2023 •

edited

Loading

flash1293 commented Feb 2, 2023 •

edited by github-actions bot

Loading

flash1293 commented Feb 2, 2023 •

edited by github-actions bot

Loading

flash1293 commented Feb 2, 2023 •

edited by github-actions bot

Loading

evantahler commented Feb 2, 2023 •

edited by github-actions bot

Loading

evantahler commented Feb 2, 2023 •

edited by github-actions bot

Loading