Source marketo: retry job creation instead of skipping

[davydov-d]

What

Marketo connector creates async jobs to read data within some of the streams. Sometimes it happens the job could not be created - the connector skips it, this means we can lose part of the data

How

Override should_retry method in a substream responsible for creating those jobs

[davydov-d]

/test connector=connectors/source-marketo

🕑 connectors/source-marketo https://github.com/airbytehq/airbyte/actions/runs/2865908426
❌ connectors/source-marketo https://github.com/airbytehq/airbyte/actions/runs/2865908426
🐛 https://gradle.com/s/k2jk4mtz5jrue

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_incremental.py::TestIncremental::test_read_sequential_slices[inputs0]
================== 1 failed, 28 passed in 7907.68s (2:11:47) ===================

[davydov-d]

/test connector=connectors/source-marketo

🕑 connectors/source-marketo https://github.com/airbytehq/airbyte/actions/runs/2866934256
❌ connectors/source-marketo https://github.com/airbytehq/airbyte/actions/runs/2866934256
🐛 https://gradle.com/s/4fdfv25gouwe6

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - AssertionError: All ...
================== 1 failed, 28 passed in 3287.21s (0:54:47) ===================

[davydov-d]

/test connector=connectors/source-marketo

🕑 connectors/source-marketo https://github.com/airbytehq/airbyte/actions/runs/2867715633
❌ connectors/source-marketo https://github.com/airbytehq/airbyte/actions/runs/2867715633
🐛 https://gradle.com/s/fqzubwvgcyq5m

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - AssertionError: All ...
================== 1 failed, 28 passed in 3226.95s (0:53:46) ===================

[davydov-d]

/test connector=connectors/source-marketo

🕑 connectors/source-marketo https://github.com/airbytehq/airbyte/actions/runs/2868097417
❌ connectors/source-marketo https://github.com/airbytehq/airbyte/actions/runs/2868097417
🐛 https://gradle.com/s/5ifzs3jjgy57a

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - AssertionError: All ...
================== 1 failed, 28 passed in 3107.10s (0:51:47) ===================

[davydov-d]

/test connector=connectors/source-marketo

🕑 connectors/source-marketo https://github.com/airbytehq/airbyte/actions/runs/2868706641
✅ connectors/source-marketo https://github.com/airbytehq/airbyte/actions/runs/2868706641
Python tests coverage:

Name                         Stmts   Miss  Cover
------------------------------------------------
source_marketo/__init__.py       2      0   100%
source_marketo/utils.py         33      7    79%
source_marketo/source.py       259    111    57%
------------------------------------------------
TOTAL                          294    118    60%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       121     25    79%   21-23, 29-31, 36-43, 48-61, 208-216
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       110     48    56%   23-26, 32, 36, 39-64, 67-69, 72-74, 77-79, 82-84, 87-89, 92-110, 144-146
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1321    463    65%

Build Passed

Test summary info:

All Passed

[davydov-d]

/publish connector=connectors/source-marketo

🕑 Publishing the following connectors:
connectors/source-marketo
https://github.com/airbytehq/airbyte/actions/runs/2874553347

Connector	Did it publish?	Were definitions generated?
connectors/source-marketo	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️