migrate source GA connectors to per-stream states (2) #17410

alafanechere · 2022-09-29T18:18:17Z

What

The following GA connectors were not yet migrated to per-stream states:

source-twilio
~~source-snapchat-marketing~~
source-google-sheets
source-freshdesk

alafanechere · 2022-09-29T18:18:51Z

/test connector=connectors/source-snapchat-marketing

🕑 connectors/source-snapchat-marketing https://github.com/airbytehq/airbyte/actions/runs/3153628741
❌ connectors/source-snapchat-marketing https://github.com/airbytehq/airbyte/actions/runs/3153628741
🐛 https://gradle.com/s/54haqi4le7ckc

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_incremental.py::TestIncremental::test_read_sequential_slices[inputs0]
FAILED test_incremental.py::TestIncremental::test_read_sequential_slices[inputs1]
FAILED test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_incremental.py:23: `future_state_path` not specified, skipping
============= 3 failed, 34 passed, 1 skipped in 323.03s (0:05:23) ==============

alafanechere · 2022-09-29T18:21:31Z

/test connector=connectors/source-twilio

🕑 connectors/source-twilio https://github.com/airbytehq/airbyte/actions/runs/3153645095
❌ connectors/source-twilio https://github.com/airbytehq/airbyte/actions/runs/3153645095
🐛 https://gradle.com/s/scfvvw7gccbdy

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Stream avail...
=================== 1 failed, 31 passed in 278.79s (0:04:38) ===================

alafanechere · 2022-09-29T18:24:23Z

/test connector=connectors/source-freshdesk

🕑 connectors/source-freshdesk https://github.com/airbytehq/airbyte/actions/runs/3153661603
✅ connectors/source-freshdesk https://github.com/airbytehq/airbyte/actions/runs/3153661603
Python tests coverage:

Name                           Stmts   Miss  Cover
--------------------------------------------------
source_freshdesk/source.py        31      0   100%
source_freshdesk/__init__.py       2      0   100%
source_freshdesk/streams.py      222      6    97%
source_freshdesk/utils.py         20      1    95%
--------------------------------------------------
TOTAL                            275      7    97%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       152     26    83%   21-23, 29-31, 36-43, 48-61, 239, 250-258
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-67, 70-72, 75-77, 80-82, 85-87, 90-92, 95-113, 147-149
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1358    466    66%

Build Passed

Test summary info:

All Passed

alafanechere · 2022-09-29T18:27:31Z

/test connector=connectors/source-google-sheets

🕑 connectors/source-google-sheets https://github.com/airbytehq/airbyte/actions/runs/3153683284
❌ connectors/source-google-sheets https://github.com/airbytehq/airbyte/actions/runs/3153683284
🐛 https://gradle.com/s/olkzix5ujknqq

Build Failed

Test summary info:

Could not find result summary

alafanechere · 2022-09-29T18:46:40Z

🐞 snapchat-marketing: expired credentials -> tried to get a new refresh token in GSM
🐞 twilio: expected records does not match
🐞 googlesheet: pydantic cdk version does not match connector version -> upgraded google sheets deps

alafanechere · 2022-09-29T18:50:22Z

/test connector=connectors/source-google-sheets

🕑 connectors/source-google-sheets https://github.com/airbytehq/airbyte/actions/runs/3153815671
✅ connectors/source-google-sheets https://github.com/airbytehq/airbyte/actions/runs/3153815671
Python tests coverage:

Name                                                Stmts   Miss  Cover
-----------------------------------------------------------------------
google_sheets_source/models/spreadsheet_values.py      12      0   100%
google_sheets_source/models/spreadsheet.py             34      0   100%
google_sheets_source/models/__init__.py                 2      0   100%
google_sheets_source/__init__.py                        2      0   100%
google_sheets_source/helpers.py                       139     26    81%
google_sheets_source/client.py                         23      6    74%
google_sheets_source/google_sheets_source.py          107     85    21%
-----------------------------------------------------------------------
TOTAL                                                 319    117    63%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       152     26    83%   21-23, 29-31, 36-43, 48-61, 239, 250-258
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-67, 70-72, 75-77, 80-82, 85-87, 90-92, 95-113, 147-149
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1358    466    66%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestIncremental.test_two_sequential_reads because not found in the config
================== 28 passed, 1 skipped in 135.23s (0:02:15) ===================

alafanechere · 2022-09-29T19:02:52Z

/test connector=connectors/source-snapchat-marketing

🕑 connectors/source-snapchat-marketing https://github.com/airbytehq/airbyte/actions/runs/3153884813
❌ connectors/source-snapchat-marketing https://github.com/airbytehq/airbyte/actions/runs/3153884813
🐛 https://gradle.com/s/bifeeolhftihs

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_incremental.py::TestIncremental::test_read_sequential_slices[inputs0]
FAILED test_incremental.py::TestIncremental::test_read_sequential_slices[inputs1]
FAILED test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_incremental.py:23: `future_state_path` not specified, skipping
============= 3 failed, 34 passed, 1 skipped in 335.12s (0:05:35) ==============

brianjlai

Upgrades look good, but I think it's work updating sheets with the new method signature for consistency, beyond that everything looks good! the twilio expected records failure is interesting, I wonder if this was already failing since per-stream should only affect incrementals not full refresh that expected records checks for.

airbyte-integrations/connectors/source-google-sheets/Dockerfile

airbyte-integrations/connectors/source-freshdesk/integration_tests/abnormal_state.json

alafanechere · 2022-09-30T07:56:54Z

/test connector=connectors/source-snapchat-marketing

🕑 connectors/source-snapchat-marketing https://github.com/airbytehq/airbyte/actions/runs/3157173695
❌ connectors/source-snapchat-marketing https://github.com/airbytehq/airbyte/actions/runs/3157173695
🐛 https://gradle.com/s/5qzpghnavdiys

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_incremental.py::TestIncremental::test_read_sequential_slices[inputs0]
FAILED test_incremental.py::TestIncremental::test_read_sequential_slices[inputs1]
FAILED test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_incremental.py:23: `future_state_path` not specified, skipping
============= 3 failed, 34 passed, 1 skipped in 317.84s (0:05:17) ==============

alafanechere · 2022-09-30T08:35:17Z

/test connector=connectors/source-snapchat-marketing

🕑 connectors/source-snapchat-marketing https://github.com/airbytehq/airbyte/actions/runs/3157399383
❌ connectors/source-snapchat-marketing https://github.com/airbytehq/airbyte/actions/runs/3157399383
🐛 https://gradle.com/s/333l34esdjcxe

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs2]
FAILED test_incremental.py::TestIncremental::test_read_sequential_slices[inputs0]
FAILED test_incremental.py::TestIncremental::test_read_sequential_slices[inputs1]
FAILED test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_incremental.py:23: `future_state_path` not specified, skipping
============= 4 failed, 33 passed, 1 skipped in 436.09s (0:07:16) ==============

alafanechere · 2022-09-30T09:43:09Z

/publish connector=connectors/source-google-sheets

🕑 Publishing the following connectors:
connectors/source-google-sheets
https://github.com/airbytehq/airbyte/actions/runs/3157808889

Connector	Did it publish?	Were definitions generated?
connectors/source-google-sheets	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

alafanechere · 2022-09-30T10:33:56Z

/publish connector=connectors/source-freshdesk run-tests=false

🕑 Publishing the following connectors:
connectors/source-freshdesk
https://github.com/airbytehq/airbyte/actions/runs/3158109062

Connector	Did it publish?	Were definitions generated?
connectors/source-freshdesk	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

…m:airbytehq/airbyte into augustin/migrate-ga-to-per-stream-state-3

alafanechere · 2022-09-30T13:13:41Z

/publish connector=connectors/source-twilio

🕑 Publishing the following connectors:
connectors/source-twilio
https://github.com/airbytehq/airbyte/actions/runs/3159023201

Connector	Did it publish?	Were definitions generated?
connectors/source-twilio	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

alafanechere · 2022-09-30T13:50:55Z

I won't fix the snapchat failing tests in this PR: #17461

…ations * master: (75 commits) source-sentry: migrate to per-stream states (#17466) Greg/clickhouse polishing (#17483) upgrade debezium version to 1.9.6 (#17459) 🐛 Source Twilio: Lookback_window config (#17478) hide S3 source connector from catalog (#17472) 🪟 Migrate styles for Connection-related Components (#17339) Added new title (#17480) Refactor & convert `PageTitle` to SCSS (#17139) updated releaseStage for zendesk-talk (#17477) [low-code] Apply log level to stream loggers (#17284) 🐛 Source Salesforce: filter out objects not supported by the Bulk API (#17453) Source Marketo: certify GA (#17445) Update greenhouse paginator (#17429) Add some services start validation to acceptance_tests.sh (#17425) 📖 Removes $ from terminal commands to allow direct copying. (#17467) migrate source GA connectors to per-stream states (2) (#17410) Source Klaviyo: bump CDK dependency (#17422) Source Pinterest: change releaseStage to GA (#17045) Source Pinterest: Set start_date dynamically based on API restrictions for lookup (#17387) updated releaseStage to generally_available (#17374) ...

source-snapchat-marketing

e6709f8

github-actions bot added the area/connectors Connector related issues label Sep 29, 2022

source-snapchat-marketing

669c707

octavia-squidington-iv added the connectors/source/snapchat-marketing label Sep 29, 2022

github-actions bot added the area/documentation Improvements or additions to documentation label Sep 29, 2022

source-twilio

7cab9ae

octavia-squidington-iv added the connectors/source/twilio label Sep 29, 2022

source-freshdesk

53c5df7

source-google-sheets

3ef698e

octavia-squidington-iv added connectors/source/freshdesk connectors/source/google-sheets labels Sep 29, 2022

alafanechere marked this pull request as ready for review September 29, 2022 18:29

Merge branch 'master' into augustin/migrate-ga-to-per-stream-state-3

96b9e17

alafanechere requested review from brianjlai and pedroslopez September 29, 2022 18:29

alafanechere mentioned this pull request Sep 29, 2022

[per-stream cdk] publish all GA API source connectors #13136

Closed

upgrade pydantic

cb54bb0

brianjlai approved these changes Sep 30, 2022

View reviewed changes

airbyte-integrations/connectors/source-google-sheets/Dockerfile Show resolved Hide resolved

airbyte-integrations/connectors/source-freshdesk/integration_tests/abnormal_state.json Outdated Show resolved Hide resolved

change read signature in google-sheets

20a2c02

reformat json

76b8cb4

auto-bump connector version [ci skip]

2ab1638

octavia-squidington-iii and others added 3 commits September 30, 2022 10:48

auto-bump connector version [ci skip]

c9dcc03

update expected record

aeec8cd

Merge branch 'augustin/migrate-ga-to-per-stream-state-3' of github.co…

6673659

…m:airbytehq/airbyte into augustin/migrate-ga-to-per-stream-state-3

alafanechere temporarily deployed to more-secrets September 30, 2022 13:15 Inactive

auto-bump connector version [ci skip]

562da96

revert snapchat changes

3fe7b1a

alafanechere temporarily deployed to more-secrets September 30, 2022 13:59 Inactive

alafanechere merged commit 7ab162a into master Sep 30, 2022

alafanechere deleted the augustin/migrate-ga-to-per-stream-state-3 branch September 30, 2022 14:44

artem1205 pushed a commit that referenced this pull request Sep 30, 2022

migrate source GA connectors to per-stream states (2) (#17410)

eb9d27e

octavia-squidington-iii mentioned this pull request Oct 5, 2022

Bump Airbyte version from 0.40.10 to 0.40.11 #17610

Merged

jhammarstedt pushed a commit to jhammarstedt/airbyte that referenced this pull request Oct 31, 2022

migrate source GA connectors to per-stream states (2) (airbytehq#17410)

9457db9

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

migrate source GA connectors to per-stream states (2) #17410

migrate source GA connectors to per-stream states (2) #17410

alafanechere commented Sep 29, 2022 •

edited

Loading

alafanechere commented Sep 29, 2022 •

edited by github-actions bot

Loading

alafanechere commented Sep 29, 2022 •

edited by github-actions bot

Loading

alafanechere commented Sep 29, 2022 •

edited by github-actions bot

Loading

alafanechere commented Sep 29, 2022 •

edited by github-actions bot

Loading

alafanechere commented Sep 29, 2022 •

edited

Loading

alafanechere commented Sep 29, 2022 •

edited by github-actions bot

Loading

alafanechere commented Sep 29, 2022 •

edited by github-actions bot

Loading

brianjlai left a comment

alafanechere commented Sep 30, 2022 •

edited by github-actions bot

Loading

alafanechere commented Sep 30, 2022 •

edited by github-actions bot

Loading

alafanechere commented Sep 30, 2022 •

edited by github-actions bot

Loading

alafanechere commented Sep 30, 2022 •

edited by github-actions bot

Loading

alafanechere commented Sep 30, 2022 •

edited by github-actions bot

Loading

alafanechere commented Sep 30, 2022

migrate source GA connectors to per-stream states (2) #17410

migrate source GA connectors to per-stream states (2) #17410

Conversation

alafanechere commented Sep 29, 2022 • edited Loading

What

alafanechere commented Sep 29, 2022 • edited by github-actions bot Loading

Build Failed

alafanechere commented Sep 29, 2022 • edited by github-actions bot Loading

Build Failed

alafanechere commented Sep 29, 2022 • edited by github-actions bot Loading

Build Passed

alafanechere commented Sep 29, 2022 • edited by github-actions bot Loading

Build Failed

alafanechere commented Sep 29, 2022 • edited Loading

alafanechere commented Sep 29, 2022 • edited by github-actions bot Loading

Build Passed

alafanechere commented Sep 29, 2022 • edited by github-actions bot Loading

Build Failed

brianjlai left a comment

Choose a reason for hiding this comment

alafanechere commented Sep 30, 2022 • edited by github-actions bot Loading

Build Failed

alafanechere commented Sep 30, 2022 • edited by github-actions bot Loading

Build Failed

alafanechere commented Sep 30, 2022 • edited by github-actions bot Loading

alafanechere commented Sep 30, 2022 • edited by github-actions bot Loading

alafanechere commented Sep 30, 2022 • edited by github-actions bot Loading

alafanechere commented Sep 30, 2022

alafanechere commented Sep 29, 2022 •

edited

Loading

alafanechere commented Sep 29, 2022 •

edited by github-actions bot

Loading

alafanechere commented Sep 29, 2022 •

edited by github-actions bot

Loading

alafanechere commented Sep 29, 2022 •

edited by github-actions bot

Loading

alafanechere commented Sep 29, 2022 •

edited by github-actions bot

Loading

alafanechere commented Sep 29, 2022 •

edited

Loading

alafanechere commented Sep 29, 2022 •

edited by github-actions bot

Loading

alafanechere commented Sep 29, 2022 •

edited by github-actions bot

Loading

alafanechere commented Sep 30, 2022 •

edited by github-actions bot

Loading

alafanechere commented Sep 30, 2022 •

edited by github-actions bot

Loading

alafanechere commented Sep 30, 2022 •

edited by github-actions bot

Loading

alafanechere commented Sep 30, 2022 •

edited by github-actions bot

Loading

alafanechere commented Sep 30, 2022 •

edited by github-actions bot

Loading