🐛 Source Zendesk Talk: migrate to CDK #7173

gaart · 2021-10-19T15:17:48Z

What

Closes #6599 and migrate to CDK implementation

How

Handle the next page link on the last page, upon fixing the initial issue I decided to enable SAT, but this required proper catalog and features of CDK, so I decided to migrate to CDK as well.
bonus features:

fixed a few issues with incorrect json schema
added caching
added correct backoff that use retry-after values
improve records layout to avoid data duplication in different streams (see IVR-related streams)

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

gaart · 2021-10-19T15:18:09Z

/test connector=connectors/source-zendesk-talk

🕑 connectors/source-zendesk-talk https://github.com/airbytehq/airbyte/actions/runs/1359797632
❌ connectors/source-zendesk-talk https://github.com/airbytehq/airbyte/actions/runs/1359797632
🐛

gaart · 2021-10-31T10:27:09Z

/test connector=connectors/source-zendesk-talk

🕑 connectors/source-zendesk-talk https://github.com/airbytehq/airbyte/actions/runs/1404462071
✅ connectors/source-zendesk-talk https://github.com/airbytehq/airbyte/actions/runs/1404462071
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                       Stmts   Miss  Cover
	 --------------------------------------------------------------
	 base_python/__init__.py                       13      0   100%
	 base_python/catalog_helpers.py                10      6    40%
	 base_python/cdk/__init__.py                    0      0   100%
	 base_python/cdk/abstract_source.py            83     59    29%
	 base_python/cdk/streams/__init__.py            0      0   100%
	 base_python/cdk/streams/auth/__init__.py       0      0   100%
	 base_python/cdk/streams/auth/core.py           8      1    88%
	 base_python/cdk/streams/auth/jwt.py            5      5     0%
	 base_python/cdk/streams/auth/oauth.py         37     26    30%
	 base_python/cdk/streams/auth/token.py          9      4    56%
	 base_python/cdk/streams/core.py               63     32    49%
	 base_python/cdk/streams/exceptions.py         10      2    80%
	 base_python/cdk/streams/http.py               67     33    51%
	 base_python/cdk/streams/rate_limiting.py      30     14    53%
	 base_python/cdk/utils/__init__.py              0      0   100%
	 base_python/cdk/utils/casing.py                4      0   100%
	 base_python/client.py                         56     33    41%
	 base_python/entrypoint.py                     70     56    20%
	 base_python/integration.py                    52     25    52%
	 base_python/logger.py                         33     19    42%
	 base_python/schema_helpers.py                 56     41    27%
	 base_python/source.py                         51     34    33%
	 main_dev.py                                    3      3     0%
	 --------------------------------------------------------------
	 TOTAL                                        660    393    40%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                              Stmts   Miss  Cover
	 -----------------------------------------------------
	 main_dev.py                           6      6     0%
	 source_zendesk_talk/__init__.py       2      0   100%
	 source_zendesk_talk/api.py          169     45    73%
	 source_zendesk_talk/client.py        28     11    61%
	 source_zendesk_talk/errors.py         6      0   100%
	 source_zendesk_talk/source.py         4      0   100%
	 -----------------------------------------------------
	 TOTAL                               215     62    71%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                              Stmts   Miss  Cover
	 -----------------------------------------------------
	 main_dev.py                           6      6     0%
	 source_zendesk_talk/__init__.py       2      0   100%
	 source_zendesk_talk/api.py          169     44    74%
	 source_zendesk_talk/client.py        28      3    89%
	 source_zendesk_talk/errors.py         6      0   100%
	 source_zendesk_talk/source.py         4      0   100%
	 -----------------------------------------------------
	 TOTAL                               215     53    75%

harshithmullapudi · 2021-11-09T11:51:57Z

@keu any update on this ?

CLAassistant · 2021-11-09T16:35:53Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ gaart
❌ eugene-kulak
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

…pagination

keu · 2021-11-13T02:43:20Z

/test connector=source-zendesk-talk

🕑 source-zendesk-talk https://github.com/airbytehq/airbyte/actions/runs/1455549052
❌ source-zendesk-talk https://github.com/airbytehq/airbyte/actions/runs/1455549052
🐛 https://gradle.com/s/oylpeu3i4jucq

sherifnada

Looks great

sherifnada · 2021-11-13T08:12:03Z

airbyte-integrations/connectors/source-zendesk-talk/unit_tests/unit_test.py

-
-    assert alive
-    assert not error
+def test_example_method():


we can probably remove this file

…pagination

keu · 2021-11-13T21:34:25Z

/test connector=source-zendesk-talk

🕑 source-zendesk-talk https://github.com/airbytehq/airbyte/actions/runs/1457411908
✅ source-zendesk-talk https://github.com/airbytehq/airbyte/actions/runs/1457411908
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        75      8    89%
	 source_acceptance_test/conftest.py                     108    108     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              200     94    53%
	 source_acceptance_test/tests/test_full_refresh.py       38     27    29%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  41     24    41%
	 source_acceptance_test/utils/compare.py                 62     25    60%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  896    440    51%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                              Stmts   Miss  Cover
	 -----------------------------------------------------
	 source_zendesk_talk/__init__.py       2      0   100%
	 source_zendesk_talk/source.py        33     13    61%
	 source_zendesk_talk/streams.py      139     20    86%
	 -----------------------------------------------------
	 TOTAL                               174     33    81%

keu · 2021-11-13T21:42:15Z

/publish connector=connectors/source-zendesk-talk

🕑 connectors/source-zendesk-talk https://github.com/airbytehq/airbyte/actions/runs/1457424464
✅ connectors/source-zendesk-talk https://github.com/airbytehq/airbyte/actions/runs/1457424464

* Upd pagination edge case * migrate to CDK and connect SAT + added primary key based on datetime.now + added unittests Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>

Upd pagination edge case

7ad0cc8

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Oct 19, 2021

gaart linked an issue Oct 19, 2021 that may be closed by this pull request

Zendesk Talk appears to paginate results #6599

Closed

jrhizor temporarily deployed to more-secrets October 19, 2021 15:19 Inactive

danieldiamond mentioned this pull request Oct 29, 2021

🐛 Source Zendesk chat: fix chats stream is only pulling for first page #7210

Merged

38 tasks

Merge branch 'master' into gaart/6599-zendesk-pagination

8024d7f

gaart temporarily deployed to more-secrets October 31, 2021 10:28 Inactive

jrhizor temporarily deployed to more-secrets October 31, 2021 10:29 Inactive

migrate to CDK and connect SAT

ce990b3

keu temporarily deployed to more-secrets November 9, 2021 16:37 Inactive

clean-up

112186b

keu temporarily deployed to more-secrets November 9, 2021 21:04 Inactive

format

c6dcfb7

keu temporarily deployed to more-secrets November 9, 2021 22:20 Inactive

eugene-kulak added 2 commits November 10, 2021 01:15

fix dockerfile

3dc0f59

Merge remote-tracking branch 'origin/master' into gaart/6599-zendesk-…

525b8f7

…pagination

keu temporarily deployed to more-secrets November 9, 2021 23:18 Inactive

fix tests

004aa14

keu temporarily deployed to more-secrets November 9, 2021 23:54 Inactive

fix stream names

8d78298

keu temporarily deployed to more-secrets November 10, 2021 00:21 Inactive

github-actions bot added the CDK Connector Development Kit label Nov 10, 2021

keu temporarily deployed to more-secrets November 10, 2021 18:08 Inactive

keu changed the title ~~🐛 Source Zendesk Talk: Fix pagination edge case~~ 🐛 Source Zendesk Talk: migrate to CDK Nov 11, 2021

keu requested a review from sherifnada November 11, 2021 00:13

Merge remote-tracking branch 'origin/master' into gaart/6599-zendesk-…

c806553

…pagination

keu temporarily deployed to more-secrets November 13, 2021 02:06 Inactive

fix tests

3b9dfda

keu temporarily deployed to more-secrets November 13, 2021 02:39 Inactive

format

903c9bb

keu requested a review from sherifnada November 13, 2021 02:42

keu temporarily deployed to more-secrets November 13, 2021 02:43 Inactive

jrhizor temporarily deployed to more-secrets November 13, 2021 02:45 Inactive

sherifnada approved these changes Nov 13, 2021

View reviewed changes

fix schema and update spec.json

8ef5ae8

keu temporarily deployed to more-secrets November 13, 2021 21:18 Inactive

eugene-kulak added 2 commits November 13, 2021 21:29

fix tests

d909f8a

Merge remote-tracking branch 'origin/master' into gaart/6599-zendesk-…

e3ac241

…pagination

keu temporarily deployed to more-secrets November 13, 2021 21:31 Inactive

eugene-kulak added 2 commits November 13, 2021 23:33

revert bumping

4fa7e39

delete obsolete unittest

6e195f3

keu temporarily deployed to more-secrets November 13, 2021 21:35 Inactive

jrhizor temporarily deployed to more-secrets November 13, 2021 21:36 Inactive

jrhizor temporarily deployed to more-secrets November 13, 2021 21:44 Inactive

bumping

d31d6b2

keu merged commit 5a24ef5 into master Nov 13, 2021

keu deleted the gaart/6599-zendesk-pagination branch November 13, 2021 21:59

keu temporarily deployed to more-secrets November 13, 2021 22:00 Inactive

This was referenced Nov 16, 2021

Bump Airbyte version from 0.32.0-alpha to 0.32.1-alpha #8023

Closed

Bump Airbyte version from 0.32.0-alpha to 0.32.1-alpha #8044

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Source Zendesk Talk: migrate to CDK #7173

🐛 Source Zendesk Talk: migrate to CDK #7173

gaart commented Oct 19, 2021 •

edited by keu

Loading

gaart commented Oct 19, 2021 •

edited by github-actions bot

Loading

gaart commented Oct 31, 2021 •

edited by github-actions bot

Loading

harshithmullapudi commented Nov 9, 2021

CLAassistant commented Nov 9, 2021

keu commented Nov 13, 2021 •

edited by github-actions bot

Loading

sherifnada left a comment

sherifnada Nov 13, 2021

keu Nov 13, 2021

keu commented Nov 13, 2021 •

edited by github-actions bot

Loading

keu commented Nov 13, 2021 •

edited by github-actions bot

Loading

🐛 Source Zendesk Talk: migrate to CDK #7173

🐛 Source Zendesk Talk: migrate to CDK #7173

Conversation

gaart commented Oct 19, 2021 • edited by keu Loading

What

How

Pre-merge Checklist

Community member or Airbyter

Airbyter

gaart commented Oct 19, 2021 • edited by github-actions bot Loading

gaart commented Oct 31, 2021 • edited by github-actions bot Loading

harshithmullapudi commented Nov 9, 2021

CLAassistant commented Nov 9, 2021

keu commented Nov 13, 2021 • edited by github-actions bot Loading

sherifnada left a comment

Choose a reason for hiding this comment

sherifnada Nov 13, 2021

Choose a reason for hiding this comment

keu Nov 13, 2021

Choose a reason for hiding this comment

keu commented Nov 13, 2021 • edited by github-actions bot Loading

keu commented Nov 13, 2021 • edited by github-actions bot Loading

gaart commented Oct 19, 2021 •

edited by keu

Loading

gaart commented Oct 19, 2021 •

edited by github-actions bot

Loading

gaart commented Oct 31, 2021 •

edited by github-actions bot

Loading

keu commented Nov 13, 2021 •

edited by github-actions bot

Loading

keu commented Nov 13, 2021 •

edited by github-actions bot

Loading

keu commented Nov 13, 2021 •

edited by github-actions bot

Loading