Source Zendesk Chat: Process large amount of data in batches for incr…

…emental (#14214) * increased the limit of itens in request * Configuration for max api pages on requests * included api_pagination_limit in sample * included api_pagination_limit in invalid_config * creating new table for chat_session * reverted api_pagination_limit approach * removed api_pagination_limit from TimeIncrementalStream * correct chat json * bump connector version * add changelog * run format * auto-bump connector version Co-authored-by: Roberto Bonnet <robertojuarezwp@gmail.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
airbytehq · Jun 28, 2022 · 4a51c89 · 4a51c89
1 parent 45446de
commit 4a51c89
Show file tree

Hide file tree

Showing 6 changed files with 45 additions and 5 deletions.
diff --git a/airbyte-config/init/src/main/resources/seed/source_definitions.yaml b/airbyte-config/init/src/main/resources/seed/source_definitions.yaml
@@ -1025,7 +1025,7 @@
 - name: Zendesk Chat
   sourceDefinitionId: 40d24d0f-b8f9-4fe0-9e6c-b06c0f3f45e4
   dockerRepository: airbyte/source-zendesk-chat
-  dockerImageTag: 0.1.7
+  dockerImageTag: 0.1.8
   documentationUrl: https://docs.airbyte.io/integrations/sources/zendesk-chat
   icon: zendesk.svg
   sourceType: api

diff --git a/airbyte-config/init/src/main/resources/seed/source_specs.yaml b/airbyte-config/init/src/main/resources/seed/source_specs.yaml
@@ -9594,7 +9594,7 @@
     supportsNormalization: false
     supportsDBT: false
     supported_destination_sync_modes: []
-- dockerImage: "airbyte/source-zendesk-chat:0.1.7"
+- dockerImage: "airbyte/source-zendesk-chat:0.1.8"
   spec:
     documentationUrl: "https://docs.airbyte.io/integrations/sources/zendesk-chat"
     connectionSpecification:

diff --git a/airbyte-integrations/connectors/source-zendesk-chat/Dockerfile b/airbyte-integrations/connectors/source-zendesk-chat/Dockerfile
@@ -16,5 +16,5 @@ RUN pip install .
 
 ENTRYPOINT ["python", "/airbyte/integration_code/main_dev.py"]
 
-LABEL io.airbyte.version=0.1.7
+LABEL io.airbyte.version=0.1.8
 LABEL io.airbyte.name=airbyte/source-zendesk-chat
diff --git a/airbyte-integrations/connectors/source-zendesk-chat/source_zendesk_chat/schemas/chats.json b/airbyte-integrations/connectors/source-zendesk-chat/source_zendesk_chat/schemas/chats.json
@@ -207,8 +207,43 @@
       "additionalProperties": false
     },
     "session": {
-      "type": ["null", "object"],
-      "additionalProperties": true
+      "properties": {
+        "end_date": {
+          "format": "date-time",
+          "type": ["null", "string"]
+        },
+        "country_code": {
+          "type": ["null", "string"]
+        },
+        "city": {
+          "type": ["null", "string"]
+        },
+        "browser": {
+          "type": ["null", "string"]
+        },
+        "ip": {
+          "type": ["null", "string"]
+        },
+        "country_name": {
+          "type": ["null", "string"]
+        },
+        "id": {
+          "type": ["null", "string"]
+        },
+        "region": {
+          "type": ["null", "string"]
+        },
+        "platform": {
+          "type": ["null", "string"]
+        },
+        "user_agent": {
+          "type": ["null", "string"]
+        },
+        "start_date": {
+          "format": "date-time",
+          "type": ["null", "string"]
+        }
+      }
     },
     "started_by": {
       "type": ["null", "string"]

diff --git a/airbyte-integrations/connectors/source-zendesk-chat/source_zendesk_chat/streams.py b/airbyte-integrations/connectors/source-zendesk-chat/source_zendesk_chat/streams.py
@@ -104,6 +104,8 @@ def _field_to_datetime(value: Union[int, str]) -> pendulum.datetime:
 
 
 class TimeIncrementalStream(BaseIncrementalStream, ABC):
+    state_checkpoint_interval = 1000
+
     def __init__(self, start_date, **kwargs):
         super().__init__(**kwargs)
         self._start_date = pendulum.parse(start_date)
@@ -186,6 +188,7 @@ class AgentTimelines(TimeIncrementalStream):
     cursor_field = "start_time"
     data_field = "agent_timeline"
     name = "agent_timeline"
+    limit = 1000
 
     def request_params(self, **kwargs) -> MutableMapping[str, Any]:
         params = super().request_params(**kwargs)
@@ -226,6 +229,7 @@ class Chats(TimeIncrementalStream):
 
     cursor_field = "update_timestamp"
     data_field = "chats"
+    limit = 1000
 
 
 class Shortcuts(Stream):

diff --git a/docs/integrations/sources/zendesk-chat.md b/docs/integrations/sources/zendesk-chat.md
@@ -72,6 +72,7 @@ We recommend creating a restricted, read-only key specifically for Airbyte acces
 
 | Version | Date | Pull Request | Subject |
 | :--- | :--- | :--- | :--- |
+| 0.1.8 | 2022-06-28 | [13387](https://github.com/airbytehq/airbyte/pull/13387) | Add state checkpoint to allow long runs |
 | 0.1.7 | 2022-05-25 | [12883](https://github.com/airbytehq/airbyte/pull/12883) | Passing timeout in request to prevent a stuck connection |
 | 0.1.6 | 2021-12-15 | [7313](https://github.com/airbytehq/airbyte/pull/7313) | Added support of `OAuth 2.0` authentication. [8819](https://github.com/airbytehq/airbyte/pull/8819) Fixed the issue with `created_at` can now be `null` for `bans` stream |
 | 0.1.5 | 2021-12-06 | [8425](https://github.com/airbytehq/airbyte/pull/8425) | Update title, description fields in spec |