Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Source Zendesk Support: fix pagination for stream TicketMetricEvents #28612

Merged
merged 4 commits into from
Jul 25, 2023
Merged

Source Zendesk Support: fix pagination for stream TicketMetricEvents #28612

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
ddf6b1f
Source Zendesk Support: fix pagination for stream TicketMetricEvents
artem1205 Jul 24, 2023
4cf8113
Source Zendesk Support: update docs
artem1205 Jul 24, 2023
a093ef0
Source Zendesk Support: fix formatting
artem1205 Jul 24, 2023
600ca70
Source Zendesk Support: add test for ticket_metric_events
artem1205 Jul 25, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-zendesk-support/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,5 +25,5 @@ COPY source_zendesk_support ./source_zendesk_support
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

LABEL io.airbyte.version=0.10.2
LABEL io.airbyte.version=0.10.3
LABEL io.airbyte.name=airbyte/source-zendesk-support
5 changes: 1 addition & 4 deletions airbyte-integrations/connectors/source-zendesk-support/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,16 +7,13 @@ data:
connectorType: source
maxSecondsBetweenMessages: 10800
definitionId: 79c1aa37-dae3-42ae-b333-d1c105477715
dockerImageTag: 0.10.2
dockerImageTag: 0.10.3
dockerRepository: airbyte/source-zendesk-support
githubIssueLabel: source-zendesk-support
icon: zendesk-support.svg
license: ELv2
name: Zendesk Support
registries:
cloud:
dockerImageTag: 0.10.0 # Fixing P0 https://github.com/airbytehq/oncall/issues/2530
enabled: true
oss:
enabled: true
releaseStage: generally_available
Expand Down
14 changes: 14 additions & 0 deletions airbyte-integrations/connectors/source-zendesk-support/source_zendesk_support/streams.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -614,6 +614,20 @@ class TicketMetricEvents(CursorPaginationZendeskSupportStream):
def path(self, **kwargs):
return "incremental/ticket_metric_events"

def request_params(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@artem1205 ideally we would have some test coverage for this. Is there a reason it's not covered by integration tests?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All those stream use the same pagination mechanism implemented in CursorPaginationZendeskSupportStream, and raise Error if start_date IS IN the request_params. But yep, i'll make another test for this particular stream.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay thanks. It seems that there probably wasn't test coverage for the other streams too, since there wasn't a test failure that alerted us to the issue before the code was merged. So I think it's definitely worthwhile to add it.

self,
stream_state: Mapping[str, Any],
stream_slice: Mapping[str, Any] = None,
next_page_token: Mapping[str, Any] = None,
) -> MutableMapping[str, Any]:
params = {
"start_time": self.check_stream_state(stream_state),
"page[size]": self.page_size,
}
if next_page_token: # need keep start_time for this stream
params.update(next_page_token)
return params


class Macros(SourceZendeskSupportStream):
"""Macros stream: https://developer.zendesk.com/api-reference/ticketing/business-rules/macros/"""
Expand Down
94 changes: 72 additions & 22 deletions airbyte-integrations/connectors/source-zendesk-support/unit_tests/unit_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -415,10 +415,10 @@ def test_url_base(self, stream_cls):
(Macros, {}, {"updated_at": "2022-03-17T16:03:07Z"}, {"updated_at": "2022-03-17T16:03:07Z"}),
(Posts, {}, {"updated_at": "2022-03-17T16:03:07Z"}, {"updated_at": "2022-03-17T16:03:07Z"}),
(
Organizations,
{"updated_at": "2022-03-17T16:03:07Z"},
{"updated_at": "2023-03-17T16:03:07Z"},
{"updated_at": "2023-03-17T16:03:07Z"},
Organizations,
{"updated_at": "2022-03-17T16:03:07Z"},
{"updated_at": "2023-03-17T16:03:07Z"},
{"updated_at": "2023-03-17T16:03:07Z"},
),
(Groups, {}, {"updated_at": "2022-03-17T16:03:07Z"}, {"updated_at": "2022-03-17T16:03:07Z"}),
(SatisfactionRatings, {}, {"updated_at": "2022-03-17T16:03:07Z"}, {"updated_at": "2022-03-17T16:03:07Z"}),
Expand Down Expand Up @@ -604,30 +604,37 @@ def test_get_updated_state(self, stream_cls, current_state, last_record, expecte
[
(GroupMemberships, {}, None),
(TicketForms, {}, None),
(TicketMetricEvents, {}, None),
(TicketMetricEvents, {
"meta": {"has_more": True, "after_cursor": "<after_cursor>", "before_cursor": "<before_cursor>"},
"links": {
"prev": "https://subdomain.zendesk.com/api/v2/ticket_metrics.json?page%5Bbefore%5D=<before_cursor>%3D&page%5Bsize%5D=2",
"next": "https://subdomain.zendesk.com/api/v2/ticket_metrics.json?page%5Bafter%5D=<after_cursor>%3D&page%5Bsize%5D=2",
},
},
{"page[after]": "<after_cursor>"}),
(TicketAudits, {}, None),
(
TicketMetrics,
{
"meta": {"has_more": True, "after_cursor": "<after_cursor>", "before_cursor": "<before_cursor>"},
"links": {
"prev": "https://subdomain.zendesk.com/api/v2/ticket_metrics.json?page%5Bbefore%5D=<before_cursor>%3D&page%5Bsize%5D=2",
"next": "https://subdomain.zendesk.com/api/v2/ticket_metrics.json?page%5Bafter%5D=<after_cursor>%3D&page%5Bsize%5D=2",
TicketMetrics,
{
"meta": {"has_more": True, "after_cursor": "<after_cursor>", "before_cursor": "<before_cursor>"},
"links": {
"prev": "https://subdomain.zendesk.com/api/v2/ticket_metrics.json?page%5Bbefore%5D=<before_cursor>%3D&page%5Bsize%5D=2",
"next": "https://subdomain.zendesk.com/api/v2/ticket_metrics.json?page%5Bafter%5D=<after_cursor>%3D&page%5Bsize%5D=2",
},
},
},
{"page[after]": "<after_cursor>"},
{"page[after]": "<after_cursor>"},
),
(SatisfactionRatings, {}, None),
(
OrganizationMemberships,
{
"meta": {"has_more": True, "after_cursor": "<after_cursor>", "before_cursor": "<before_cursor>"},
"links": {
"prev": "https://subdomain.zendesk.com/api/v2/ticket_metrics.json?page%5Bbefore%5D=<before_cursor>%3D&page%5Bsize%5D=2",
"next": "https://subdomain.zendesk.com/api/v2/ticket_metrics.json?page%5Bafter%5D=<after_cursor>%3D&page%5Bsize%5D=2",
OrganizationMemberships,
{
"meta": {"has_more": True, "after_cursor": "<after_cursor>", "before_cursor": "<before_cursor>"},
"links": {
"prev": "https://subdomain.zendesk.com/api/v2/ticket_metrics.json?page%5Bbefore%5D=<before_cursor>%3D&page%5Bsize%5D=2",
"next": "https://subdomain.zendesk.com/api/v2/ticket_metrics.json?page%5Bafter%5D=<after_cursor>%3D&page%5Bsize%5D=2",
},
},
},
{"page[after]": "<after_cursor>"},
{"page[after]": "<after_cursor>"},
),
(
TicketSkips,
Expand Down Expand Up @@ -655,7 +662,6 @@ def test_get_updated_state(self, stream_cls, current_state, last_record, expecte
)
def test_next_page_token(self, requests_mock, stream_cls, response, expected):
stream = stream_cls(**STREAM_ARGS)
# stream_name = snake_case(stream.__class__.__name__)
requests_mock.get(STREAM_URL, json=response)
test_response = requests.get(STREAM_URL)
output = stream.next_page_token(test_response)
Expand Down Expand Up @@ -948,3 +954,47 @@ def test_read_post_comment_votes_stream(requests_mock):
stream = PostCommentVotes(subdomain="subdomain", start_date="2020-01-01T00:00:00Z")
records = read_full_refresh(stream)
assert records == votes


def test_read_ticket_metric_events_request_params(requests_mock):
first_page_response = {
"ticket_metric_events": [
{"id": 1, "ticket_id": 123, "metric": "agent_work_time", "instance_id": 0, "type": "measure", "time": "2020-01-01T01:00:00Z"},
{"id": 2, "ticket_id": 123, "metric": "pausable_update_time", "instance_id": 0, "type": "measure", "time": "2020-01-01T01:00:00Z"},
{"id": 3, "ticket_id": 123, "metric": "reply_time", "instance_id": 0, "type": "measure", "time": "2020-01-01T01:00:00Z"},
{"id": 4, "ticket_id": 123, "metric": "requester_wait_time", "instance_id": 1, "type": "activate", "time": "2020-01-01T01:00:00Z"}
],
"meta": {
"has_more": True,
"after_cursor": "<after_cursor>",
"before_cursor": "<before_cursor>"
},
"links": {
"prev": "https://subdomain.zendesk.com/api/v2/incremental/ticket_metric_events.json?page%5Bbefore%5D=<before_cursor>&page%5Bsize%5D=100&start_time=1577836800",
"next": "https://subdomain.zendesk.com/api/v2/incremental/ticket_metric_events.json?page%5Bafter%5D=<after_cursor>&page%5Bsize%5D=100&start_time=1577836800"
},
"end_of_stream": False
}

second_page_response = {
"ticket_metric_events": [
{"id": 5163373143183, "ticket_id": 130, "metric": "reply_time", "instance_id": 1, "type": "fulfill", "time": "2022-07-18T16:39:48Z"},
{"id": 5163373143311, "ticket_id": 130, "metric": "requester_wait_time", "instance_id": 0, "type": "measure", "time": "2022-07-18T16:39:48Z"}
],
"meta": {
"has_more": False,
"after_cursor": "<before_cursor>",
"before_cursor": "<before_cursor>"
},
"links": {
"prev": "https://subdomain.zendesk.com/api/v2/incremental/ticket_metric_events.json?page%5Bbefore%5D=<before_cursor>&page%5Bsize%5D=100&start_time=1577836800",
"next": "https://subdomain.zendesk.com/api/v2/incremental/ticket_metric_events.json?page%5Bbefore%5D=<before_cursor>&page%5Bsize%5D=100&start_time=1577836800"
},
"end_of_stream": True
}
request_history = requests_mock.get("https://subdomain.zendesk.com/api/v2/incremental/ticket_metric_events", [
{"json": first_page_response}, {"json": second_page_response}])
stream = TicketMetricEvents(subdomain="subdomain", start_date="2020-01-01T00:00:00Z")
read_full_refresh(stream)
assert request_history.call_count == 2
assert request_history.last_request.qs == {"page[after]": ["<after_cursor>"], "page[size]": ["100"], "start_time": ["1577836800"]}
1 change: 1 addition & 0 deletions docs/integrations/sources/zendesk-support.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ The Zendesk connector ideally should not run into Zendesk API limitations under

| Version | Date | Pull Request | Subject |
|:---------|:-----------|:---------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `0.10.3` | 2023-07-24 | [28612](https://github.com/airbytehq/airbyte/pull/28612) | Fix pagination for stream `TicketMetricEvents` |
| `0.10.2` | 2023-07-19 | [28487](https://github.com/airbytehq/airbyte/pull/28487) | Remove extra page from params |
| `0.10.1` | 2023-07-10 | [28096](https://github.com/airbytehq/airbyte/pull/28096) | Replace `offset` pagination with `cursor` pagination |
| `0.10.0` | 2023-07-06 | [27991](https://github.com/airbytehq/airbyte/pull/27991) | add streams: `PostVotes`, `PostCommentVotes` |
Expand Down