🐛 Source Zendesk Support/Chat: got stuck after 100k records

[marcosmarxm]

Is this your first time deploying Airbyte: No
OS Version / Instance: Ubuntu 20.04
Memory / Disk: 8Gb / 100 GB ssd
Deployment: Docker
Airbyte Version: 0.35.61-alpha
Source name/version: airbyte/source-zendesk-chat:0.1.6 (latest)
Destination name/version: airbyte/destination-bigquery: 1.0.5 (latest)
Step: normal sync.
Description: everytime the process starts to sync it freezes after some hours, the same happens with source-zendesk-support:0.2.3 (latest).

See discussion in Discourse: https://discuss.airbyte.io/t/zendesk-chat-and-zendesk-support-freezes-after-some-hours/455

[RobertoBonnet]

Hello guys. I have a solution for Zendesk Chat

[RobertoBonnet]

TL DR;

• Zendesk API changed the rate limit for offset pagination. Rate limit down to 10 request/minute after page 1000 -https://support.zendesk.com/hc/en-us/articles/4408824785946-Announcing-API-rate-limits-for-Offset-based-Pagination
• Sleep to asynchronous requests with 8 works (default) is a problem -https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-zendesk-support/source_zendesk_support/streams.py#L275

I started to debug the connector to Zendesk suport. The problem only affect clients with to much data, indeed.

Some routes requests are mades using asynchronous requests. This work to small data, but the process is stucking everything when the error start for large data.

Searching I have found some changed in Zendesk API that affect this code. The rate limit change for pagination greater than 1000. Here at Hurb we are making some changes:

SatisfactionRatings - Process request by request using cursor-based pagination
TicketMetrics - The cursor-based pagination aren't working here. There is a error in Zendesk DB. To avoid the problem, we are processing request by request applying the rule of 10 request per minute after pagination 1000 using offset-based pagination

[RobertoBonnet]

There is a way to see the error. Just make a request to https://{account}.zendesk.com/api/v2/ticket_metrics.json?page=999999 11 times in a minute

[marcosmarxm]

Thanks @RobertoBonnet to dive into this! Really helpful analysis

[marcosmarxm]

There is a way to see the error. Just make a request to https://{account}.zendesk.com/api/v2/ticket_metrics.json?page=999999 11 times in a minute

Should we add this new request limits to your PR?

[RobertoBonnet]

@marcosmarxm It won't be needed by the tests I performed. The way the data is fetched does not go through this rate limit. I made some changes and I'm processing the full sync. Waiting to see what will happen. 3 days have passed.

[RobertoBonnet]

The task failed :(. Is there any strategy to not lose all the data as downloaded? There are so many data from Zendesk here and, when the task faills, we lost to many time. I'll try to limit the pagination of each router from api that we use incremental.

[RobertoBonnet]

logs-428.txt

[marcosmarxm]

The task failed :(. Is there any strategy to not lose all the data as downloaded? There are so many data from Zendesk here and, when the task faills, we lost to many time. I'll try to limit the pagination of each router from api that we use incremental.

You must use checkpoint to send data to destination and next time dont need to start from scratch. but only works with incremental streams

[bazarnov]

Uploading logs-428.txt…

@RobertoBonnet Please attach the valid logs, since i'm afraid this link leads to other PR - not logs)

[RobertoBonnet]

@bazarnov Sorry... I fixed the link

[bazarnov]

Similar issue: #12155

[marcosmarxm]

@RobertoBonnet are you going to work for Zendesk Support too?