TDL-6756: Fix Infinite Loop for Users #103
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dbshah1212
reviewed
Apr 4, 2022
dbshah1212
reviewed
Apr 4, 2022
dbshah1212
reviewed
Apr 5, 2022
tap_zendesk/streams.py
Outdated
| users = self.client.search("", updated_after=parsed_start, updated_before=parsed_end, type="user") | ||
|
|
||
| # NB: Zendesk will return an error on the 1001st record, so we | ||
| # need to check total response size before iterating | ||
| # See: https://develop.zendesk.com/hc/en-us/articles/360022563994--BREAKING-New-Search-API-Result-Limits | ||
| if users.count > 1000: | ||
| if search_window_size > 1: | ||
| # to avoid infinite loop behavior we should reduce the window if it is greater than 2 |
There was a problem hiding this comment.
Suggested change
| # to avoid infinite loop behavior we should reduce the window if it is greater than 2 | |
| # To avoid infinite loop behavior we should reduce the window if it is greater than 2 |
There was a problem hiding this comment.
Added suggested changes
dbshah1212
approved these changes
Apr 5, 2022
savan-chovatiya
approved these changes
Apr 18, 2022
savan-chovatiya
requested review from
KrisPersonal,
dmosorast,
kspeer825 and
RushiT0122
RushiT0122
requested changes
Apr 20, 2022
| @@ -244,20 +244,21 @@ def sync(self, state): | |||
| while start < sync_end: | |||
| parsed_start = singer.strftime(start, "%Y-%m-%dT%H:%M:%SZ") | |||
| parsed_end = min(singer.strftime(end, "%Y-%m-%dT%H:%M:%SZ"), parsed_sync_end) | |||
There was a problem hiding this comment.
Good to have, elaborate why we need to take min of 2 timestamps.
There was a problem hiding this comment.
Tap is using window so if 'end' is minimum than window size then consider it otherwise it will use normal 'end' (start + date window)
|
|
||
| class TestUserSyncCheck(unittest.TestCase): | ||
|
|
||
| def test_many_records_in_one_seconds_for_user(self): |
There was a problem hiding this comment.
On local env, this unit test is failing,
$ nosetests test/unittests/test_user_infinite_loop.py:TestUserSyncCheck.test_many_records_in_one_seconds_for_user
.
.
======================================================================
FAIL: Reproduce infinite looping behavior for Users stream when user have many record in single seconds
----------------------------------------------------------------------
Traceback (most recent call last):
File "/opt/code/tap-zendesk/test/unittests/test_user_infinite_loop.py", line 36, in test_many_records_in_one_seconds_for_user
self.assertEqual(str(e.exception), 'users - Unable to get all users within minimum window of a single second (2022-03-30T08:45:21Z), found 1001 users within this timestamp. Zendesk can only provide a maximum of 1000 users per request. See: https://de/
velop.zendesk.com/hc/en-us/articles/360022563994--BREAKING-New-Search-API-Result-Limits')
AssertionError: 'user[71 chars]022-04-14T08:45:20Z), found 1001 users within [176 chars]mits' != 'user[71 chars]022-03-30T08:45:21Z), found 1001 users within [176 chars]mits'
Diff is 801 characters long. Set self.maxDiff to None to see it.
-------------------- >> begin captured stdout << ---------------------
{"type": "STATE", "value": {"bookmarks": {"users": {"updated_at": "2022-04-14T08:45:20Z"}}}}
There was a problem hiding this comment.
We expect static datetime in the exception message, but the unit test is internally using the current datetime. So, we mocked now() with static datetime
kspeer825
approved these changes
May 2, 2022
RushiT0122
approved these changes
Sep 14, 2022
…p.py' and correct that behaviour
RushiT0122
force-pushed
the
TDL-6756-Fix-Infinite-Loop-for-Users
branch
from
bb081e1
to
bb939de
Compare
sgandhi1311
approved these changes
May 23, 2023
leslievandemark
approved these changes
May 23, 2023
Karthipillai
pushed a commit
to MarletteFunding/singer-tap-zendesk
that referenced
this pull request
Oct 2, 2023
* Reproduce infinite loop behaviour in unittest 'test_user_infinite_loop.py' and correct that behaviour * Correct exception mesaage in unittest * change logger message * correct logger message * add comment in streams.py and add 2sec window test case * add comment in streams.py and add 2sec window test case * Change exception message * Correct comment message * resolved unitetst error * update behaviour for 1 second window size * changelog and verison bump --------- Co-authored-by: Rushikesh Todkar <98420315+RushiT0122@users.noreply.github.com> Co-authored-by: kethan1122 <kcherukuri@talend.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of change
Manual QA steps
Risks
Rollback steps