TDL-15860 Implement Request Timeouts #79

prijendev · 2021-10-19T10:34:31Z

Description of change

Implemented request timeout.
Added unittest case for the same.
If config parameter does not pass or pass the value "", "0", 0, "0.0" then set timeout to default value 5min

Note :

Get call in get_incremental_export is replaced by call_api which is covered in PR#74. So, once that PR merged, timeout changes will covered.
Disabled 'consider-using-ternary' pylint error in code.

Manual QA steps

Check that tap retry 10 times on timeout error.

Risks

Rollback steps

revert this branch

…com/singer-io/tap-zendesk into TDL-15860-Implement-Request-Timeout

dbshah1212 · 2021-10-27T08:23:52Z

README.md

@@ -19,7 +19,8 @@ OAuth is the default authentication method for `tap-zendesk`. To use OAuth, you
 {
  "access_token": "AVERYLONGOAUTHTOKEN",
  "subdomain": "acme",
-  "start_date": "2000-01-01T00:00:00Z"
+  "start_date": "2000-01-01T00:00:00Z",
+  "request_timeout": 300


Please add details of it as it is the optional parameter.

karanpanchal-crest · 2021-10-25T14:43:52Z

.circleci/config.yml

      - run:
          name: 'pylint'
          command: |
            source /usr/local/share/virtualenvs/tap-zendesk/bin/activate
-            make test
+            pylint tap_zendesk -d missing-docstring,invalid-name,line-too-long,too-many-locals,too-few-public-methods,fixme,stop-iteration-return,too-many-branches,useless-import-alias,no-else-return,logging-not-lazy


@prijendev Did you go through the errors returned by pylint which you have skipped intentionally?
Is it something that cannot be resolved?

make test command runs both pylint and nosetests. But now as we are adding code coverage with unittests we need to change nosetests with some extra argument. So, it's just spllited make test command. We have not changed existing pylint command. Existing command can be seen under pylint section of circleci

karanpanchal-crest · 2021-11-01T12:00:53Z

tap_zendesk/__init__.py

+    # Set request timeout to config param `request_timeout` value.
+    # If value is 0,"0","" or not passed then it set default to 300 seconds.
+    config_request_timeout = parsed_args.config.get('request_timeout')
+    request_timeout = config_request_timeout and float(config_request_timeout) or REQUEST_TIMEOUT # pylint: disable=consider-using-ternary


@prijendev Please update this condition as follows for better visibility and understanding
request_timeout = (config_request_timeout and float(config_request_timeout)) or REQUEST_TIMEOUT

karanpanchal-crest · 2021-11-01T12:07:58Z

tap_zendesk/http.py

-    response = requests.get(url, params=params, headers=headers)
+@backoff.on_exception(backoff.expo,Timeout, #As timeout error does not have attribute status_code, hence giveup does not work in this case.
+                      max_tries=10) # So, here we added another backoff expression.
+def call_api(url, request_timeout, params, headers):


@prijendev I see all the calls to function call_api is updated due to addition of a single parameter in the function argument.
My suggestion:
Since request_timeout is initialized in the main() function, can't we directly utilize it in the call_api function instead of passing it as an argument? The benefit of this change will be, changes to call_api function can be removed. Minimal changes will be resulted in the PR. Code would be more robust and generic.

CC: @dbshah1212

In this tap some of the streams using Zenpy sdk. So, to initialize sdk client we have read request_timeout in main() function. While for other streams each sync method using call_api function. Updated initialization of request_timeout to common parent Stream class to minimize the changes.

karanpanchal-crest · 2021-11-01T12:10:47Z

tap_zendesk/streams.py

-        for page in http.get_cursor_based(url, self.config['access_token'], **kwargs):
+        # Set and pass request timeout to config param `request_timeout` value.
+        # If value is 0,"0","" or not passed then it set default to 300 seconds.
+        config_request_timeout = self.config.get('request_timeout')


@prijendev This code is already written in the main() function of init file. What is the need of initializing these variables again? Configuration parameters should be read only once in the starting of the tap.

In this tap some of the streams using Zenpy sdk. So, to initialize sdk client we have read request_timeout in main() function. While for other streams updated initialization of request_timeout to common parent Stream class to minimize the changes. So, overall we require to read request_timeout 2 times.

karanpanchal-crest · 2021-11-01T12:11:02Z

tap_zendesk/streams.py

-        for page in http.get_incremental_export(url, self.config['access_token'], start_time):
+        # Set and pass request timeout to config param `request_timeout` value.
+        # If value is 0,"0","" or not passed then it set default to 300 seconds.
+        config_request_timeout = self.config.get('request_timeout')


@prijendev Same comment here as well

In this tap some of the streams using Zenpy sdk. So, to initialize sdk client we have read request_timeout in main() function. While for other streams updated initialization of request_timeout to common parent Stream class to minimize the changes. So, overall we require to read request_timeout 2 times.

karanpanchal-crest · 2021-11-01T12:12:25Z

test/unittests/test_http.py

@@ -13,6 +13,7 @@
             'after_cursor': 'some_cursor'}


@prijendev Did we write test cases for different scenarios of request_timeout variable?

Yes, We write test cases for different scenarios of request_timeout variable in test_request_timeout.py file.

karanpanchal-crest · 2021-11-01T12:13:03Z

tap_zendesk/streams.py

+        # If value is 0,"0","" or not passed then it set default to 300 seconds.
+        config_request_timeout = self.config.get('request_timeout')
+        request_timeout = config_request_timeout and float(config_request_timeout) or REQUEST_TIMEOUT # pylint: disable=consider-using-ternary
+        pages = http.get_offset_based(url, self.config['access_token'], request_timeout)


Did we write test cases for this change?

Yes, we write test cases for this change in test_request_timeout.py. Method name - test_ticket_metrics_timeout_error_with_empty_value

…com/singer-io/tap-zendesk into TDL-15860-Implement-Request-Timeout

karanpanchal-crest · 2021-11-02T04:32:16Z

tap_zendesk/streams.py

+        # Set and pass request timeout to config param `request_timeout` value.
+        # If value is 0,"0","" or not passed then it set default to 300 seconds.
+        config_request_timeout = self.config.get('request_timeout')
+        self.request_timeout = (config_request_timeout and float(config_request_timeout)) or REQUEST_TIMEOUT # pylint: disable=consider-using-ternary


@prijendev We should figure out an alternative to resolve the pylint error. Why to disable it completely?
Can you research on what can be the alternative ways to assign values to self.request_timeout?

Removed disable of pylint error. Updated code.

dbshah1212 · 2021-11-16T09:57:49Z

tap_zendesk/http.py

@@ -22,12 +22,14 @@ def is_fatal(exception):
                      requests.exceptions.HTTPError,
                      max_tries=10,
                      giveup=is_fatal)
-def call_api(url, params, headers):
-    response = requests.get(url, params=params, headers=headers)
+@backoff.on_exception(backoff.expo,Timeout, #As timeout error does not have attribute status_code, hence giveup does not work in this case.


We are adding max_tries=5 with factor=2 so please update accordingly to give significant delay between 2 retries.

Updated it.

dbshah1212 · 2021-11-25T06:57:53Z

tap_zendesk/__init__.py

+    if config_request_timeout and float(config_request_timeout):
+        request_timeout = float(config_request_timeout)
+    else:
+        request_timeout = REQUEST_TIMEOUT # If value is 0,"0","" or not passed then it set default to 300 seconds.


Suggested change

request_timeout = REQUEST_TIMEOUT # If value is 0,"0","" or not passed then it set default to 300 seconds.

request_timeout = REQUEST_TIMEOUT # If value is 0, "0", "" or not passed then it sets default to 300 seconds.

Implemented request timeout

529912c

prijendev changed the title ~~Implemented request timeout~~ TDL-15860 Implement Request Timeouts Oct 19, 2021

prijendev and others added 4 commits October 20, 2021 13:34

Added config param for request timeout

c9dc74a

added coverage report

c85a98d

added support of float datatype for timeout param

a8eb0e4

Updated readme file with request_timeout param.

c9f0f1b

prijendev requested review from dbshah1212 and karanpanchal-crest October 25, 2021 06:07

prijendev added 4 commits October 25, 2021 12:18

Updated timeout logic

f996ed8

Merge branch 'TDL-15860-Implement-Request-Timeout' of https://github.…

72def92

…com/singer-io/tap-zendesk into TDL-15860-Implement-Request-Timeout

Updated request timeout and unittest cases

603928d

Disabled consider-using-ternary pylint error

e1166c6

dbshah1212 reviewed Oct 27, 2021

View reviewed changes

Updated readme file.

a64ce2b

prijendev requested a review from dbshah1212 October 28, 2021 09:19

dbshah1212 approved these changes Nov 1, 2021

View reviewed changes

karanpanchal-crest reviewed Nov 1, 2021

View reviewed changes

prijendev added 2 commits November 1, 2021 18:43

Increased reusability

ac716f9

Merge branch 'TDL-15860-Implement-Request-Timeout' of https://github.…

2620ffd

…com/singer-io/tap-zendesk into TDL-15860-Implement-Request-Timeout

prijendev requested a review from karanpanchal-crest November 2, 2021 02:55

karanpanchal-crest reviewed Nov 2, 2021

View reviewed changes

Resolved disabled pylint error

43f73be

karanpanchal-crest approved these changes Nov 2, 2021

View reviewed changes

umangagrawal-crest requested a review from KrisPersonal November 2, 2021 06:34

dbshah1212 reviewed Nov 16, 2021

View reviewed changes

prijendev added 4 commits November 16, 2021 15:47

Updated max_tries 10 to 5

90b03d3

Merge branch 'master' into TDL-15860-Implement-Request-Timeout

2c50697

removed duplicate REQUEST_TIMEOUT

cceb9fb

removed empty line

1ebd5c1

Merge branch 'master' into TDL-15860-Implement-Request-Timeout

9711835

dbshah1212 reviewed Nov 25, 2021

View reviewed changes

updated message

a067bb8

prijendev requested review from dbshah1212 and karanpanchal-crest November 25, 2021 12:20

KrisPersonal requested a review from kspeer825 December 6, 2021 15:27

KrisPersonal approved these changes Dec 6, 2021

View reviewed changes

kspeer825 approved these changes Dec 6, 2021

View reviewed changes

Merge branch 'master' into TDL-15860-Implement-Request-Timeout

20c8858

KrisPersonal merged commit 2172f2a into master Dec 7, 2021

KrisPersonal deleted the TDL-15860-Implement-Request-Timeout branch December 7, 2021 18:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TDL-15860 Implement Request Timeouts #79

TDL-15860 Implement Request Timeouts #79

prijendev commented Oct 19, 2021 •

edited

Loading

dbshah1212 Oct 27, 2021

prijendev Oct 28, 2021

karanpanchal-crest Oct 25, 2021

prijendev Nov 1, 2021

karanpanchal-crest Nov 1, 2021

prijendev Nov 1, 2021

karanpanchal-crest Nov 1, 2021

prijendev Nov 1, 2021 •

edited

Loading

karanpanchal-crest Nov 1, 2021

prijendev Nov 1, 2021 •

edited

Loading

karanpanchal-crest Nov 1, 2021

prijendev Nov 1, 2021

karanpanchal-crest Nov 1, 2021

prijendev Nov 1, 2021

karanpanchal-crest Nov 1, 2021

prijendev Nov 1, 2021

karanpanchal-crest Nov 2, 2021

prijendev Nov 2, 2021

dbshah1212 Nov 16, 2021

prijendev Nov 16, 2021

dbshah1212 Nov 25, 2021

prijendev Nov 25, 2021

	request_timeout = REQUEST_TIMEOUT # If value is 0,"0","" or not passed then it set default to 300 seconds.
	request_timeout = REQUEST_TIMEOUT # If value is 0, "0", "" or not passed then it sets default to 300 seconds.

TDL-15860 Implement Request Timeouts #79

TDL-15860 Implement Request Timeouts #79

Conversation

prijendev commented Oct 19, 2021 • edited Loading

Description of change

Manual QA steps

Risks

Rollback steps

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

prijendev Nov 1, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

prijendev Nov 1, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

prijendev commented Oct 19, 2021 •

edited

Loading

prijendev Nov 1, 2021 •

edited

Loading

prijendev Nov 1, 2021 •

edited

Loading