Implement long running polling #1649
Conversation
p3ck
force-pushed
the
inventory_polling
branch
2 times, most recently
from
1f1dc43 to
93e32ea
Compare
|
Possible fix for #1614 |
Fred-sun
added
medium_priority
This implements long running polling the same way the GenericRestClient does with a 600 second overall timeout and 30 second polling. Implement retries and backoffs in the batch processing. I looked at the Azure SDK's pipeline which supports policies like RetryPolicy but I don't think it will help here. It doesn't look like we get a 429 when sending the request to the /batch endpoint, only in the batch responses.
p3ck
added
the
ready_for_review
| # 429: Too many requests Error, Backoff and Retry | ||
| batch_retry = True | ||
| if batch_retry: | ||
| time.sleep(backoff_factor * (2 ** (retry_count))) |
There was a problem hiding this comment.
Regarding the backoff policy, retry time should based on the response. Other wise the retry request will be rejected too.
https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/troubleshooting-throttling-errors#throttling-error-details
There was a problem hiding this comment.
The issue here is that in the current implementation the wait sleep is done once for the all batch together
Do you have any suggestion/thoughts how to change it to work for each vm separately?
There was a problem hiding this comment.
@nirarg I see, misunderstand the retry policy here. The backoff policy and retry is focus on the local batch requests here so we cannot determine the retry time for all the request.
This implements long running polling the same way the GenericRestClient does with a 600 second overall timeout and 30 second polling. Implement retries and backoffs in the batch processing. I looked at the Azure SDK's pipeline which supports policies like RetryPolicy but I don't think it will help here. It doesn't look like we get a 429 when sending the request to the /batch endpoint, only in the batch responses.
SUMMARY
This implements long running polling the same way the GenericRestClient does with a 600 second overall timeout and 30 second polling.
ISSUE TYPE
COMPONENT NAME
plugins/inventory/azure_rm.py
ADDITIONAL INFORMATION
I ran the inventory integration tests but I don't have enough hosts in my azure instance to recreate the original issue. So all I can do is verify that it still works with my limited test environment.