Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Azure API throttling error & incorrect page size request for workflows #17

Merged
merged 2 commits into from
Sep 28, 2022

Conversation

0xffhh
Copy link
Contributor

@0xffhh 0xffhh commented Sep 28, 2022

This PR fixes two issues which only seem to occur in (very) large environments.

  1. The current AzureHound version results in a lot of "The request was throttled." error messages from the Azure API. The root cause seems to be the collection command "list-storage-containers" which then cascades into the same throttling for other collection methods. See also here: https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits#storage-resource-provider-limits. By tuning down the concurrency of this storage command, all other throttling issues seem to disappear. This issue is also reported already as issue "The request was throttled" errors when using AzureHound #7 .
  2. The following error occurs when collecting workbooks:
    Error: map[error:map[code:InvalidTopInQueryString message:The value '2147483647' was provided for top; a value less than or equal to '1000' must be provided.]]
    So changed the value to 1000, got a similar error. Changed to 999 to compensate for an off-by-one on Azure's side, but still got the same error
    Error: map[error:map[code:PageSizeLimitExceeded message:The requested page size of '999' exceeds the allowed limit of '100'.]]
    Setting the value to 100 fixes the issue. This is not a permanent fix, as obviously there could be more than 100 workbooks in that subscription. However, at least it collects some data for now, until there is a more permanent fix.

Results for comparison: the rolling version without this patch collects an output file of 4.2GB after this patch the output is 31GB.

@github-actions
Copy link

github-actions bot commented Sep 28, 2022

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

@0xffhh
Copy link
Contributor Author

0xffhh commented Sep 28, 2022

I have read the CLA Document and I hereby sign the CLA

@ddlees ddlees merged commit 15f35e6 into SpecterOps:main Sep 28, 2022
@github-actions github-actions bot locked and limited conversation to collaborators Sep 28, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants