Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GitHub repo discovery performance #4449

Merged
merged 2 commits into from
Jul 30, 2024
Merged

Conversation

chris-rock
Copy link
Member

In the past we did some optimizations to fetch repositories only once. While this makes sense when we do discovery there are cases where fetching all repos from an org or user does not make sense. A specific case is when I just want to scan a single repo.

cnquery shell github repo microsoftgraph/msgraph-sdk-go

That call should not fetch all repos of an org. It makes scanning an org for large organizations slower. This PR fixes it. It changes the code to leverage a cache. If that cache exists (eg. because a user called github.organization.repositories) then its used.

before

DEBUG=1 cnquery shell github repo microsoftgraph/msgraph-sdk-go
DBG performing request method=GET url=https://api.github.com/zen
DBG fetching organization microsoftgraph
DBG performing request method=GET url=https://api.github.com/orgs/microsoftgraph
DBG performing request method=GET url=https://api.github.com/orgs/microsoftgraph/repos?per_page=100&type=all
DBG performing request method=GET url=https://api.github.com/orgs/microsoftgraph/repos?page=2&per_page=100&type=all
DBG performing request method=GET url=https://api.github.com/orgs/microsoftgraph/repos?page=3&per_page=100&type=all

after

DBG fetching organization microsoftgraph
DBG performing request method=GET url=https://api.github.com/orgs/microsoftgraph
DBG performing request method=GET url=https://api.github.com/repos/microsoftgraph/msgraph-sdk-go

@chris-rock chris-rock changed the title Chris rock/GitHub discovery performance GitHub repo discovery performance Jul 30, 2024
@chris-rock chris-rock requested a review from imilchev July 30, 2024 20:33
Copy link
Contributor

@vjeffrey vjeffrey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉

@tas50 tas50 merged commit a6f06f9 into main Jul 30, 2024
15 checks passed
@tas50 tas50 deleted the chris-rock/github-discovery-performance branch July 30, 2024 23:35
@github-actions github-actions bot locked and limited conversation to collaborators Jul 30, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants