Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use key manager in github data access #2838

Merged
merged 2 commits into from
Jun 29, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions augur/tasks/github/util/github_data_access.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
import logging

Check warning on line 1 in augur/tasks/github/util/github_data_access.py

View workflow job for this annotation

GitHub Actions / runner / pylint

[pylint] reported by reviewdog 🐶 C0114: Missing module docstring (missing-module-docstring) Raw Output: augur/tasks/github/util/github_data_access.py:1:0: C0114: Missing module docstring (missing-module-docstring)
import time
import httpx
from tenacity import retry, stop_after_attempt, wait_fixed, retry_if_exception, RetryError
from urllib.parse import urlparse, parse_qs, urlencode


class RatelimitException(Exception):

Check warning on line 8 in augur/tasks/github/util/github_data_access.py

View workflow job for this annotation

GitHub Actions / runner / pylint

[pylint] reported by reviewdog 🐶 C0115: Missing class docstring (missing-class-docstring) Raw Output: augur/tasks/github/util/github_data_access.py:8:0: C0115: Missing class docstring (missing-class-docstring)

def __init__(self, response, message="Github Rate limit exceeded") -> None:

Expand All @@ -13,10 +13,10 @@

super().__init__(message)

class UrlNotFoundException(Exception):

Check warning on line 16 in augur/tasks/github/util/github_data_access.py

View workflow job for this annotation

GitHub Actions / runner / pylint

[pylint] reported by reviewdog 🐶 C0115: Missing class docstring (missing-class-docstring) Raw Output: augur/tasks/github/util/github_data_access.py:16:0: C0115: Missing class docstring (missing-class-docstring)
pass

class GithubDataAccess:

Check warning on line 19 in augur/tasks/github/util/github_data_access.py

View workflow job for this annotation

GitHub Actions / runner / pylint

[pylint] reported by reviewdog 🐶 C0115: Missing class docstring (missing-class-docstring) Raw Output: augur/tasks/github/util/github_data_access.py:19:0: C0115: Missing class docstring (missing-class-docstring)

def __init__(self, key_manager, logger: logging.Logger):

Expand All @@ -39,7 +39,7 @@

return (100 * (num_pages -1)) + len(data)

def paginate_resource(self, url):

Check warning on line 42 in augur/tasks/github/util/github_data_access.py

View workflow job for this annotation

GitHub Actions / runner / pylint

[pylint] reported by reviewdog 🐶 R1711: Useless return at end of function or method (useless-return) Raw Output: augur/tasks/github/util/github_data_access.py:42:4: R1711: Useless return at end of function or method (useless-return)

response = self.make_request_with_retries(url)
data = response.json()
Expand Down Expand Up @@ -85,7 +85,7 @@

return int(parse_qs(parsed_url.query)['page'][0])
except (KeyError, ValueError):
raise Exception(f"Unable to parse 'last' url from response: {response.links['last']}")

Check warning on line 88 in augur/tasks/github/util/github_data_access.py

View workflow job for this annotation

GitHub Actions / runner / pylint

[pylint] reported by reviewdog 🐶 W0707: Consider explicitly re-raising using 'except (KeyError, ValueError) as exc' and 'raise Exception(f"Unable to parse 'last' url from response: {response.links['last']}") from exc' (raise-missing-from) Raw Output: augur/tasks/github/util/github_data_access.py:88:12: W0707: Consider explicitly re-raising using 'except (KeyError, ValueError) as exc' and 'raise Exception(f"Unable to parse 'last' url from response: {response.links['last']}") from exc' (raise-missing-from)

def get_resource(self, url):

Expand All @@ -93,17 +93,17 @@

return response.json()

# TODO: Handle timeout exceptions better

Check warning on line 96 in augur/tasks/github/util/github_data_access.py

View workflow job for this annotation

GitHub Actions / runner / pylint

[pylint] reported by reviewdog 🐶 W0511: TODO: Handle timeout exceptions better (fixme) Raw Output: augur/tasks/github/util/github_data_access.py:96:5: W0511: TODO: Handle timeout exceptions better (fixme)
def make_request(self, url, method="GET", timeout=100):

with httpx.Client() as client:

response = client.request(method=method, url=url, timeout=timeout, follow_redirects=True)
response = client.request(method=method, url=url, auth=self.key_manager, timeout=timeout, follow_redirects=True)

if response.status_code in [403, 429]:
ABrain7710 marked this conversation as resolved.
Show resolved Hide resolved
raise RatelimitException(response)

elif response.status_code == 404:
if response.status_code == 404:
raise UrlNotFoundException(f"Could not find {url}")

response.raise_for_status()
Expand Down
Loading