fix(ingestion/bigquery): user exceeded quota for concurrent project.lists requests #10578

shubhamjagtap639 · 2024-05-23T10:47:44Z

Checklist

The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
Links to related issues (if applicable)
Tests for the changes have been added/updated (if applicable)
Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

hsheth2 · 2024-05-23T17:58:42Z

metadata-ingestion/src/datahub/ingestion/source/bigquery_v2/bigquery_schema.py

+                page_token: Optional[str] = None
+                while True:
+                    projects_iterator = self.bq_client.list_projects(
+                        max_results=5, page_token=page_token


why is max_results=5?

Sorry, this should be 50 as API returns max 50 projects per page. Refer link

hsheth2 · 2024-05-23T17:58:52Z

metadata-ingestion/src/datahub/ingestion/source/bigquery_v2/bigquery_schema.py

-                    BigqueryProject(id=p.project_id, name=p.friendly_name)
-                    for p in projects
-                ]
+                projects: List[BigqueryProject] = []


add a comment explaining why this was required

Yes, Let me add the comment.
The thing is we are not able to regenerate the error but from doc we got to know two project.lists request is allowed per second and assumed that bigquery client method list_projects is not adding any limit in calling request.
Hence we are trying to add limit in requests per second externally,

Yup I understand that, but having it as a comment in the code will help future editors of this file

…eded

hsheth2 · 2024-05-24T18:41:01Z

metadata-ingestion/src/datahub/ingestion/source/bigquery_v2/bigquery_schema.py

@@ -165,6 +170,9 @@ def get_projects(self) -> List[BigqueryProject]:
                    page_token = projects_iterator.next_page_token
                    if page_token is None:
                        break
+                    # Sleep of 30 sec to make limit of two requests per second
+                    time.sleep(30)


we have a RateLimiter helper class - please use that instead

hsheth2 · 2024-05-28T17:16:46Z

metadata-ingestion/src/datahub/ingestion/source/bigquery_v2/bigquery_schema.py

@@ -157,21 +157,19 @@ def get_projects(self) -> List[BigqueryProject]:
                # Also project.list returns max 50 projects per page.
                # Assuming list_projects method internally not adding any limit in requests call,
                # externally we are adding limit in requests call per second.
+                rate_limiter = RateLimiter(max_calls=2, period=1)


Suggested change

rate_limiter = RateLimiter(max_calls=2, period=1)

rate_limiter = RateLimiter(max_calls=2, period=60)

Quata for projects.list is 2 requests per second not per minute.
I know earlier I added the sleep of 30 sec but it was incorrect.

hsheth2 · 2024-05-28T17:17:07Z

metadata-ingestion/src/datahub/ingestion/source/bigquery_v2/bigquery_schema.py

-                    )
+                    with rate_limiter:
+                        projects_iterator = self.bq_client.list_projects(
+                            max_results=30, page_token=page_token


what is the max number of results in a single page? can you add a link to the docs here?

Its 50 projects per page
https://cloud.google.com/bigquery/docs/reference/rest/v2/projects/list#query-parameters

hsheth2 · 2024-06-11T04:29:13Z

metadata-ingestion/src/datahub/ingestion/source/bigquery_v2/bigquery_schema.py

@@ -149,6 +150,10 @@ def get_query_result(self, query: str) -> RowIterator:
        return resp.result()

    def get_projects(self) -> List[BigqueryProject]:
+        def _should_retry(exc: BaseException) -> bool:
+            logger.debug(f"Exception reason: {str(exc)}. Type: {type(exc)}")


add a comment here explaining what we previously tried and how we arrived at this solution

hsheth2 · 2024-06-11T04:31:48Z

metadata-ingestion/src/datahub/ingestion/source/bigquery_v2/bigquery_schema.py

+                projects: List[BigqueryProject] = []
+                page_token: Optional[str] = None
+                # Bigquery API has limit in calling project.list request i.e. 2 request per second.
+                # Also project.list returns max 50 projects per page.


add a link to the bigquery docs

hsheth2 · 2024-06-11T04:32:03Z

metadata-ingestion/src/datahub/ingestion/source/bigquery_v2/bigquery_schema.py

+                # Also project.list returns max 50 projects per page.
+                # Assuming list_projects method internally not adding any limit in requests call,
+                # externally we are adding limit in requests call per second.
+                rate_limiter = RateLimiter(max_calls=1, period=2)


this is 1 request per 2 seconds, not 2 requests per second?

hsheth2 · 2024-06-11T04:32:27Z

metadata-ingestion/src/datahub/ingestion/source/bigquery_v2/bigquery_schema.py

+                            page_token=page_token,
+                            retry=retry.Retry(predicate=_should_retry, timeout=20),
+                        )
+                        count = 0


make this more descriptive

hsheth2 · 2024-06-11T04:32:34Z

metadata-ingestion/src/datahub/ingestion/source/bigquery_v2/bigquery_schema.py

+                                BigqueryProject(id=p.project_id, name=p.friendly_name)
+                            )
+                            count += 1
+                        logger.debug(f"Projects count per request {str(count)}")


no need for str

…eded

…ists requests (datahub-project#10578) Co-authored-by: Harshal Sheth <hsheth2@gmail.com>

…ists requests (#10578) Co-authored-by: Harshal Sheth <hsheth2@gmail.com>

add Bigquery list projects rate limit

6385a63

github-actions bot added the ingestion PR or Issue related to the ingestion of metadata label May 23, 2024

vercel bot deployed to Preview May 23, 2024 11:00 View deployment

hsheth2 reviewed May 23, 2024

View reviewed changes

shubhamjagtap639 added 2 commits May 24, 2024 13:15

Address review comments

fc9a8cc

Modify comment

58194de

vercel bot deployed to Preview May 24, 2024 08:14 View deployment

Merge branch 'master' into bigquery-project.lists-requests-quota-exce…

1bb7819

…eded

hsheth2 reviewed May 24, 2024

View reviewed changes

vercel bot deployed to Preview May 24, 2024 18:55 View deployment

shubhamjagtap639 added 2 commits May 27, 2024 19:16

Add rate limiter for bq client list projects request

bcedaf5

Revert test case changes

18e3b90

vercel bot deployed to Preview May 27, 2024 14:15 View deployment

Modify ratelimiter code and reduce the max result of list project api

8061a37

vercel bot deployed to Preview May 28, 2024 10:44 View deployment

hsheth2 reviewed May 28, 2024

View reviewed changes

Modify the max result and page size in bq client list project

206a123

vercel bot deployed to Preview May 30, 2024 05:30 View deployment

temporary debug changes

9fa2318

vercel bot deployed to Preview June 5, 2024 12:58 View deployment

Temporary debug changes

02b69a0

vercel bot deployed to Preview June 5, 2024 14:04 View deployment

Add retry to bq client list project api

6fc9842

vercel bot deployed to Preview June 5, 2024 18:51 View deployment

hsheth2 reviewed Jun 11, 2024

View reviewed changes

Modify bq get project method

f4f7388

vercel bot deployed to Preview June 11, 2024 10:20 View deployment

Modify bq get project method

31eece4

vercel bot deployed to Preview June 11, 2024 11:02 View deployment

Add debug log in bg list project request

29a22a6

vercel bot deployed to Preview June 11, 2024 12:22 View deployment

Add back retry to bq client list project request

28b4a6e

vercel bot deployed to Preview June 11, 2024 13:55 View deployment

Revert the get_projects method and add retry in list project request

be8a63c

vercel bot deployed to Preview June 12, 2024 05:56 View deployment

shubhamjagtap639 added 2 commits June 12, 2024 15:08

Modify get_project method and list project count in report

a8bb758

Modify comment in get_project method

ae42a79

vercel bot deployed to Preview June 12, 2024 10:07 View deployment

hsheth2 approved these changes Jun 12, 2024

View reviewed changes

Merge branch 'master' into bigquery-project.lists-requests-quota-exce…

a685b27

…eded

hsheth2 added the merge-pending-ci A PR that has passed review and should be merged once CI is green. label Jun 14, 2024

vercel bot deployed to Preview June 14, 2024 19:11 View deployment

hsheth2 merged commit e6246c9 into datahub-project:master Jun 15, 2024
57 of 82 checks passed

sleeperdeep pushed a commit to sleeperdeep/datahub that referenced this pull request Jun 25, 2024

fix(ingestion/bigquery): user exceeded quota for concurrent project.l…

3610da0

…ists requests (datahub-project#10578) Co-authored-by: Harshal Sheth <hsheth2@gmail.com>

yoonhyejin pushed a commit that referenced this pull request Jul 16, 2024

fix(ingestion/bigquery): user exceeded quota for concurrent project.l…

7e5cfdc

…ists requests (#10578) Co-authored-by: Harshal Sheth <hsheth2@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(ingestion/bigquery): user exceeded quota for concurrent project.lists requests #10578

fix(ingestion/bigquery): user exceeded quota for concurrent project.lists requests #10578

shubhamjagtap639 commented May 23, 2024

hsheth2 May 23, 2024

shubhamjagtap639 May 24, 2024

hsheth2 May 23, 2024

shubhamjagtap639 May 24, 2024

hsheth2 May 24, 2024

hsheth2 May 24, 2024

shubhamjagtap639 May 27, 2024

hsheth2 May 28, 2024

shubhamjagtap639 May 29, 2024

hsheth2 May 28, 2024

shubhamjagtap639 May 29, 2024

hsheth2 Jun 11, 2024

hsheth2 Jun 11, 2024

hsheth2 Jun 11, 2024

hsheth2 Jun 11, 2024

hsheth2 Jun 11, 2024

	rate_limiter = RateLimiter(max_calls=2, period=1)
	rate_limiter = RateLimiter(max_calls=2, period=60)

fix(ingestion/bigquery): user exceeded quota for concurrent project.lists requests #10578

fix(ingestion/bigquery): user exceeded quota for concurrent project.lists requests #10578

Conversation

shubhamjagtap639 commented May 23, 2024

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment