Skip to content
This repository has been archived by the owner on Sep 3, 2022. It is now read-only.

RuntimeError: maximum recursion depth exceeded #1251

Closed
HaipengSu opened this issue Mar 7, 2017 · 8 comments · Fixed by googledatalab/pydatalab#296
Closed

RuntimeError: maximum recursion depth exceeded #1251

HaipengSu opened this issue Mar 7, 2017 · 8 comments · Fixed by googledatalab/pydatalab#296
Assignees
@ojarjur
Labels
bug

Comments

@HaipengSu
Copy link

Hi,

After updating the Datalab to the newest version, I got this Runtime Error.

RuntimeError: maximum recursion depth exceeded

Apparently, this is related to the depth of recursive calls.

For the same notebook, I didn't get the Runtime Error before, but only for the new Datalab. Is there any change related to this issue?

Thank you.

Or should I just use
import sys sys.setrecursionlimit(10000)

to increase the recursion limit?
@nikhilk
Copy link
Contributor

nikhilk commented Mar 7, 2017

Could you provide more information when this happens?

@HaipengSu
Copy link
Author

Hi @nikhilk

From Datalab,
I have
notebooks = [ "notebook1.ipynb",
"notebook2.ipynb" ,
.
.
.
"notebook20.ipynb" ]
for notebook in notebooks:
%run $notebook

Never had any issues running this, but since updating to the newest version, I got the Runtime Error. Not still not sure about whether it is related.

Thx

@HaipengSu
Copy link
Author

Hi @nikhilk,

I am trying to get the error reproducible, so that it wouldn't confuse you and others.

Will let you know once I get more information.

thx @parthea for pointing out.

@parthea
Copy link
Contributor

parthea commented Mar 7, 2017 •
edited
Loading

@HaipengSu shared a link to a notebook session which is currently in a failed state. The issue only occurs in this specific notebook session. If I open a new notebook and run the same code the issue disappears.

from datalab.bigquery import _api
import datalab
table = bq.Table('geotab-embedded:Test_Haipeng.sgs')
api = _api.Api(datalab.context.Context.default())
api.tables_get(table._name_parts)

I'm still looking into this but at first glance it appears that the recursion is happening in oauth2client.


RuntimeErrorTraceback (most recent call last)
<ipython-input-47-5908590773f2> in <module>()
      2 table = bq.Table('geotab-embedded:Test_Haipeng.sgs')
      3 api = _api.Api(datalab.context.Context.default())
----> 4 api.tables_get(table._name_parts)

/usr/local/lib/python2.7/dist-packages/datalab/bigquery/_api.pyc in tables_get(self, table_name)
    357     """
    358     url = Api._ENDPOINT + (Api._TABLES_PATH % table_name)
--> 359     return datalab.utils.Http.request(url, credentials=self._credentials)
    360 
    361   def tables_list(self, dataset_name, max_results=0, page_token=None):

/usr/local/lib/python2.7/dist-packages/datalab/utils/_http.pyc in request(url, args, data, headers, method, credentials, raw_response, stats)
    132                                        method=method,
    133                                        body=data,
--> 134                                        headers=headers)
    135       if 200 <= response.status < 300:
    136         if raw_response:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614                 logger.info('Attempting refresh to obtain '
    615                             'initial access_token')
--> 616                 self._refresh(request_orig)
    617 
    618             # Clone and modify the request headers to add the appropriate

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         """
    128         response, content = http_request(
--> 129             META, headers={'Metadata-Flavor': 'Google'})
    130         content = _from_bytes(content)
    131         if response.status == http_client.OK:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614                 logger.info('Attempting refresh to obtain '
    615                             'initial access_token')
--> 616                 self._refresh(request_orig)
    617 
    618             # Clone and modify the request headers to add the appropriate

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         """
    128         response, content = http_request(
--> 129             META, headers={'Metadata-Flavor': 'Google'})
    130         content = _from_bytes(content)
    131         if response.status == http_client.OK:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614                 logger.info('Attempting refresh to obtain '
    615                             'initial access_token')
--> 616                 self._refresh(request_orig)
    617 
    618             # Clone and modify the request headers to add the appropriate

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         """
    128         response, content = http_request(
--> 129             META, headers={'Metadata-Flavor': 'Google'})
    130         content = _from_bytes(content)
    131         if response.status == http_client.OK:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614                 logger.info('Attempting refresh to obtain '
    615                             'initial access_token')
--> 616                 self._refresh(request_orig)
    617 
    618             # Clone and modify the request headers to add the appropriate

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         """
    128         response, content = http_request(
--> 129             META, headers={'Metadata-Flavor': 'Google'})
    130         content = _from_bytes(content)
    131         if response.status == http_client.OK:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614                 logger.info('Attempting refresh to obtain '
    615                             'initial access_token')
--> 616                 self._refresh(request_orig)
    617 
    618             # Clone and modify the request headers to add the appropriate

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         """
    128         response, content = http_request(
--> 129             META, headers={'Metadata-Flavor': 'Google'})
    130         content = _from_bytes(content)
    131         if response.status == http_client.OK:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614                 logger.info('Attempting refresh to obtain '
    615                             'initial access_token')
--> 616                 self._refresh(request_orig)
    617 
    618             # Clone and modify the request headers to add the appropriate

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         """
    128         response, content = http_request(
--> 129             META, headers={'Metadata-Flavor': 'Google'})
    130         content = _from_bytes(content)
    131         if response.status == http_client.OK:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614                 logger.info('Attempting refresh to obtain '
    615                             'initial access_token')
--> 616                 self._refresh(request_orig)
    617 
    618             # Clone and modify the request headers to add the appropriate

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         """
    128         response, content = http_request(
--> 129             META, headers={'Metadata-Flavor': 'Google'})
    130         content = _from_bytes(content)
    131         if response.status == http_client.OK:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614                 logger.info('Attempting refresh to obtain '
    615                             'initial access_token')
--> 616                 self._refresh(request_orig)
    617 
    618             # Clone and modify the request headers to add the appropriate

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         """
    128         response, content = http_request(
--> 129             META, headers={'Metadata-Flavor': 'Google'})
    130         content = _from_bytes(content)
    131         if response.status == http_client.OK:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614                 logger.info('Attempting refresh to obtain '
    615                             'initial access_token')
--> 616                 self._refresh(request_orig)
    617 
    618             # Clone and modify the request headers to add the appropriate

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         """
    128         response, content = http_request(
--> 129             META, headers={'Metadata-Flavor': 'Google'})
    130         content = _from_bytes(content)
    131         if response.status == http_client.OK:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614                 logger.info('Attempting refresh to obtain '
    615                             'initial access_token')
--> 616                 self._refresh(request_orig)
    617 
    618             # Clone and modify the request headers to add the appropriate

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         """
    128         response, content = http_request(
--> 129             META, headers={'Metadata-Flavor': 'Google'})
    130         content = _from_bytes(content)
    131         if response.status == http_client.OK:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614                 logger.info('Attempting refresh to obtain '
    615                             'initial access_token')
--> 616                 self._refresh(request_orig)
    617 
    618             # Clone and modify the request headers to add the appropriate

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         """
    128         response, content = http_request(
--> 129             META, headers={'Metadata-Flavor': 'Google'})
    130         content = _from_bytes(content)
    131         if response.status == http_client.OK:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    629             resp, content = request_orig(uri, method, body,
    630                                          clean_headers(headers),
--> 631                                          redirections, connection_type)
    632 
    633             # A stored token may expire between the time it is retrieved and

... last 1 frames repeated, from the frame below ...

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    629             resp, content = request_orig(uri, method, body,
    630                                          clean_headers(headers),
--> 631                                          redirections, connection_type)
    632 
    633             # A stored token may expire between the time it is retrieved and

RuntimeError: maximum recursion depth exceeded while calling a Python object

Here are the results after upgrading to oauth2client==4.0.0

Collecting oauth2client==4.0.0
  Downloading oauth2client-4.0.0-py2.py3-none-any.whl (184kB)
    100% |################################| 194kB 3.2MB/s 
Requirement already satisfied: httplib2>=0.9.1 in /usr/local/lib/python2.7/dist-packages (from oauth2client==4.0.0)
Requirement already satisfied: rsa>=3.1.4 in /usr/local/lib/python2.7/dist-packages (from oauth2client==4.0.0)
Requirement already satisfied: pyasn1>=0.1.7 in /usr/local/lib/python2.7/dist-packages (from oauth2client==4.0.0)
Requirement already satisfied: pyasn1-modules>=0.0.5 in /usr/local/lib/python2.7/dist-packages (from oauth2client==4.0.0)
Requirement already satisfied: six>=1.6.1 in /usr/local/lib/python2.7/dist-packages (from oauth2client==4.0.0)
Installing collected packages: oauth2client
  Found existing installation: oauth2client 2.2.0
    Uninstalling oauth2client-2.2.0:
      Successfully uninstalled oauth2client-2.2.0
Successfully installed oauth2client-4.0.0

RuntimeErrorTraceback (most recent call last)
<ipython-input-50-5908590773f2> in <module>()
      2 table = bq.Table('geotab-embedded:Test_Haipeng.sgs')
      3 api = _api.Api(datalab.context.Context.default())
----> 4 api.tables_get(table._name_parts)

/usr/local/lib/python2.7/dist-packages/datalab/bigquery/_api.pyc in tables_get(self, table_name)
    357     """
    358     url = Api._ENDPOINT + (Api._TABLES_PATH % table_name)
--> 359     return datalab.utils.Http.request(url, credentials=self._credentials)
    360 
    361   def tables_list(self, dataset_name, max_results=0, page_token=None):

/usr/local/lib/python2.7/dist-packages/datalab/utils/_http.pyc in request(url, args, data, headers, method, credentials, raw_response, stats)
    132                                        method=method,
    133                                        body=data,
--> 134                                        headers=headers)
    135       if 200 <= response.status < 300:
    136         if raw_response:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614         retval = cls(
    615             data['access_token'],
--> 616             data['client_id'],
    617             data['client_secret'],
    618             data['refresh_token'],

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         try:
    128             self._retrieve_info(http)
--> 129             self.access_token, self.token_expiry = _metadata.get_token(
    130                 http, service_account=self.service_account_email)
    131         except http_client.HTTPException as err:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614         retval = cls(
    615             data['access_token'],
--> 616             data['client_id'],
    617             data['client_secret'],
    618             data['refresh_token'],

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         try:
    128             self._retrieve_info(http)
--> 129             self.access_token, self.token_expiry = _metadata.get_token(
    130                 http, service_account=self.service_account_email)
    131         except http_client.HTTPException as err:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614         retval = cls(
    615             data['access_token'],
--> 616             data['client_id'],
    617             data['client_secret'],
    618             data['refresh_token'],

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         try:
    128             self._retrieve_info(http)
--> 129             self.access_token, self.token_expiry = _metadata.get_token(
    130                 http, service_account=self.service_account_email)
    131         except http_client.HTTPException as err:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614         retval = cls(
    615             data['access_token'],
--> 616             data['client_id'],
    617             data['client_secret'],
    618             data['refresh_token'],

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         try:
    128             self._retrieve_info(http)
--> 129             self.access_token, self.token_expiry = _metadata.get_token(
    130                 http, service_account=self.service_account_email)
    131         except http_client.HTTPException as err:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614         retval = cls(
    615             data['access_token'],
--> 616             data['client_id'],
    617             data['client_secret'],
    618             data['refresh_token'],

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         try:
    128             self._retrieve_info(http)
--> 129             self.access_token, self.token_expiry = _metadata.get_token(
    130                 http, service_account=self.service_account_email)
    131         except http_client.HTTPException as err:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614         retval = cls(
    615             data['access_token'],
--> 616             data['client_id'],
    617             data['client_secret'],
    618             data['refresh_token'],

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         try:
    128             self._retrieve_info(http)
--> 129             self.access_token, self.token_expiry = _metadata.get_token(
    130                 http, service_account=self.service_account_email)
    131         except http_client.HTTPException as err:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614         retval = cls(
    615             data['access_token'],
--> 616             data['client_id'],
    617             data['client_secret'],
    618             data['refresh_token'],

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         try:
    128             self._retrieve_info(http)
--> 129             self.access_token, self.token_expiry = _metadata.get_token(
    130                 http, service_account=self.service_account_email)
    131         except http_client.HTTPException as err:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614         retval = cls(
    615             data['access_token'],
--> 616             data['client_id'],
    617             data['client_secret'],
    618             data['refresh_token'],

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         try:
    128             self._retrieve_info(http)
--> 129             self.access_token, self.token_expiry = _metadata.get_token(
    130                 http, service_account=self.service_account_email)
    131         except http_client.HTTPException as err:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614         retval = cls(
    615             data['access_token'],
--> 616             data['client_id'],
    617             data['client_secret'],
    618             data['refresh_token'],

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         try:
    128             self._retrieve_info(http)
--> 129             self.access_token, self.token_expiry = _metadata.get_token(
    130                 http, service_account=self.service_account_email)
    131         except http_client.HTTPException as err:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614         retval = cls(
    615             data['access_token'],
--> 616             data['client_id'],
    617             data['client_secret'],
    618             data['refresh_token'],

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         try:
    128             self._retrieve_info(http)
--> 129             self.access_token, self.token_expiry = _metadata.get_token(
    130                 http, service_account=self.service_account_email)
    131         except http_client.HTTPException as err:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614         retval = cls(
    615             data['access_token'],
--> 616             data['client_id'],
    617             data['client_secret'],
    618             data['refresh_token'],

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         try:
    128             self._retrieve_info(http)
--> 129             self.access_token, self.token_expiry = _metadata.get_token(
    130                 http, service_account=self.service_account_email)
    131         except http_client.HTTPException as err:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614         retval = cls(
    615             data['access_token'],
--> 616             data['client_id'],
    617             data['client_secret'],
    618             data['refresh_token'],

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         try:
    128             self._retrieve_info(http)
--> 129             self.access_token, self.token_expiry = _metadata.get_token(
    130                 http, service_account=self.service_account_email)
    131         except http_client.HTTPException as err:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    629 
    630     @property
--> 631     def access_token_expired(self):
    632         """True if the credential is expired or invalid.
    633 

... last 1 frames repeated, from the frame below ...

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    629 
    630     @property
--> 631     def access_token_expired(self):
    632         """True if the credential is expired or invalid.
    633 

RuntimeError: maximum recursion depth exceeded while calling a Python object

@HaipengSu
Copy link
Author

Thx @parthea for keeping track on this issue.

@parthea
Copy link
Contributor

parthea commented Mar 7, 2017

Here is some code that @HaipengSu and I used to create the issue locally:

import datalab.storage as gs
bucketName = "uspto-pair"
bucket = gs.Bucket(bucketName)
for i in range(0,1000):
  for item in bucket.items():
    pass

results in

RuntimeErrorTraceback (most recent call last)
<ipython-input-9-5daf0d8d76e0> in <module>()
      3 bucket = gs.Bucket(bucketName)
      4 for i in range(0,1000):
----> 5   for item in bucket.items():
      6     pass

/usr/local/lib/python2.7/dist-packages/datalab/utils/_iterator.pyc in __iter__(self)
     34     """Provides iterator functionality."""
     35     while self._first_page or (self._page_token is not None):
---> 36       items, next_page_token = self._retriever(self._page_token, self._count)
     37 
     38       self._page_token = next_page_token

/usr/local/lib/python2.7/dist-packages/datalab/storage/_item.pyc in _retrieve_items(self, page_token, _)
    276                                          page_token=page_token)
    277     except Exception as e:
--> 278       raise e
    279 
    280     items = list_info.get('items', [])

RuntimeError: maximum recursion depth exceeded while calling a Python object

The issue appears to be resolved if we revert the changes made in googledatalab/pydatalab#195 .

cc @nikhilk
cc @yebrahim

@ojarjur
Copy link
Contributor

ojarjur commented Mar 8, 2017

It looks like the source of the bug is that every time an HTTP request is issued, the call to credentials.authorize adds another wrapper around the http instance's request method.

Then, when http.request(...) gets invoked, all of those layers of wrappers have to be passed through before the final request actually gets performed.

In essence, we have a leak where the level of method wrapping grows unbounded, and eventually hits the recursive depth limit.

We can avoid this if the number of request wrappers was fixed (e.g. at 1). That, in turn, might be doable by making sure that the request method in the shared http instance cannot be overwritten.

I'll try making a copy of the shared instance before adding the credentials, and see if that prevents the bug.

@ojarjur ojarjur self-assigned this Mar 8, 2017
@ojarjur ojarjur mentioned this issue Mar 8, 2017
Merged
@HaipengSu
Copy link
Author

Thank you all for looking into this.

@chmeyers chmeyers added the bug label Mar 9, 2017
rajivpb pushed a commit to rajivpb/datalab that referenced this issue Jun 16, 2017
@ojarjur
Fix a bug where an unbounded number of credentials wrappers got added. (
84ccb95 
googledatalab#296)

* Fix a bug where an unbounded number of credentials wrappers got added.

This fixes googledatalab#1251, where the fact that we were sharing a single
`httplib2.Http` instance caused each request to add an additional
credentials wrapper around that single instance.

That, in turn, caused a `RuntimeError: maximum recursion depth ...`
error message once too many API calls had been made.

* Only copy the http instance when we need to

* Include the fix for authorizing HTTP requests in the older `datalab` module
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@ojarjur ojarjur

Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix a bug where an unbounded number of credentials wrappers got added. googledatalab/pydatalab
5 participants
@nikhilk @ojarjur @parthea @chmeyers @HaipengSu