Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lending.py: Add logging to exceptions #3350

Merged
merged 9 commits into from
Apr 17, 2020
Merged

lending.py: Add logging to exceptions #3350

merged 9 commits into from
Apr 17, 2020

Conversation

cclauss
Copy link
Collaborator

@cclauss cclauss commented Apr 14, 2020

Fix. openlibrary/core/lending.py contains many bare exceptions which make it difficult to spot the problems that are causing pytest openlibrary/plugins/openlibrary/tests/test_home.py to fail on Python 3. This PR logs the exceptions in lending.py using
https://docs.python.org/3/library/logging.html#logging.Logger.exception

Technical

Testing

Yes.

Evidence

Stakeholders

@cclauss cclauss added Theme: Upgrade to Python 3 Needs: Review This issue/PR needs to be reviewed in order to be closed or merged (see comments). [managed] labels Apr 14, 2020
@cclauss cclauss added this to the Active Sprint milestone Apr 14, 2020
@cclauss cclauss mentioned this pull request Apr 14, 2020
77 tasks
@cclauss cclauss removed the Needs: Review This issue/PR needs to be reviewed in order to be closed or merged (see comments). [managed] label Apr 14, 2020
Copy link
Collaborator

@cdrini cdrini left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some questions about some of the new non-exception logs; otherwise lgtm.

openlibrary/plugins/openlibrary/home.py Outdated Show resolved Hide resolved
openlibrary/mocks/mock_infobase.py Outdated Show resolved Hide resolved
@cdrini cdrini added the Needs: Submitter Input Waiting on input from the creator of the issue/pr [managed] label Apr 16, 2020
@cdrini
Copy link
Collaborator

cdrini commented Apr 17, 2020

Could you resolve the conflicts on this branch? I tried to but I can't quite figure out what it's supposed to be; it looks like there were some big changes in the test-py3 file. Otherwise code lgtm; I just need to toss on dev to test borrowing 👍

@cdrini cdrini added Needs: Testing and removed Needs: Submitter Input Waiting on input from the creator of the issue/pr [managed] labels Apr 17, 2020
Copy link
Collaborator

@cdrini cdrini left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested borrowing works from homepage; lgtm!

Also tested https://openlibrary.org/account/loans works, and return book works.

@cdrini cdrini merged commit 6e446ae into internetarchive:master Apr 17, 2020
@cclauss cclauss deleted the logging-exceptions-in-lending.py branch April 17, 2020 14:32
@cdrini
Copy link
Collaborator

cdrini commented Apr 17, 2020

Already seeing some errors logged!

Traceback (most recent call last):
  File "/opt/openlibrary/deploys/openlibrary/openlibrary/openlibrary/core/lending.py", line 245, in get_availability
    content = urllib.request.urlopen(url=url, timeout=config_http_request_timeout).read()
  File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 448, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 404: Not Found
2020-04-17 14:32:29 [28818] [openlibrary.core.lending] [ERROR] get_available([])
Traceback (most recent call last):
  File "/opt/openlibrary/deploys/openlibrary/openlibrary/openlibrary/core/lending.py", line 216, in get_available
    request = urllib.request.Request(url=url)
  File "/usr/lib/python2.7/urllib2.py", line 202, in __init__
    self.__original = unwrap(url)
  File "/usr/lib/python2.7/urllib.py", line 1060, in unwrap
    url = url.strip()
AttributeError: 'list' object has no attribute 'strip'
2020-04-17 14:32:29 [28818] [openlibrary.core.lending] [ERROR] get_available([])
Traceback (most recent call last):
  File "/opt/openlibrary/deploys/openlibrary/openlibrary/openlibrary/core/lending.py", line 216, in get_available
    request = urllib.request.Request(url=url)
  File "/usr/lib/python2.7/urllib2.py", line 202, in __init__
    self.__original = unwrap(url)
  File "/usr/lib/python2.7/urllib.py", line 1060, in unwrap
    url = url.strip()

@cdrini
Copy link
Collaborator

cdrini commented Apr 17, 2020

Note to self (and @mekarpeles ); let's keep on eye on this during deploy to make sure it doesn't explode the size of our log files.

@cclauss
Copy link
Collaborator Author

cclauss commented Apr 17, 2020

2020-04-17 14:32:29 [28818] [openlibrary.core.lending] [ERROR] get_available([])

url is an empty list! Wrong datatype and no content. A posterchild for bare exceptions.

@cclauss
Copy link
Collaborator Author

cclauss commented Apr 17, 2020

https://github.com/internetarchive/openlibrary/blob/master/openlibrary/core/lending.py#L165-L166 is the problem because compose_ia_url() is supposed to return a url (str) but the code here is returning an empty list. Need those Python 3 type hints ;-)

@cclauss
Copy link
Collaborator Author

cclauss commented Apr 20, 2020

Were there many of these in the logs? How do we hunt down why this is happening?

@mekarpeles mekarpeles added python Pull requests that update Python code and removed Module: Python labels Sep 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
python Pull requests that update Python code
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants