Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for better handling error from ES search #422

Merged
merged 5 commits into from
Apr 8, 2021

Conversation

ankur09011
Copy link
Contributor

JIRA Ticket Number

JIRA TICKET: ML-2249

Description of change

Adding support for handling exceptions raised from ES for:

  1. When no query is hit
  2. ES is going down

Checklist (OPTIONAL):

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules

@haptik-deployment
Copy link

👍 No lint errors found.

@haptik-deployment
Copy link

👍 No lint errors found.

@haptik-deployment
Copy link

UNIT TESTS HAVE PASSED... Good To Merge

@codecov
Copy link

codecov bot commented Apr 2, 2021

Codecov Report

Merging #422 (64a4e8c) into develop (799f882) will decrease coverage by 0.00%.
The diff coverage is 15.04%.

Impacted file tree graph

@@             Coverage Diff             @@
##           develop     #422      +/-   ##
===========================================
- Coverage    41.03%   41.02%   -0.01%     
===========================================
  Files           91       91              
  Lines         9436     9433       -3     
===========================================
- Hits          3872     3870       -2     
+ Misses        5564     5563       -1     
Impacted Files Coverage Δ
datastore/elastic_search/query.py 0.00% <0.00%> (ø)
datastore/exceptions.py 0.00% <0.00%> (ø)
ner_v2/detectors/textual/queries.py 94.64% <ø> (ø)
ner_v1/api.py 22.60% <9.09%> (-1.21%) ⬇️
ner_v2/api.py 13.04% <14.28%> (+0.10%) ⬆️
ner_v2/detectors/textual/elastic_search.py 64.04% <14.28%> (-3.02%) ⬇️
ner_v2/detectors/textual/utils.py 87.14% <78.57%> (+1.24%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d05f29b...64a4e8c. Read the comment docs.

@haptik-deployment
Copy link

👍 No lint errors found.

@ankur09011 ankur09011 changed the title {WIP}: Support for better handling error from ES search Support for better handling error from ES search Apr 6, 2021
@haptik-deployment
Copy link

👍 No lint errors found.

@haptik-deployment
Copy link

UNIT TESTS HAVE PASSED... Good To Merge

ner_v2/api.py Show resolved Hide resolved
Copy link
Contributor

@chiragjn chiragjn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

current set of changes are okay but they don't address the wrong response code and don't log the failing es queries

For hits miss raised Rwquest Exceptions
@haptik-deployment
Copy link

👍 No lint errors found.

@ankur09011 ankur09011 requested a review from chiragjn April 7, 2021 08:31
@haptik-deployment
Copy link

UNIT TESTS HAVE PASSED... Good To Merge

datastore/exceptions.py Outdated Show resolved Hide resolved
ner_v1/api.py Outdated
Comment on lines 280 to 291
except TypeError as err:
ner_logger.exception(f"Error in text_synonym for: {request.path}, error: {err}")
return HttpResponse(status=500)
except KeyError as err:
ner_logger.exception(f"Error in text_synonym for: {request.path}, error: {err}")
return HttpResponse(status=500)
except es_exceptions.ConnectionTimeout as err:
ner_logger.exception(f"Error in text_synonym for: {request.path}, error: {err}")
return HttpResponse(status=500)
except es_exceptions.ConnectionError as err:
ner_logger.exception(f"Error in text_synonym for: {request.path}, error: {err}")
return HttpResponse(status=500)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor, considering we are doing the exact same thing we can combine the blocks

except (TypeError, KeyError, ...) as err

Comment on lines 437 to +448
except TypeError as e:
ner_logger.exception('Exception for city: %s ' % e)
return HttpResponse(status=500)
except KeyError as e:
ner_logger.exception('Exception for text_synonym: %s ' % e)
return HttpResponse(status=500)
except es_exceptions.ConnectionTimeout as e:
ner_logger.exception('Exception for text_synonym: %s ' % e)
return HttpResponse(status=500)
except es_exceptions.ConnectionError as e:
ner_logger.exception('Exception for text_synonym: %s ' % e)
return HttpResponse(status=500)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment as above

Copy link
Contributor

@chiragjn chiragjn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some more suggestions, please verify it will be easy to grep the Datastore exception cases along with the queries and the responses from logs because without structured logging multiline logs (like tracebacks) from multiple parallel workers can get entangled and hard to decipher

…ute to catch at root level

- Fix error messages to make it more readble
@ankur09011 ankur09011 requested a review from chiragjn April 7, 2021 17:56
@haptik-deployment
Copy link

👍 No lint errors found.

@sonarqubecloud
Copy link

sonarqubecloud bot commented Apr 7, 2021

Kudos, SonarCloud Quality Gate passed!

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 2 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

Copy link
Contributor

@chiragjn chiragjn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes LGTM 👍
Please manually test the new ES exception cases if they are being raised and logged in a useable format

Can delete the empty datastore_exceptions.py file at the root, but that's not a blocker

@haptik-deployment
Copy link

UNIT TESTS HAVE PASSED... Good To Merge

@ankur09011 ankur09011 merged commit bd5503f into develop Apr 8, 2021
@chiragjn chiragjn mentioned this pull request Apr 8, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants