✨ 💫 sent char_span through with spaCy & regex & ♻️ Refactoring for more languages support

[nipunsadvilkar]

🐛Old Approach

TLDR - Sentences(string) were right ✅ Sentence char spans were wrong ❌

Earlier pysbd v0.2.3 depended on creating Doc object through regex search - SENTENCE_BOUNDARY_REGEX - on a preprocessed text. The preprocessing is an essential part of pysbd rules which adds Unicode symbols to decide when to/not to break the sentence. Due to the addition of Unicode symbols, char_spans found by regex were not proper in contrast to original text. Another caveat was - spaCy doesn't provide Doc object if there is leading/trailing whitespace and returns None which used to lead to wrong char spans.

✨New Approach

Use Token.is_sent_start attribute over getting an Doc object to get correct char_spans

Rely on pysbd to give proper sentences (strings) out of the original text -> use regex to find those sentence within original text & keep start char indices of sentence matches > iterate over each Token of Doc object and set Token.is_sent_start True (if in sent_start_token_idx) else False.

In preliminary testing, new approach seems to be resolving issues raised by developers.

• Add tests for following issues - Incorrect text span start and end returned #49, Incorrect text span start and end returned #53, Different segmentation with Spacy and when using pySBD directly #55 , Handle irregularities between pySBD & pySBD + spaCy sentence output #59, Long number stalls process. #58
• 🔐Fix security vulnerability bleach - Bump bleach from 3.1.0 to 3.1.4 #62
• Use Github Actions for CI
• Refactoring & cleanups

CHANGELOG

• ✨ 💫 sent char_span through with spaCy & regex approach - ✨ 💫 sent char_span through with spaCy & regex & ♻️ Refactoring for more languages support #63
• ♻️ Refactoring to support multiple languages
• ✨ 💫Initial language support for - Hindi, Marathi, Chinese, Spanish
• ✅ Updated tests - more coverage & regression tests for issues
• 👷👷🏻‍♀️ GitHub actions for CI-CD
• 💚☂️ Add code coverage - coverage.py Add Codecov
• 🐛 Fix incorrect text span & vanilla pysbd vs spacy output discrepancy - Incorrect text span start and end returned #49, Incorrect text span start and end returned #53, Different segmentation with Spacy and when using pySBD directly #55 , Handle irregularities between pySBD & pySBD + spaCy sentence output #59
• 🐛 Fix NUMBERED_REFERENCE_REGEX for zero or one time - Long number stalls process. #58
• 🔐Fix security vulnerability bleach - Bump bleach from 3.1.0 to 3.1.4 #62

[codecov-commenter]

Codecov Report

❗ No coverage uploaded for pull request base (master@f7c640f). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master      #63   +/-   ##
=========================================
  Coverage          ?   97.45%           
=========================================
  Files             ?       20           
  Lines             ?      748           
  Branches          ?        0           
=========================================
  Hits              ?      729           
  Misses            ?       19           
  Partials          ?        0           

Flag	Coverage Δ
#unittests	97.45% <0.00%> (?)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f7c640f...0ab7a6f. Read the comment docs.