crfcut: Ensure splitting of sentences using Terminal Punctuation #905
Conversation
Crfcut creating issues for split using terminal punctuation commonly '.' (full stop) which should be treated as end of the sentence, Modified the function such that it should split using terminal punctuations and avoid any kind of empty strings.
|
Hello @varunkatiyar819! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:
Comment last updated at 2024-04-03 07:51:42 UTC |
Formatting checks handled
Removed redundant brackets.
There was a problem hiding this comment.
It look great!
This pull request change crfcut.
Before
>>> sent_tokenize("สวัสดีชาวโลก. \n โลกนี้เต็มไปด้วยความลึกลับ")
['สวัสดีชาวโลก. \n โลกนี้เต็มไปด้วยความลึกลับ']
>>> sent_tokenize("สวัสดีชาวโลก! \n โลกนี้เต็มไปด้วยความลึกลับ")
['สวัสดีชาวโลก! \n โลกนี้เต็มไปด้วยความลึกลับ']After
>>> sent_tokenize("สวัสดีชาวโลก. \n โลกนี้เต็มไปด้วยความลึกลับ")
['สวัสดีชาวโลก.', ' \n โลกนี้เต็มไปด้วยความลึกลับ']
>>> sent_tokenize("สวัสดีชาวโลก! \n โลกนี้เต็มไปด้วยความลึกลับ")
['สวัสดีชาวโลก!', ' \n โลกนี้เต็มไปด้วยความลึกลับ']Thank you!
There was a problem hiding this comment.
It look the test is break. Can you fix? @varunkatiyar819
['ฉันไปโรงเรียน เธอไปโรงพยาบาล'] != ['ฉันไปโรงเรียน ', 'เธอไปโรงพยาบาล']
Modified the logic for splitting of sentences due to empty strings or spaces.
|
Logic for the Code is been modified and tested as well. Please do have a check. Thanks |
bact
changed the title
Crfcut creating issues for split using terminal punctuation commonly '.' (full stop) which should be treated as end of the sentence, Modified the function such that it should split using terminal punctuations and avoid any kind of empty strings.
What does this changes
Brief summary of the changes
What was wrong
Description of what was the root cause of the issue.
How this fixes it
Description of how the changes fix the issue.
Fixes #904
Your checklist for this pull request
🚨Please review the guidelines for contributing to this repository.