Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Workaround no longer required as of spaCy v2.1.0 #224

Merged
merged 3 commits into from
Mar 28, 2019
Merged

Workaround no longer required as of spaCy v2.1.0 #224

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
72cddff
Workaround no longer required for Japanese as of spaCy v2.1.0
Mar 21, 2019
506d393
Merge branch 'master' into refactor/130
lukehsiao Mar 26, 2019
4690fd3
Update CHANGELOG.rst
lukehsiao Mar 26, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,8 @@ Fixed
spaCy only support version 0.7.
* `@HiromuHota`_: Use black 18.9b0 or higher to be consistent with isort.
(`#225 <https://github.com/HazyResearch/fonduer/issues/225>`_)
* `@HiromuHota`_: Workaround no longer required for Japanese as of spaCy v2.1.0.
(`#224 <https://github.com/HazyResearch/fonduer/pull/224>`_)
* `@senwu`_: Update the metal version.
* `@senwu`_: Expose the ``b`` and ``pos_label`` in training.
* `@senwu`_: Fix the issue that pdfinfo causes parsing error when it contains
Expand Down
14 changes: 0 additions & 14 deletions src/fonduer/parser/spacy_parser.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,20 +125,6 @@ def load_lang_model(self):
language_module = importlib.import_module(f"spacy.lang.{self.lang}")
language_method = getattr(language_module, self.alpha_languages[self.lang])
model = language_method()
""" TODO: Depending on OS (Linux/macOS) and on the sentence to be parsed,
UnicodeDecodeError or ValueError happens at the first use when lang='ja'.
As a workaround, the model parses some sentence before actually being used.
"""
if self.lang == "ja":
try:
model("初期化")
except (UnicodeDecodeError, ValueError):
pass
if self.lang == "zh":
try:
model("初始化")
except (UnicodeDecodeError, ValueError):
pass
self.model = model

def sentence_list_separator_function(self, all_sentence_objs):
Expand Down