Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix roberta retokenization error #982

Merged
merged 4 commits into from
Jan 8, 2020
Merged

fix roberta retokenization error #982

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
c18840d
fix roberta tokenization error
HaokunLiu Dec 28, 2019
77fcd2f
format
HaokunLiu Dec 28, 2019
22ab49b
Merge branch 'master' into tokenization_fix
Jan 3, 2020
4bccaac
Merge branch 'master' into tokenization_fix
Jan 8, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions jiant/utils/retokenize.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,16 +99,16 @@ def create_tokenization_alignment(
tokens: Sequence[str], tokenizer_name: str
) -> Sequence[Tuple[str, str]]:
"""
Builds alignment mapping between space tokenization and tokenization of
choice.
Builds alignment mapping between space tokenization and tokenization of
choice.

Example:
Input: ['Larger', 'than', 'life.']
Output: [('Larger', ['ĠL', 'arger']), ('than', ['Ġthan']), ('life.', ['Ġlife', '.'])]

Parameters
-----------------------
tokens: list[(str)]. list of tokens,
tokens: list[(str)]. list of tokens,
tokenizer_name: str

Returns
Expand Down Expand Up @@ -336,7 +336,7 @@ def process_sentencepiece_for_alignment(t):

def process_bytebpe_for_alignment(t):
"""Add <w> markers to ensure word-boundary alignment."""
if t.startswith("▁"):
if t.startswith("Ġ"):
return "<w>" + re.sub(r"^Ġ", "", t)
else:
return t
Expand Down
6 changes: 3 additions & 3 deletions tests/test_retokenize.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -337,13 +337,13 @@ def test_bytebpe(self):
]
self.token_index_tgt = [
[[0], [1], [2], [3], [4, 5], [6], [7]],
[[0], [1], [2], [3, 4], [5], [6, 7], [8], [9, 10, 11]],
[[0], [1, 2, 3], [4], [5], [6], [7], [8], [9, 10, 11], [12], [13], [14, 15]],
[[0], [1], [2], [3, 4], [5, 6], [7], [8], [9, 10, 11]],
[[0, 1], [2, 3], [4], [5], [6], [7], [8], [9, 10, 11], [12], [13], [14, 15]],
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wasn't computing this manually. I just printed out the output and skim over it.
I know this isn't how test cases are suppose to be made.
Me trying to save 5 minutes in the wrong place has wasted everyone else hours, there can't be enough apology for this.

[[0, 1]],
]
self.span_index_tgt = [
[(0, 4), (6, 8)],
[(0, 1), (3, 6)],
[(0, 1), (3, 7)],
[(0, 4), (8, 16), (8, 12), (9, 16)],
[(0, 2)],
]
Expand Down