Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[TokenizerFast] Fix setting prefix space in __init__ #3

Closed
wants to merge 5 commits into from
Closed

[TokenizerFast] Fix setting prefix space in __init__ #3

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
17b233b
properly support Sequence of pretokenizers
ArthurZucker Aug 16, 2023
00138c3
actual fix
ArthurZucker Aug 16, 2023
ae9c0d8
make sure the fix works. Tests are not working for sure!
ArthurZucker Aug 17, 2023
9631582
hacky way
ArthurZucker Aug 17, 2023
6affdd7
add TODO
ArthurZucker Aug 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 7 additions & 9 deletions src/transformers/models/bloom/tokenization_bloom_fast.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,9 @@
"""Tokenization classes for Bloom."""


import json
import pickle
from typing import TYPE_CHECKING, List, Optional, Tuple

from tokenizers import pre_tokenizers

from ...tokenization_utils_base import BatchEncoding
from ...tokenization_utils_fast import PreTrainedTokenizerFast
from ...utils import logging
Expand Down Expand Up @@ -130,12 +128,12 @@ def __init__(
clean_up_tokenization_spaces=clean_up_tokenization_spaces,
**kwargs,
)
pre_tok_state = json.loads(self.backend_tokenizer.pre_tokenizer.__getstate__())
if pre_tok_state.get("add_prefix_space", add_prefix_space) != add_prefix_space:
pre_tok_class = getattr(pre_tokenizers, pre_tok_state.pop("type"))
pre_tok_state["add_prefix_space"] = add_prefix_space
self.backend_tokenizer.pre_tokenizer = pre_tok_class(**pre_tok_state)

# overwride add_prefix_space
pre_tok_state = pickle.dumps(self.backend_tokenizer.pre_tokenizer)
if add_prefix_space:
pre_tok_state = pre_tok_state.replace(b'"add_prefix_space":false',b'"add_prefix_space": true')
# TODO decoder should also be udpated to reflect this
self.backend_tokenizer.pre_tokenizer = pickle.loads(pre_tok_state)
self.add_prefix_space = add_prefix_space

def _batch_encode_plus(self, *args, **kwargs) -> BatchEncoding:
Expand Down
Loading