[Bug] `tokenizer.model_max_length` is different when loading model from shortcut or local path

[ShaneTian]

To reproduce

When loading model from local path, the tokenizer.model_max_length is VERY_LARGE_INTEGER:

from transformers import GPT2Tokenizer

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
print(tokenizer.model_max_length)
# 1024

tokenizer = GPT2Tokenizer.from_pretrained("path/to/local/gpt2")
print(tokenizer.model_max_length)
# 1000000000000000019884624838656

Related code

transformers/src/transformers/tokenization_utils_base.py

Lines 1858 to 1864 in 25156eb

	# Set max length if needed
	if pretrained_model_name_or_path in cls.max_model_input_sizes:
	# if we're using a pretrained model, ensure the tokenizer
	# wont index sequences longer than the number of positional embeddings
	model_max_length = cls.max_model_input_sizes[pretrained_model_name_or_path]
	if model_max_length is not None and isinstance(model_max_length, (int, float)):
	init_kwargs["model_max_length"] = min(init_kwargs.get("model_max_length", int(1e30)), model_max_length)

transformers/src/transformers/models/gpt2/tokenization_gpt2.py

Line 153 in ebbe8cc

max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES

transformers/src/transformers/models/gpt2/tokenization_gpt2.py

Lines 56 to 62 in ebbe8cc

	PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {
	"gpt2": 1024,
	"gpt2-medium": 1024,
	"gpt2-large": 1024,
	"gpt2-xl": 1024,
	"distilgpt2": 1024,
	}

Expected behavior

Assign correct model_max_length when loading model from local path:

from transformers import GPT2Tokenizer

tokenizer = GPT2Tokenizer.from_pretrained("path/to/local/gpt2")
print(tokenizer.model_max_length)
# 1024

[shabie]

I've also encountered the same problem specially obvious working on clusters having no internet access. The problem stems from the fact that other than the string passed to the from_pretrained, there really is no good way right now of knowing what model is in the folder. I thought model_type might help but that's not the case.

Which means as a quick fix, we can resort to a solution that works in "many" cases.

So the key bit of code that is skipped if path to the model folder is different is this:

transformers/src/transformers/tokenization_utils_base.py

Lines 1858 to 1864 in 25156eb

	# Set max length if needed
	if pretrained_model_name_or_path in cls.max_model_input_sizes:
	# if we're using a pretrained model, ensure the tokenizer
	# wont index sequences longer than the number of positional embeddings
	model_max_length = cls.max_model_input_sizes[pretrained_model_name_or_path]
	if model_max_length is not None and isinstance(model_max_length, (int, float)):
	init_kwargs["model_max_length"] = min(init_kwargs.get("model_max_length", int(1e30)), model_max_length)

If modified to rely on the heuristic that most configs, if available, do have either max_position_embeddings or n_positions, it solves the problem for those models (and doesn't hurt those who dont):

try:
    config
except NameError:
    config = None

# Set max length if needed
if pretrained_model_name_or_path in cls.max_model_input_sizes:
    model_max_length = cls.max_model_input_sizes[pretrained_model_name_or_path]
    if model_max_length is not None and isinstance(model_max_length, (int, float)):
        init_kwargs["model_max_length"] = min(init_kwargs.get("model_max_length", int(1e30)), model_max_length)

elif hasattr(config, "max_position_embeddings") or hasattr(config, "n_positions"):
    model_max_length = config.n_positions if hasattr(config, "n_positions") else config.max_position_embeddings
    if model_max_length is not None and isinstance(model_max_length, (int, float)):
        init_kwargs["model_max_length"] = min(init_kwargs.get("model_max_length", int(1e30)), model_max_length)

Another way is to raise a warning if its not found:

# Set max length if needed
if pretrained_model_name_or_path in cls.max_model_input_sizes:
    model_max_length = cls.max_model_input_sizes[pretrained_model_name_or_path]
    if model_max_length is not None and isinstance(model_max_length, (int, float)):
        init_kwargs["model_max_length"] = min(init_kwargs.get("model_max_length", int(1e30)), model_max_length)
else:
    # raise warning about passing the `model_max_length`

This option has the downside that people will need to look into the config.json itself or try to find it somehow which may not be immediately obivous.

[LysandreJik]

@SaulLu, could you take a look at this issue?

[SaulLu]

Thank you very much for this detailed issue! Indeed, I completely understand that this behavior is not satisfactory.

We added a new feature some time ago that saves the tokenizer type in the tokenizer_config.json file in the tokenizer_class key. This new key allows retrieving the value of model_max_length. The problem you are experiencing is related to the fact that the tokenizer_config file of gpt2 was created before this feature existed.

While waiting for me to come back to you with a more satisfactory solution, I advise you to load the GPT2 tokenizer then save it in a local folder and use these local files for the tokenizer you want to use locally.

from transformers import GPT2TokenizerFast

tokenizer = GPT2TokenizerFast.from_pretrained("gpt2")
tokenizer.save_pretrained("gpt2_tokenizer_fixed")

from transformers import GPT2Tokenizer

tokenizer = GPT2Tokenizer.from_pretrained("gpt2_tokenizer_fixed")
print(tokenizer.model_max_length)
# 1024

You can even push it to your account on HF if you prefer to retrieve it easily with .from_pretrained 😊 .

tokenizer = GPT2TokenizerFast.from_pretrained("gpt2")
tokenizer.push_to_hub("SaulLu/gpt2_tokenizer_fixed") # with your HF username

tokenizer = GPT2Tokenizer.from_pretrained("SaulLu/gpt2_tokenizer_fixed")
print(tokenizer.model_max_length)
# 1024

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[kofi2614]

Ideally it would be better to save all the related attributes into the config file so that users can load the exact same model that's saved.
In my case I used the legal-bert tokenizer but customized the model_max_length:
tokenizer = BertTokenizer.from_pretrained('nlpaueb/legal-bert-base-uncased')
tokenizer.model_max_length=512
but this attr won't be saved when calling the save_pretrained function, when I load the model from my local directory, the model length will still be the original value

[SaulLu]

I understand that this is not ideal. At the moment, if you want this to be saved in the tokenizer_config.json you need to pass the argument in the from_pretrained method:

tokenizer = BertTokenizer.from_pretrained('nlpaueb/legal-bert-base-uncased', model_max_length=512)

[spreadingmind]

This issue still persists.
Downloaded files from https://huggingface.co/bert-base-multilingual-cased/tree/main and tried to create tokenizer using from_pretrained from local files