Documentation issue, tokenization_llama.py, legacy = True

[dpaleka]

System Info

v4.32.1

Who can help?

@ArthurZucker

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

The error message in tokenization_llama.py asks the user to provide a flag which does nothing:

transformers/src/transformers/models/llama/tokenization_llama.py

Lines 140 to 148 in 77713d1

	if legacy is None:
	logger.warning_once(
	f"You are using the default legacy behaviour of the {self.__class__}. If you see this, DO NOT PANIC! This is"
	" expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you."
	" If you want to use the new behaviour, set `legacy=True`. This should only be set if you understand what it"
	" means, and thouroughly read the reason why this was added as explained in"
	" https://github.com/huggingface/transformers/pull/24565"
	)
	legacy = True

seems wrong:

(Or the documentation is faulty. Does legacy = True turn the legacy behavior on or off?)

Expected behavior

Not sure, but possibly, the error message should ask the user to set legacy=False to get the new behavior.

[ArthurZucker]

No, not everyone needs / should be using the new behaviour. legacy = True means True, I want to use legacy.
However there is indeed a type, legacy = False should be set for the new behaviour !