Revert "[PERF] Use faster way of decode in tokenizer: avoid useless list-to-list conversion (#20000)"

[DarkLight1337]

This reverts commit 58eee5f.

Purpose

See #20000 (comment), this optimization is not necessary anymore

Test Plan

Test Result

(Optional) Documentation Update

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

[gemini-code-assist]

Code Review

This pull request reverts a previous performance optimization in the decode_tokens function. The change is simple and reverts the code to a previous state. I have one suggestion to refactor the updated function to improve consistency with other utility functions in the same file, which will aid future maintainability.

[gemini-code-assist]

For better consistency with the encode_tokens function that follows, consider refactoring this to use a kwargs dictionary. This approach makes the function's structure similar to its counterpart, improving the overall maintainability and readability of the file.

Suggested change

	if skip_special_tokens is not None:
	return decode_method(token_ids,
	skip_special_tokens=skip_special_tokens)
	return tokenizer.decode(token_ids,
	skip_special_tokens=skip_special_tokens)
	return decode_method(token_ids)
	return tokenizer.decode(token_ids)
	kwargs = {}
	if skip_special_tokens is not None:
	kwargs["skip_special_tokens"] = skip_special_tokens
	return tokenizer.decode(token_ids, **kwargs)

[22quinn]

LGTM