Removed tokenizer/processor creation from example scripts #4211
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
qgallouedec
added a commit
that referenced
this pull request
Oct 6, 2025
commit ae6837f Author: Sergio Paniego Blanco <sergiopaniegoblanco@gmail.com> Date: Mon Oct 6 18:40:18 2025 +0200 Removed tokenizer/processor creation from example scripts (#4211) commit 56a8f11 Author: Albert Villanova del Moral <8515462+albertvillanova@users.noreply.github.com> Date: Mon Oct 6 17:45:44 2025 +0200 Replace setup with pyproject and fix packaging unintended modules (#4194) commit 5291015 Author: Sergio Paniego Blanco <sergiopaniegoblanco@gmail.com> Date: Mon Oct 6 16:04:06 2025 +0200 Remove `Optional` from `processing_class` in `PPOTrainer` (#4212) commit 0588b1f Author: Sergio Paniego Blanco <sergiopaniegoblanco@gmail.com> Date: Mon Oct 6 15:57:17 2025 +0200 Updated vLLM integration guide (#4162) Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com> commit 45ee98b Author: Albert Villanova del Moral <8515462+albertvillanova@users.noreply.github.com> Date: Mon Oct 6 11:14:54 2025 +0200 Replace unittest with pytest (#4188) commit 3800a6e Author: Albert Villanova del Moral <8515462+albertvillanova@users.noreply.github.com> Date: Mon Oct 6 11:13:21 2025 +0200 Hotfix: Exclude transformers 4.57.0 for Python 3.9 (#4209) Co-authored-by: Sergio Paniego Blanco <sergiopaniegoblanco@gmail.com> commit 7ad9ce8 Author: Sergio Paniego Blanco <sergiopaniegoblanco@gmail.com> Date: Mon Oct 6 11:04:20 2025 +0200 Remove tokenizer creation from `sft` example script (#4197) commit 0c2dc14 Author: Albert Villanova del Moral <8515462+albertvillanova@users.noreply.github.com> Date: Mon Oct 6 08:31:58 2025 +0200 Remove custome_container for building the docs (#4198) commit ced8b33 Author: burtenshaw <ben.burtenshaw@gmail.com> Date: Mon Oct 6 08:23:11 2025 +0200 [DOCS/FIX] lora without regrets - fix lr (#4207)
qgallouedec
added a commit
that referenced
this pull request
Oct 6, 2025
commit 65eb45c Author: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com> Date: Mon Oct 6 13:07:18 2025 -0600 Apply style and revert change in `sft_video_llm` example (#4214) commit ae6837f Author: Sergio Paniego Blanco <sergiopaniegoblanco@gmail.com> Date: Mon Oct 6 18:40:18 2025 +0200 Removed tokenizer/processor creation from example scripts (#4211) commit 56a8f11 Author: Albert Villanova del Moral <8515462+albertvillanova@users.noreply.github.com> Date: Mon Oct 6 17:45:44 2025 +0200 Replace setup with pyproject and fix packaging unintended modules (#4194) commit 5291015 Author: Sergio Paniego Blanco <sergiopaniegoblanco@gmail.com> Date: Mon Oct 6 16:04:06 2025 +0200 Remove `Optional` from `processing_class` in `PPOTrainer` (#4212) commit 0588b1f Author: Sergio Paniego Blanco <sergiopaniegoblanco@gmail.com> Date: Mon Oct 6 15:57:17 2025 +0200 Updated vLLM integration guide (#4162) Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com> commit 45ee98b Author: Albert Villanova del Moral <8515462+albertvillanova@users.noreply.github.com> Date: Mon Oct 6 11:14:54 2025 +0200 Replace unittest with pytest (#4188) commit 3800a6e Author: Albert Villanova del Moral <8515462+albertvillanova@users.noreply.github.com> Date: Mon Oct 6 11:13:21 2025 +0200 Hotfix: Exclude transformers 4.57.0 for Python 3.9 (#4209) Co-authored-by: Sergio Paniego Blanco <sergiopaniegoblanco@gmail.com> commit 7ad9ce8 Author: Sergio Paniego Blanco <sergiopaniegoblanco@gmail.com> Date: Mon Oct 6 11:04:20 2025 +0200 Remove tokenizer creation from `sft` example script (#4197) commit 0c2dc14 Author: Albert Villanova del Moral <8515462+albertvillanova@users.noreply.github.com> Date: Mon Oct 6 08:31:58 2025 +0200 Remove custome_container for building the docs (#4198) commit ced8b33 Author: burtenshaw <ben.burtenshaw@gmail.com> Date: Mon Oct 6 08:23:11 2025 +0200 [DOCS/FIX] lora without regrets - fix lr (#4207)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Removing tokenizer/processor creation from example scripts when possible. The idea comes from this comment
In some trainers, it's still needed to pass directly the tokenizer/processor so they're not updated here.
Before submitting
Pull Request section?
to it if that's the case.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.