Use code on the Hub from another repo #22698
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The documentation is not available anymore as the PR was closed or merged. |
amyeroberts
approved these changes
Apr 12, 2023
| @@ -667,6 +667,11 @@ def _get_config_dict( | |||
| else: | |||
| logger.info(f"loading configuration file {configuration_file} from cache at {resolved_config_file}") | |||
|
|
|||
| if "auto_map" in config_dict and not is_local: | |||
| config_dict["auto_map"] = { | |||
| k: (f"{pretrained_model_name_or_path}--{v}" if "--" not in v else v) | |||
There was a problem hiding this comment.
Is v always a single object here and never a list or tuple, like in the conversion that happens in tokenization_utils_base.py ?
There was a problem hiding this comment.
Should be, though it doesn't hurt to be careful. Will adapt.
LysandreJik
approved these changes
Apr 13, 2023
sgugger
added a commit
that referenced
this pull request
Apr 17, 2023
This was referenced Apr 17, 2023
novice03
pushed a commit
to novice03/transformers
that referenced
this pull request
Jun 23, 2023
* initial work * Add other classes * Refactor code * Move warning and fix dynamic pipeline * Issue warning when necessary * Add test
novice03
pushed a commit
to novice03/transformers
that referenced
this pull request
Jun 23, 2023
Revert "Use code on the Hub from another repo (huggingface#22698)" This reverts commit ea7b0a5.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This makes it easier to maintain only one source of ground truth when using the code on the Hub feature by storing the repo ID on top of the module containing the class inside the config. Thus, when saving and re-pushing a model using code on the Hub, the code is not copied over anymore, but a reference to the original repo containing the code is put.
This might be breaking if some users relied on the code being copied over when
save_pretrained(xxx)is executed. To enable that old behavior, one only needs to call theregister_for_auto_classmethod:then some_path only contains the config and weights of the model. The config will contain links to the repo where the code of the model is defined (
hf-internal-testing/test_dynamic_model) so that it can be reloaded viaTo get the custom code file copied other (behavior before this PR) just do: