-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Harmonize HF environment variables + deprecate use_auth_token #6066
Conversation
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Why did you think that removing this arg is a good idea? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That looks great - thanks for all the effort!
@patrickvonplaten Why do you think that removing use_auth_token arg is a good idea? |
@patrickvonplaten also why do you approve this commit with not all checks passed? |
@kopyl Argument Regarding CI checks I guess the failing error was unrelated to this PR. |
@patrickvonplaten thank you very much. I either did not read properly the description, or there was no description originally. |
…gface#6066) * Harmonize HF environment variables + deprecate use_auth_token * fix import * fix
…gface#6066) * Harmonize HF environment variables + deprecate use_auth_token * fix import * fix
…gface#6066) * Harmonize HF environment variables + deprecate use_auth_token * fix import * fix
(Similar to huggingface/transformers#27564. Sorry in advance for the size of the diff, I hope it'll be ok :) )
This PR aims at harmonizing environment variables in the HF ecosystem (following up on huggingface/huggingface_hub#1786). I also took the time to review the integration of
huggingface_hub
intodiffusers
to update to latest "standard". I tried to break nothing -staying at iso-feature- but the review will be much appreciated 😄. The overall goal is simply to have less duplication in the codebase.List of changes:
use_auth_token
in favor oftoken
everywhere and in a backward-compatible way. To do do I used the@validate_hf_hub_args
decorator that checks which ofuse_auth_token
and/ortoken
are passed. For now usinguse_auth_token
do not raise a warning but it will in the future. The goal is to slowly adapt scripts before deprecating officially. And the deprecation will only have to happen inhuggingface_hub
thanks to the decorator.use_auth_token=True
. This is not necessary anymore since the token is always passed to the server by default (already the case since ~1y ago).local_files_only
toHF_HUB_OFFLINE
by default. This is already done inhuggingface_hub
(less duplicated code)cache_dir
toDIFFUSERS_CACHE
by default. This is already done inhuggingface_hub
(less duplicated code)DIFFUSERS_CACHE
which is strictly equal toHF_HUB_CACHE
(shared with other libraries)HF_HUB_OFFLINE
andDISABLE_TELEMETRY
from environment variables but usehuggingface_hub
values insteadhuggingface_hub.constants.HF_HOME
instead ofhuggingface_hub.constants.hf_cache_home
huggingface_hub.get_full_repo_name
(instead of defining it in diffusers)_get_model_file
(that one was for myself - it looks like a heavily used method)And that's it. Mostly cleaning some stuff in a backward-compatible way to remove duplicated work. Sorry in advance for the tedious review 🙈