Harmonize HF environment variables + other cleaning #27564

Wauplin · 2023-11-17T15:59:45Z

This PR aims at harmonizing environment variables in the HF ecosystem (following up on huggingface/huggingface_hub#1786). I also took the time to review some integration of huggingface_hub into transformers (trying to break nothing). The goal is simply to have less duplication in the codebase. The PR is not ready to be merged as is (see comments) but can be reviewed to discuss the few points. Feedback is very welcome @LysandreJik @amyeroberts @ArthurZucker @ydshieh as I'm necesarily missing some context sometimes.

List of changes:

HF_HOME is the preferred way to set a custom hf path. Second best solution is HF_HUB_CACHE (hf_hub_cache is only for model/dataset caching while hf_home also contains the token and the cached modules -when trust_remote_code-). Therefore:
- using PYTORCH_PRETRAINED_BERT_CACHE is deprecated (v5) but still respected
- using PYTORCH_TRANSFORMERS_CACHE is deprecated (v5) but still respected
- using TRANSFORMERS_CACHE is deprecated (v5) but still respected
DISABLE_TELEMETRY is deprecated in favor of HF_HUB_DISABLE_TELEMETRY
added a deprecation warning around get_cached_models => looks like a legacy unused method (and not compatible with new cache system)
removed try_to_load_from_cache (moved to huggingface_hub)
unsure: I think we should harmonize between get_file_from_repo and cached_file to keep only one of them. The first one raises if the file doesn't exist while the second returns None in such a case. I think having a single method and a public raise_on_error argument should be enough (but open to discussion). Also both methods have the exact same docstring which is misleading.
use huggingface_hub.send_telemetry to send telemetry data. Result is exactly the same but the HTTP call in made in a separate thread (meaning better UX as the user is not blocked)
(nit) in download_url (already deprecated): it's safer to close the file

Wauplin · 2023-11-17T16:02:45Z

src/transformers/utils/hub.py

+# TODO: deprecate `get_file_from_repo` or document it differently?
+#       Docstring is exactly the same as `cached_repo` but behavior is slightly different. If file is missing or if
+#       there is a connection error, `cached_repo` will return None while `get_file_from_repo` will raise an error.
+#       IMO we should keep only 1 method and have a single `raise_error` argument (to be discussed).
 def get_file_from_repo(


To be discussed. I tend to prefer to have a single cached_file method with a raise_on_error argument (and with the correct typing to let the user know when to expect a None value and when it's not possible). I can do it here or in a separate PR.

Yes, I'd vote to uniformize everything under hfh's banner.

Actually I'd be ok for moving the logic to huggingface_hub but this method doesn't exist at the moment.

In hfh we have hf_hub_download that does the job to download/retrieve a file given a repo_id + filename. In transformers, cached_file is slightly different as it accept either a repo_id or a local path (in path_or_repo_id parameter), making it more versatile.

So for now I'd be more in favor of uniforming within transformers and then later move it to huggingface_hub.

get_file_from_repo calls cached_file (while that method has True as default values

_raise_exceptions_for_missing_entries=False, _raise_exceptions_for_connection_errors=False,

Not against to keep only one of them. But the changes might get larger. I would personally do this in a separate PR.

Good idea, let's wait for a separate PR 👍

LysandreJik

Thank you for this important work @Wauplin!

LysandreJik · 2023-11-20T09:17:37Z

src/transformers/utils/hub.py

 )
+from huggingface_hub.utils._deprecation import _deprecate_method


Should we make this a public method to leverage?

Not ideal to depend on private attributes from hfh 🫣

Yeah I realized about that while working on this PR 😭

I'm fine with either making them public (huggingface_hub.utils.deprecate_method ?) which would require to wait next hfh release or making an exception for this one since it's a quite internal method anyway (so using private attribute in transformers).

As mentioned in #27564 (comment) I suggest to still use the private attribute until it's made more official in hfh side. Worse case scenario, the plan is to remove this in 3 transformers releases.

LysandreJik · 2023-11-20T09:18:57Z

src/transformers/utils/hub.py

+# TODO: deprecate `get_file_from_repo` or document it differently?
+#       Docstring is exactly the same as `cached_repo` but behavior is slightly different. If file is missing or if
+#       there is a connection error, `cached_repo` will return None while `get_file_from_repo` will raise an error.
+#       IMO we should keep only 1 method and have a single `raise_error` argument (to be discussed).
 def get_file_from_repo(


Yes, I'd vote to uniformize everything under hfh's banner.

ydshieh

Thank you for the work @Wauplin !

Left a few comments / questions, but overall it's good to me

src/transformers/utils/hub.py

ydshieh · 2023-11-20T10:32:51Z

src/transformers/utils/hub.py

+# TODO: deprecate `get_file_from_repo` or document it differently?
+#       Docstring is exactly the same as `cached_repo` but behavior is slightly different. If file is missing or if
+#       there is a connection error, `cached_repo` will return None while `get_file_from_repo` will raise an error.
+#       IMO we should keep only 1 method and have a single `raise_error` argument (to be discussed).
 def get_file_from_repo(


get_file_from_repo calls cached_file (while that method has True as default values

_raise_exceptions_for_missing_entries=False, _raise_exceptions_for_connection_errors=False,

Not against to keep only one of them. But the changes might get larger. I would personally do this in a separate PR.

src/transformers/utils/hub.py

ydshieh · 2023-11-20T12:53:23Z

Thank you for the explanation. Just one more nit question, in huggingface/huggingface_hub#1786, there is a table of the re-mapped names, like HUGGINGFACE_HUB_CACHE -> HF_CACHE. I am wondering why we still have the usage of constants.HUGGINGFACE_HUB_CACHE.

Wauplin · 2023-11-20T13:48:25Z

Thanks for the reviews @LysandreJik and @ydshieh!

I addressed 2 points:

do not use use_auth_token in deprecated code (see here)
use huggingface_hub.constants.HF_HUB_CACHE instead of huggingface_hub.constants.HUGGINGFACE_HUB_CACHE (see here)

About using from huggingface_hub.utils._deprecation import _deprecate_method I would be keen to keep it as it is for now if that's ok with you @LysandreJik. It's not ideal but still ok I think. In next release of hfh I'll make it more official and in any case _deprecate_method will be used only in the next 3 versions of transformers (so EoL quite soon :) ).

Finally, regarding removing get_file_from_repo in favor of a unified cached_file` let's do it as a separate PR as suggested by @ydshieh. It is not really related to env variable harmonization.

@LysandreJik @ydshieh could you re-review the last changes + message above and approve the PR? I think we are good to merge the current version if that is ok with you. Thanks in advance!

ydshieh

👍

HuggingFaceDocBuilderDev · 2023-11-20T13:57:42Z

The documentation is not available anymore as the PR was closed or merged.

ArthurZucker · 2023-11-20T17:50:47Z

I'll review tomorrow! Thanks for this refactor 😉

amyeroberts

Thanks - really nice tidy up!

~~Only Q is about existing HUGGINGFACE_HUB_CACHE values in the environment~~

Nvm - I saw the discussion here: #27564 (comment)

amyeroberts · 2023-11-20T21:00:53Z

src/transformers/utils/hub.py

+# In code, use `HF_HUB_CACHE` as the default cache path. This variable is set by the library and is guaranteed
+# to be set to the right value.
+#
+# TODO: clean this for v5?


src/transformers/utils/hub.py

ArthurZucker

LGTM

Wauplin · 2023-11-21T17:36:21Z

Yay! 3.5 approvals, that's enough! Thanks everyone 🤗

Harmonize HF environment variables + other cleaning

c45d605

Wauplin requested review from ydshieh, amyeroberts, LysandreJik and ArthurZucker November 17, 2023 15:59

Wauplin commented Nov 17, 2023

View reviewed changes

backward compat

f1a8229

LysandreJik approved these changes Nov 20, 2023

View reviewed changes

ydshieh approved these changes Nov 20, 2023

View reviewed changes

Wauplin added 2 commits November 20, 2023 14:36

switch from HUGGINGFACE_HUB_CACHE to HF_HUB_CACHE

c73764f

revert

bdc2768

Wauplin requested review from LysandreJik and ydshieh November 20, 2023 13:48

ydshieh approved these changes Nov 20, 2023

View reviewed changes

amyeroberts approved these changes Nov 20, 2023

View reviewed changes

ArthurZucker approved these changes Nov 21, 2023

View reviewed changes

Wauplin merged commit d2a980e into main Nov 21, 2023
21 checks passed

Wauplin deleted the harmonize-environment-variables branch November 21, 2023 17:36

Wauplin mentioned this pull request Dec 5, 2023

Harmonize HF environment variables + deprecate use_auth_token huggingface/diffusers#6066

Merged

Wauplin mentioned this pull request Dec 18, 2023

Remove warning if DISABLE_TELEMETRY is used #28113

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Harmonize HF environment variables + other cleaning #27564

Harmonize HF environment variables + other cleaning #27564

Wauplin commented Nov 17, 2023

Wauplin Nov 17, 2023

LysandreJik Nov 20, 2023

Wauplin Nov 20, 2023

ydshieh Nov 20, 2023

Wauplin Nov 20, 2023

LysandreJik left a comment

LysandreJik Nov 20, 2023

Wauplin Nov 20, 2023 •

edited

Loading

Wauplin Nov 20, 2023 •

edited

Loading

LysandreJik Nov 20, 2023

ydshieh left a comment

ydshieh Nov 20, 2023

ydshieh commented Nov 20, 2023

Wauplin commented Nov 20, 2023

ydshieh left a comment

HuggingFaceDocBuilderDev commented Nov 20, 2023 •

edited

Loading

ArthurZucker commented Nov 20, 2023

amyeroberts left a comment •

edited

Loading

amyeroberts Nov 20, 2023

ArthurZucker left a comment

Wauplin commented Nov 21, 2023

		)
		from huggingface_hub.utils._deprecation import _deprecate_method

Harmonize HF environment variables + other cleaning #27564

Harmonize HF environment variables + other cleaning #27564

Conversation

Wauplin commented Nov 17, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LysandreJik left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Wauplin Nov 20, 2023 • edited Loading

Choose a reason for hiding this comment

Wauplin Nov 20, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ydshieh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ydshieh commented Nov 20, 2023

Wauplin commented Nov 20, 2023

ydshieh left a comment

Choose a reason for hiding this comment

HuggingFaceDocBuilderDev commented Nov 20, 2023 • edited Loading

ArthurZucker commented Nov 20, 2023

amyeroberts left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ArthurZucker left a comment

Choose a reason for hiding this comment

Wauplin commented Nov 21, 2023

Wauplin Nov 20, 2023 •

edited

Loading

Wauplin Nov 20, 2023 •

edited

Loading

HuggingFaceDocBuilderDev commented Nov 20, 2023 •

edited

Loading

amyeroberts left a comment •

edited

Loading