`AutoTokenizer.from_pretrained` does not propagate `token`

[anakin87]

Since Transformers 4.52, the Autotokenizer.from_pretrained loading mechanism has changed and the token is not correctly propagated.

System Info

(Colab)

• transformers version: 4.52.4
• Platform: Linux-6.1.123+-x86_64-with-glibc2.35
• Python version: 3.11.13
• Huggingface_hub version: 0.33.0
• Safetensors version: 0.5.3
• Accelerate version: 1.7.0
• Accelerate config: not found
• DeepSpeed version: not installed
• PyTorch version (GPU?): 2.6.0+cu124 (False)
• Tensorflow version (GPU?): 2.18.0 (False)
• Flax version (CPU?/GPU?/TPU?): 0.10.6 (cpu)
• Jax version: 0.5.2
• JaxLib version: 0.5.1
• Using distributed or parallel set-up in script?:

Who can help?

@Rocketknight1 @ArthurZucker @Wauplin
I have the impression that this is related to #36588

Reproduction

In this code example, I am trying to load a private tokenizer using the token parameter (not the env var).

import os
from transformers import AutoTokenizer

# we first make sure that the token is not present in environment variables
# if the env var is present, THE BUG DOES NOT OCCUR
os.environ.pop('HF_TOKEN', None)


model = "deepset/bert-base-NER" # a valid private model I can access
token = "..."

tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name_or_path=model, token=token)

Error

/usr/local/lib/python3.11/dist-packages/huggingface_hub/utils/_auth.py:86: UserWarning: Access to the secret `HF_TOKEN` has not been granted on this notebook. You will not be requested again. Please restart the session if you want to be prompted again. warnings.warn( --------------------------------------------------------------------------- HTTPError Traceback (most recent call last) [/usr/local/lib/python3.11/dist-packages/huggingface_hub/utils/_http.py](https://localhost:8080/#) in hf_raise_for_status(response, endpoint_name) 408 try: --> 409 response.raise_for_status() 410 except HTTPError as e:

8 frames
/usr/local/lib/python3.11/dist-packages/requests/models.py in raise_for_status(self)
1023 if http_error_msg:
-> 1024 raise HTTPError(http_error_msg, response=self)
1025

HTTPError: 401 Client Error: Unauthorized for url: https://huggingface.co/api/models/deepset/bert-base-NER/tree/main/additional_chat_templates?recursive=False&expand=False

The above exception was the direct cause of the following exception:

RepositoryNotFoundError Traceback (most recent call last)
/tmp/ipython-input-6-4075835132.py in <cell line: 0>()
6
----> 7 tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name_or_path=model, token=token)

/usr/local/lib/python3.11/dist-packages/transformers/models/auto/tokenization_auto.py in from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
1030
1031 if tokenizer_class_fast and (use_fast or tokenizer_class_py is None):
-> 1032 return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
1033 else:
1034 if tokenizer_class_py is not None:

/usr/local/lib/python3.11/dist-packages/transformers/tokenization_utils_base.py in from_pretrained(cls, pretrained_model_name_or_path, cache_dir, force_download, local_files_only, token, revision, trust_remote_code, *init_inputs, **kwargs)
1966 )
1967 else:
-> 1968 for template in list_repo_templates(
1969 pretrained_model_name_or_path,
1970 local_files_only=local_files_only,

/usr/local/lib/python3.11/dist-packages/transformers/utils/hub.py in list_repo_templates(repo_id, local_files_only, revision, cache_dir)
159 if not local_files_only:
160 try:
--> 161 return [
162 entry.path.removeprefix(f"{CHAT_TEMPLATE_DIR}/")
163 for entry in list_repo_tree(

/usr/local/lib/python3.11/dist-packages/transformers/utils/hub.py in (.0)
159 if not local_files_only:
160 try:
--> 161 return [
162 entry.path.removeprefix(f"{CHAT_TEMPLATE_DIR}/")
163 for entry in list_repo_tree(

/usr/local/lib/python3.11/dist-packages/huggingface_hub/hf_api.py in list_repo_tree(self, repo_id, path_in_repo, recursive, expand, revision, repo_type, token)
3166 encoded_path_in_repo = "/" + quote(path_in_repo, safe="") if path_in_repo else ""
3167 tree_url = f"{self.endpoint}/api/{repo_type}s/{repo_id}/tree/{revision}{encoded_path_in_repo}"
-> 3168 for path_info in paginate(path=tree_url, headers=headers, params={"recursive": recursive, "expand": expand}):
3169 yield (RepoFile(**path_info) if path_info["type"] == "file" else RepoFolder(**path_info))
3170

/usr/local/lib/python3.11/dist-packages/huggingface_hub/utils/_pagination.py in paginate(path, params, headers)
35 session = get_session()
36 r = session.get(path, params=params, headers=headers)
---> 37 hf_raise_for_status(r)
38 yield from r.json()
39

/usr/local/lib/python3.11/dist-packages/huggingface_hub/utils/_http.py in hf_raise_for_status(response, endpoint_name)
457 " https://huggingface.co/docs/huggingface_hub/authentication"
458 )
--> 459 raise _format(RepositoryNotFoundError, message, response) from e
460
461 elif response.status_code == 400:

RepositoryNotFoundError: 401 Client Error. (Request ID: Root=1-685bc7f1-448cc03d560cc5bc2bc95865;1708f136-0e94-4e7d-a6b4-4e38a9c50920)

Repository Not Found for url: https://huggingface.co/api/models/deepset/bert-base-NER/tree/main/additional_chat_templates?recursive=False&expand=False.
Please make sure you specified the correct repo_id and repo_type.
If you are trying to access a private or gated repo, make sure you are authenticated. For more details, see https://huggingface.co/docs/huggingface_hub/authentication
Invalid username or password.

Expected behavior

The tokenizer loads without errors.

[Wauplin]

Hi @anakin87, sorry you're facing this issue. Can you upgrade huggingface_hub to latest version, run

huggingface-cli auth list

in your terminal and copy-paste the output here?

I suspect that you have a token saved locally (not as env variable but in your $HF_HOME) and that this token is outdated.

If that's not the case, could you run huggingface-cli env and copy-paste the output as well? Thanks!

[anakin87]

Thanks for the response @Wauplin!

I am on a fresh Colab instance and I upgraded huggingface_hub.

huggingface-cli auth list output:

No access tokens found.

huggingface-cli env output:

Copy-and-paste the text below in your GitHub issue.

• huggingface_hub version: 0.33.1
• Platform: Linux-6.1.123+-x86_64-with-glibc2.35
• Python version: 3.11.13
• Running in iPython ?: No
• Running in notebook ?: No
• Running in Google Colab ?: No
• Running in Google Colab Enterprise ?: No
• Token path ?: /root/.cache/huggingface/token
• Has saved token ?: False
• Configured git credential helpers:
• FastAI: 2.7.19
• Tensorflow: 2.18.0
• Torch: 2.6.0+cu124
• Jinja2: 3.1.6
• Graphviz: 0.21
• keras: 3.8.0
• Pydot: 3.0.4
• Pillow: 11.2.1
• hf_transfer: 0.1.9
• gradio: 5.31.0
• tensorboard: N/A
• numpy: 2.0.2
• pydantic: 2.11.7
• aiohttp: 3.11.15
• hf_xet: 1.1.4
• ENDPOINT: https://huggingface.co/
• HF_HUB_CACHE: /root/.cache/huggingface/hub
• HF_ASSETS_CACHE: /root/.cache/huggingface/assets
• HF_TOKEN_PATH: /root/.cache/huggingface/token
• HF_STORED_TOKENS_PATH: /root/.cache/huggingface/stored_tokens
• HF_HUB_OFFLINE: False
• HF_HUB_DISABLE_TELEMETRY: False
• HF_HUB_DISABLE_PROGRESS_BARS: None
• HF_HUB_DISABLE_SYMLINKS_WARNING: False
• HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
• HF_HUB_DISABLE_IMPLICIT_TOKEN: False
• HF_HUB_ENABLE_HF_TRANSFER: False
• HF_HUB_ETAG_TIMEOUT: 10
• HF_HUB_DOWNLOAD_TIMEOUT: 10

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[anakin87]

Up. (Commenting to avoid the "stale" label)

[abdokaseb]

@anakin87 I believe the problem had been resolved, I tested on a fresh colab notebook this and it worked

import os
from transformers import AutoTokenizer


model = "meta-llama/Llama-3.2-1B-Instruct" # model that requires token
token = "..."

tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name_or_path=model, token=token)

with output

tokenizer_config.json: 100% 54.5k/54.5k [00:00<00:00, 1.17MB/s]
tokenizer.json: 100% 9.09M/9.09M [00:00<00:00, 16.8MB/s]
special_tokens_map.json: 100% 296/296 [00:00<00:00, 7.87kB/s]

[anakin87]

It still fails for me with the latest transformers release (4.55.2).

Try removing the token env var (os.environ.pop('HF_TOKEN', None)).

[ArthurZucker]

Gonna have to close this, I cannot reproduce with "meta-llama/Llama-3.2-1B-Instruct"

1. move my token in .cache/hub to _token making sur it is ignored
2. checked with hf auth whoami that I was not logged in
3. use your reproducer, checked it did download the files.
   Make sure you isolate the problem because the token argument is passed everywhere, or if you can provide a copy of the repo you used (so we can be sure it is the deepset/bert-base-NER that fails

[brandenkmurray]

@ArthurZucker

This still seems to be an issue with private repos. I've cloned answerdotai/ModernBERT-base to a private repo at brandenkmurray/private-model-2 and to a public repo at brandenkmurray/public-model-2.

transformers==4.56.1

import transformers
from transformers import AutoTokenizer

transformers.__version__
# '4.56.1'

token = "my-read-token"

# Works!
tokenizer = AutoTokenizer.from_pretrained("brandenkmurray/public-model-2", token=token)

# Fails!
tokenizer = AutoTokenizer.from_pretrained("brandenkmurray/private-model-2", token=token)

I get error:

huggingface_hub.errors.RepositoryNotFoundError: 401 Client Error. (Request ID: Root=1-68ca0a34-40078e047c91c123206255a5;edf531d5-3b4e-4599-b203-4257617bf28e)

Repository Not Found for url: https://huggingface.co/api/models/brandenkmurray/private-model-2/tree/main/additional_chat_templates?recursive=False&expand=False.
Please make sure you specified the correct `repo_id` and `repo_type`.
If you are trying to access a private or gated repo, make sure you are authenticated. For more details, see https://huggingface.co/docs/huggingface_hub/authentication
Invalid username or password.

If I downgrade to transformers==4.51.3 (the most recent working version), they both work.

import transformers
from transformers import AutoTokenizer

transformers.__version__
# '4.51.3'

token = "my-read-token"

# Works!
tokenizer = AutoTokenizer.from_pretrained("brandenkmurray/public-model-2", token=token)

# Works!
tokenizer = AutoTokenizer.from_pretrained("brandenkmurray/private-model-2", token=token)

[ArthurZucker]

cc @Rocketknight1 can you have a look? 🤗

[Rocketknight1]

Hi @brandenkmurray, if you're certain it worked in the past but fails on the current version, can you (or anyone else who has the same issue) do a git bisect? This will tell us the exact commit where the failure began, which will help a lot with debugging.