AttributeError: module 'torch' has no attribute 'float8_e4m3fn'

[Perpetue237]

System Info

AutoModelForCausalLM.from_pretrained now returns the above error.

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

pip install transformers

from transformers import AutoModelForCausalLM
AutoModelForCausalLM.from_pretrained(
model_path,
cache_dir= cache_dir,
device_map="auto",
quantization_config=bnb_config,
)

Expected behavior

Expected to load the model.

[amyeroberts]

Hi @Perpetue237, thanks for raising an issue!

Please make sure to follow the issue template and provide:

• The running environment: run transformers-cli env in the terminal and copy-paste the output
• A minimal reproducible code snippet. In this case, as model_path, and bnb_config are undefined, we cannot run the code to reproduce the error
• All information relevant to the error, including full error traceback

[Perpetue237]

1. transformers-cli env

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

• transformers version: 4.43.1
• Platform: Linux-5.4.0-146-generic-x86_64-with-glibc2.31
• Python version: 3.10.11
• Huggingface_hub version: 0.24.0
• Safetensors version: 0.4.3
• Accelerate version: 0.32.1
• Accelerate config: not found
• PyTorch version (GPU?): 2.0.1 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using distributed or parallel set-up in script?:
• Using GPU in script?:
• GPU type: NVIDIA A100-SXM4-80GB

2. Code snippet:

 from transformers import BitsAndBytesConfig, AutoModelForCausalLM
 import torch

 model_name = "meta-llama/Meta-Llama-3.1-8B-Instruct"
 bnb_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16)
 model = AutoModelForCausalLM.from_pretrained(
     model_name,
     device_map="auto",
     quantization_config=bnb_config
)

3. The code works well with transformers==4.42.4

[amyeroberts]

@Perpetue237 Can you share the full error traceback as well please?

cc @ArthurZucker

[Perpetue237]

---

AttributeError Traceback (most recent call last)
Cell In[1], line 8
4 model_name = "meta-llama/Meta-Llama-3.1-8B-Instruct"
7 bnb_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16)
----> 8 model = AutoModelForCausalLM.from_pretrained(
9 model_name,
10 device_map="auto",
11 quantization_config=bnb_config
12 )

File /opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:564, in _BaseAutoModelClass.from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
562 elif type(config) in cls._model_mapping.keys():
563 model_class = _get_model_class(config, cls._model_mapping)
--> 564 return model_class.from_pretrained(
565 pretrained_model_name_or_path, *model_args, config=config, **hub_kwargs, **kwargs
566 )
567 raise ValueError(
568 f"Unrecognized configuration class {config.class} for this kind of AutoModel: {cls.name}.\n"
569 f"Model type should be one of {', '.join(c.name for c in cls._model_mapping.keys())}."
570 )

File /opt/conda/lib/python3.10/site-packages/transformers/modeling_utils.py:3903, in PreTrainedModel.from_pretrained(cls, pretrained_model_name_or_path, config, cache_dir, ignore_mismatched_sizes, force_download, local_files_only, token, revision, use_safetensors, *model_args, **kwargs)
3893 if dtype_orig is not None:
3894 torch.set_default_dtype(dtype_orig)
3896 (
3897 model,
3898 missing_keys,
3899 unexpected_keys,
3900 mismatched_keys,
3901 offload_index,
3902 error_msgs,
-> 3903 ) = cls._load_pretrained_model(
3904 model,
3905 state_dict,
3906 loaded_state_dict_keys, # XXX: rename?
3907 resolved_archive_file,
3908 pretrained_model_name_or_path,
3909 ignore_mismatched_sizes=ignore_mismatched_sizes,
3910 sharded_metadata=sharded_metadata,
3911 _fast_init=_fast_init,
3912 low_cpu_mem_usage=low_cpu_mem_usage,
3913 device_map=device_map,
3914 offload_folder=offload_folder,
3915 offload_state_dict=offload_state_dict,
3916 dtype=torch_dtype,
3917 hf_quantizer=hf_quantizer,
3918 keep_in_fp32_modules=keep_in_fp32_modules,
3919 gguf_path=gguf_path,
3920 )
3922 # make sure token embedding weights are still tied if needed
3923 model.tie_weights()

File /opt/conda/lib/python3.10/site-packages/transformers/modeling_utils.py:4377, in PreTrainedModel._load_pretrained_model(cls, model, state_dict, loaded_keys, resolved_archive_file, pretrained_model_name_or_path, ignore_mismatched_sizes, sharded_metadata, _fast_init, low_cpu_mem_usage, device_map, offload_folder, offload_state_dict, dtype, hf_quantizer, keep_in_fp32_modules, gguf_path)
4373 set_module_tensor_to_device(
4374 model_to_load, key, "cpu", torch.empty(*param.size(), dtype=dtype)
4375 )
4376 else:
-> 4377 new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model(
4378 model_to_load,
4379 state_dict,
4380 loaded_keys,
4381 start_prefix,
4382 expected_keys,
4383 device_map=device_map,
4384 offload_folder=offload_folder,
4385 offload_index=offload_index,
4386 state_dict_folder=state_dict_folder,
4387 state_dict_index=state_dict_index,
4388 dtype=dtype,
4389 hf_quantizer=hf_quantizer,
4390 is_safetensors=is_safetensors,
4391 keep_in_fp32_modules=keep_in_fp32_modules,
4392 unexpected_keys=unexpected_keys,
4393 )
4394 error_msgs += new_error_msgs
4395 else:
4396 # Sharded checkpoint or whole but low_cpu_mem_usage==True

File /opt/conda/lib/python3.10/site-packages/transformers/modeling_utils.py:871, in _load_state_dict_into_meta_model(model, state_dict, loaded_state_dict_keys, start_prefix, expected_keys, device_map, offload_folder, offload_index, state_dict_folder, state_dict_index, dtype, hf_quantizer, is_safetensors, keep_in_fp32_modules, unexpected_keys)
867 set_module_kwargs = {}
869 # We convert floating dtypes to the dtype passed. We want to keep the buffers/params
870 # in int/uint/bool and not cast them.
--> 871 if dtype is not None and torch.is_floating_point(param) and param.dtype != torch.float8_e4m3fn:
872 if (
873 keep_in_fp32_modules is not None
874 and any(
(...)
877 and dtype == torch.float16
878 ):
879 param = param.to(torch.float32)

AttributeError: module 'torch' has no attribute 'float8_e4m3fn'

[amyeroberts]

cc @SunMarc, as this logic appears to have come from #32047

[SunMarc]

Oh my bad, I will fix it ! It is only available after torch 2.1

[SunMarc]

LMK if the above PR fixes it ! @Perpetue237. We will patch it soon !

[tanghengjian]

how to update???

[Tellterubouzu]

Same problem, How can I get new version?

[cckuailong]

how to update???

Solved it by run pip install transformers==4.43.2

[Tellterubouzu]

Thank you! It worked. I must have missed the new version.

[tanghengjian]

pip install transformers==4.43.2 has run, but error exist? torch version

torch                         2.0.1+cu118
torchaudio                    2.0.2
torchdiffeq                   0.2.3
torchmetrics                  1.0.3
torchsde                      0.2.6
torchvision                   0.15.2+cu118

[Tellterubouzu]

Did you reboot your kernel after the install?

[Corsky]

same error happened when I tried to load llama3.1 405B with fp8, with torch 2.4.0 (as the issue mentioned above). Tried transformers 4.43.2 but it seems not solved..

[Perpetue237]

I works for me now with the newest version of transformer. I restarted the kernel and clear the cache. And where I am working in a docker container I rebuilt the container.

[Corsky]

@Perpetue237 Which cuda version are you using?

[Perpetue237]

cuda 11.7

[kungfu-eric]

Was it the cuda version too old?

[Perpetue237]

yes @kungfu-eric

[LukeLIN-web]

yes @kungfu-eric

I am using cuda_12.1 but I still have this problem. I find one solution https://www.reddit.com/r/comfyui/comments/1ettg44/comment/lipkvex/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

[LukeLIN-web]

Meet this problem again, I am using, transformers 4.45.2 , cuda_12.2 , torch 12.1
upgrade torch to 2.4.1 can work
pip install --upgrade torch torchvision

[cvtower]

torch 2.4.1+cu124
the error still exists here.

[SunMarc]

What's the error you are getting with the traceback @LukeLIN-web @cckuailong ? It's strange that you are getting it if you've installed the latest transformers as I switched the check to a hasattr call .

[LukeLIN-web]

What's the error you are getting with the traceback @LukeLIN-web @cckuailong ? It's strange that you are getting it if you've installed the latest transformers as I switched the check to a hasattr call .

I meet AttributeError: module 'torch' has no attribute 'float8_e4m3fn' and then
pip install --upgrade torch torchvision upgrade to 2.4.1 torch solves my problem.

[SunMarc]

AttributeError: module 'torch' has no attribute 'float8_e4m3fn'

Then, i'm pretty sure this is because you are using an older version of transformers. If you sent me the full traceback, I will be able to confirm that!

[Boltzmachine]

I can only use torch==2.0.1 . Then how to solve it

[SunMarc]

Just install the latest transformers @Boltzmachine !

[ShobhaRajanna]

pip install transformers==4.46.0
and torch 2.0.1+cu117
torchao 0.6.1
torchaudio 2.0.2+cu117
torchvision 0.15.2+cu117 the error still exists here.

[ShobhaRajanna]

I can only use torch==2.0.1 . Then how to solve it

did you resolved? i usetransformers==4.46.0 the error still exists