Causal attention hard-codes lower triangular matrices as uint8

[fxmarty]

System Info

- `optimum` version: 1.5.3.dev0
- `transformers` version: 4.25.1
- Platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.35
- Python version: 3.9.12
- Huggingface_hub version: 0.12.0.dev0
- PyTorch version (GPU?): 1.13.1+cu117 (cuda availabe: True)
- Tensorflow version (GPU?): 2.9.1 (cuda availabe: True)

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Not necessarily a bug per se. Likely due to e.g. https://github.com/huggingface/transformers/blob/4d10ffd50614ebcdfe7e99c0092740f0f7234923/src/transformers/models/gpt2/modeling_gpt2.py#L196 . Similar for other models.

optimum-cli export onnx --model gpt2 --task causal-lm --for-ort gpt2_onnx

Expected behavior

/

[fxmarty]

Related: #627 #605

[fxmarty]

Actually comes from https://github.com/huggingface/transformers/blob/4d10ffd50614ebcdfe7e99c0092740f0f7234923/src/transformers/models/gpt2/modeling_gpt2.py#L130 .

I have no idea why torch.uint8 is used and not torch.bool here. Any idea @LysandreJik @sgugger ?

[sgugger]

My best guess would be because it's very old code and PyTorch did not support torch.bool back then.

[fxmarty]

Closing as in any case pytorch represents torch.bool on a byte (pytorch/pytorch#41571 (comment)), and ONNX as well (checking the initializer ByteSize()).