Support for FP32

[houEricSY]

Thanks for the code of flash-attention, it's brilliant! But it seems that it doesn't support FP32 computation, I wonder how can I use flash-attention in FP32 setting.

[tridao]

We currently support Turing (e.g. RTX 2080) and Ampere (e.g. RTX 3080) GPUs. We rely on tensor cores for matrix multiplication, which older GPUs lack.
You might be interested in the memory-efficient attention implemented by the xformers team (targeting fp32 instead of fp16):
facebookresearch/xformers#267
facebookresearch/xformers#281

[fmassa]

Hi,

We have just pushed a PR in facebookresearch/xformers#362 which contains V100 and P100 support as well, and dispatches to FlashAttention for the cases where it is supported.