Add support for flash attention

[WilliamTambellini]

Prerequisites

• I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

ggml core lib to use flash attention (v1 or v2) at least for nvidia runtime.
Refs:
https://github.com/Dao-AILab/flash-attention
https://tridao.me/publications/flash2/flash2.pdf
#2257

Current Behavior

ggml core lib not using flashattn.

Environment and Context

• Operating System, e.g. for Linux:
  Any recent Linux with 'recent' nvidia drivers/gpus.

• SDK version, e.g. for Linux:
  cuda 11 or 12, no preference

[KerfuffleV2]

There's #778 - it didn't get merged since it didn't seem to provide an advantage though.

[Aya-ZIbra]

I am interested what benchmarks are used to check if flash attn improved the current implementation.

[ggerganov]

I was testing with single-batch inference back at the time.
Need to redo the tests with parallel batches #3228

[Aya-ZIbra]

I think flash will be effective when the load consists of more prefill than decode. Parallel loads may satisfy this.

[ggerganov]

Superseded by #3365