[REQUEST] FLOPS Profiler support `scaled_dot_product_attention` and `__matmul__`

[KimmiShi]

Currently, the flops profiler does not handle F.scaled_dot_product_attention in transformer models. And it also cannot get flops of expreesions like a@b. These defects affect the flops results.

Flops profile should handle the cases decribed above.

[BitCalSaul]

Did this issue get fixed? The profiler would get a wrong result for the transformer-based model.

[KimmiShi]

Did this issue get fixed? The profiler would get a wrong result for the transformer-based model.

Hi, I fixed this in #4724, you can have a try.
And also, do not use 3rd party libs like flash-attn when profiling flops, these ops cannot be processd properly.

[BitCalSaul]

Thanks but I still found that @ operation cannot be counted. I test my model by adding line by line and found this issue. From #4724, it seems like this commit has not been merged into the main branch?

[KimmiShi]

Thanks but I still found that @ operation cannot be counted. I test my model by adding line by line and found this issue. From #4724, it seems like this commit has not been merged into the main branch?

yes, it is not merged yet, you can cherry pick this commit.

[BitCalSaul]

Thanks I found the @ operation has been counted which is so great. But there is no name corresponding to this operation in the profiler report.
By the way, I also found the transpose, reshape, permute, and some basic operation like "+", "-", "*" have not been counted. I'm wondering if it would be supported for the flops counting in the future.

[KimmiShi]

Thanks I found the @ operation has been counted which is so great. But there is no name corresponding to this operation in the profiler report. By the way, I also found the transpose, reshape, permute, and some basic operation like "+", "-", "*" have not been counted. I'm wondering if it would be supported for the flops counting in the future.

@ operation is not a module so it is not shown in report.

"some basic operation like "+", "-", "*" " these operations are memory bounded, and often have much smaller flops than matmuls, so I would ignore these operations.

[BitCalSaul]

I'm wondering if the permute, transpose, and reshape operations would consume a big computation since they did take a lot of time.

[KimmiShi]

I'm wondering if the permute, transpose, and reshape operations would consume a big computation since they did take a lot of time.

These ops usually takes less time than large matmuls. However, the profile result you posted shows that the qkv is relatively small, so ops like dropout, softmax,... their time cost cannot be ignored.

[BitCalSaul]

Thanks for your answer. I am testing the complexity of the transformer-based model to get a basic conception of it, which is the reason the qkv looks small since I gave a very short sequence into the model.