Tutorial to compute Transformer FLOPs (Equation 6 in the paper)

[yuweihao]

Hi everyone, Someone asked me how to compute Transformer FLOPs (Equation 6 in the paper). I wrote a tutorial and hope it can help. Feel free to offer feedback so that I can improve it.

You can also refer to PDF version.

[WowLucy]

Thank you very much! May I ask how you draw those colored squares for matrices? It looks like latex, but I could not find related latex code online. Thank you again!

[yuweihao]

Hi @WowLucy, I used draw.io and just clumsily assembled the small squares into a matrix.

[WowLucy]

Cool. Thank you!

[Yubel426]

Hi, the tutorial looks great! What tool did you use to create this PDF?

[yuweihao]

Hi @Yubel426 , thank you! I used draw.io to create it.