Analyzing Disassembled and Decompiled Linear Algebra Functions

A good way to see what a linear algebra function's/operation's disassembly looks like is to play around with GLM on Compiler Explorer. GLM is an open-source C++ math library that has all of the common linear algebra types and operations defined. You can add the GLM library using the "Libraries" tab in the compiler view, I also recommend adding the compiler flags -O3 -m32 to compile optimized x86_32 machine code. I have set up a playground here https://godbolt.org/z/WqKW69EEo with these prerequisites satisfied.

The GLM source code is also a useful resource for comparing decompilation. The dot product specialization definitions and cross product specialization definition can be found among other common geometric functions in glm/geometric.hpp. Common matrix functions are defined in glm/detail/func_matrix.inl and the operators defined on types, e.g. multiplication, are in the glm/detail directory. The documentation is easy to navigate and so is the code, so you shouldn't have a hard time finding anything.

Keep in mind that an optimizing compiler will inline functions and reorder instructions. When chaining these functions/operations the instructions for each may be interleaved with each other. This entangling can make it harder to reason about where one function/operation begins and ends from another. I find it helpful to clearly label the temporary variables and try to put the decompiled code in a more reasonable order in a text editor.