Matrix Calculus

In this page, we introduce a differential based method for vector and matrix derivatives (matrix calculus), which only needs a few simple rules to derive most matrix derivatives. This method is useful and well established in mathematics; however, few documents clearly or detailedly describe it. Therefore, we make this page aiming at the comprehensive introduction of matrix calculus via differentials.

* If you want results only, there is an awesome online tool Matrix Calculus. If you want "how to," let's get started.

• 0. Notation
• 1. Matrix Calculus via Differentials
  • 1.1 Differential Identities
  • 1.2 Deriving Matrix Derivatives
    • 1.2.1 Proof of chain rules (identities 3)
    • 1.2.2 Practical examples
• 2. Conclusion

0. Notation

• , , and denote , , and respectively.
• The first half of the alphabet denote constants, and the second half denote variables.
• denotes matrix transpose, is the trace, is the determinant, and is the adjugate matrix.
• is the Kronecker product and is the Hadamard product.
• Here we use numerator layout, while the online tool Matrix Calculus seems to use mixed layout. Please refer to Wiki - Matrix Calculus - Layout Conventions for the detailed layout definitions, and keep in mind that different layouts lead to different results. Below is the numerator layout,

1. Matrix Calculus via Differentials

1.1 Differential Identities

• Identities 1

• Identities 2

• Identities 3 - chain rules

• Identities 4 - total differential. Actually, all identities 1 are the matrix form of the total differential in eq. (24).

1.2 Deriving Matrix Derivatives

To derive a matrix derivative, we repeat using the identities 1 (the process is actually a chain rule) assisted by identities 2.

1.2.1 Proof of chain rules (identities 3)

• 

finally from eq. (2), we get .

• 

finally from eq. (3), we get .

• 

finally from eq. (1), we get .

• 

finally from eq. (5), we get .

1.2.2 Practical examples

E.g. 1,

finally from eq. (2), we get .

E.g. 2,

finally from eq. (3), we get . From line 3 to 4, we use the conclusion of , that is to say, we can derive more complicated matrix derivatives by properly utilizing the existing ones. From line 6 to 7, we use to introduce the in order to use eq. (3) later, which is common in scalar-by-matrix derivatives.

E.g. 3,

finally from eq. (3), we get .

E.g. 4,

finally from eq. (3), we get .

E.g. 5 - two layer neural network, , is a loss function such as Softmax Cross Entropy and MSE, is an element-wise activation function such as Sigmoid and ReLU

For ,

finally from eq. (3), we get .

For ,

finally from eq. (3), we get .

E.g. 6, prove

Since

then

therefore

* See examples.md for more examples.

2. Conclusion

Now, if we fully understand the core mind of the above examples, I believe we can derive most matrix derivatives in Wiki - Matrix Calculus by ourself. Please correct me if there is any mistake, and raise issues to request the detailed steps of computing the matrix derivatives that you are interested in.