| date | title | id |
|---|---|---|
2020-10-09 07:24:36 -0700 |
Source code transformation for Autodiff |
2020-10-09t14-24-36z |
As we saw, forward-mode autodiff allows for the interleaving of intermediate differentials to compute the derivative of a given function.
One way of implementing this is explicitly interleaving these intermediate differentials into the source program, by using a compiler that transforms the source code into a differential version of itself.
Interestingly, source code transformation can also be used for reverse-mode autodiff, as is done in Zygote [@innes_dont_2019] and Tangent [@van_merrienboer_tangent_2017]