This repo consists of work exploring and implementing RMSProp and Adam with Weight Decay for an assignment for Edinburgh University's Machine Learning Practical course. The framework used in this project is provided by the University and can be found on GitHub, including complete setup instructions. The inspiration for the coursework is from Fixing Weight Decay Regularization in Adam by Ilya Loshchilov and Frank Hutter.
This work consists of NumPy implementations of both RMSProp and Adam (see /mlp/learning_rules.py), an implementation of cosine annealing with warm restarts from Loshchilov and Hutter (see /mlp/schedulers), and an implementation of L2 regularisation and weight decay, also from Loshchilov and Hutter (see /mlp/learning_rules.py). Experiments were then conducted to compare RMSProp and Adam, explore the effects of cosine annealing and compare weight decay to L2 regularisation for Adam. Experiment results and discussion are inlcluded in the final report