Yang Liu   ·   Jeremy Bernstein   ·   Markus Meister   ·   Yisong Yue
- Grab nero.py and place it in your Pytorch project directory. Then type:
from nero import Nero
optimizer = Nero(net.parameters(), lr=0.01)- An initial learning rate of
lr = 0.01is the recommended default. This worked in almost all our experiments. Otherwise trylr=0.001. - Learning rate decay over the course of optimisation also helps.
This repository was built by Yang Liu and Jeremy Bernstein to accompany the following paper:
Learning by Turning: Neural Architecture Aware Optimisation.
We're putting this code here so that you can test out our optimisation algorithm in your own applications, and also so that you can attempt to reproduce the experiments in our paper.
If something isn't clear or isn't working, let us know in the Issues section or contact yang@abacus.ai and bernstein@caltech.edu.
.
├── cGAN/ # Class conditional GAN image generation experiments
├── cifar/ # CIFAR-10 classification experiments
├── imagenet/ # ImageNet classification experiments
├── mnist/ # MNIST experiments with deep MLP and reparameterisation
├── optim/ # optimiser definitions
├── ppo/ # reinforcement learning experiment
├── transformer-wmt16/ # large transformer
├── wikitext2/ # small transformer
├── LICENSE # license on our algorithm
├── README.md # the page you're reading now
└── nero.py # our optimiser
If you find Nero useful, feel free to cite the paper:
@misc{nero2021,
title={Learning by Turning: Neural Architecture Aware Optimisation},
author={Yang Liu and Jeremy Bernstein and Markus Meister and Yisong Yue},
year={2021},
eprint={arXiv:2102.07227}
}We are making our algorithm available under a CC BY-NC-SA 4.0 license. The other code we have used obeys other license restrictions as indicated in the subfolders.