Skip to content

Implementation of OpenAI paper with Simple Noise Scale on Fastai V2

License

Notifications You must be signed in to change notification settings

hal-314/fastai-batch-size-finder

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

10 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

README

Implementation of OpenAI paper "An Empirical Model of Large-Batch Training" for Fastai V2.

The code is based on the batch size finder implementation for Fastai V1 by DanyWind (repo V1 / blog / discussion).

This implementation differs on:

  1. It implements exactly the original article and not an aproximation (by default).
  2. Fixes a couple of bugs in noise and scale values. However, they didn't affect on Simple Noise Scale value.

However, you could use the DanyWind aproximation by settting simulate_multi_gpus to False. DanyWind aproximation is faster but numerically more inestable and finds a Simple Noise Scale smaller than the original Simple Noise Scale.

It's tested with fastai 2.1. It should work with fastai>=2.0

TODO:

About

Implementation of OpenAI paper with Simple Noise Scale on Fastai V2

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published