This tutorial explains how to visualize statistics of BN mentioned in our paper.
We provide a visualize tool for batch statistics. You can find it in cls/stat_util.py.
Please first replace MABN with naive BN, then run train.py with option --record_statistics. The total command is like:
python -m torch.distributed.launch --nproc_per_node=8 train.py \
--save ./your_path_to_logs \
--train_dir ./your_imagenet_training_dataset_dir \
--val_dir ./your_imagenet_eval_dataset_dir \
--gpu_num=8 --record_statistics --batch_size 256Following figures illustrate batch statistics with batch size 2 and batch size 32. Batch statistics collapse much faster in small batch settings. (We randomly visualize one BN layer here, most BN layers have the same phenomenon.)
Visualization of four statistics mentioned in our paper.
$\mu_{\mathcal{B}}$
$\sigma^2_{\mathcal{B}}$
$g_{\mathcal{B}}$
$\Psi_{\mathcal{B}}$
