Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

modifying SyncBN doc for FP16 use case #14041

Merged
merged 1 commit into from
Feb 5, 2019
Merged

modifying SyncBN doc for FP16 use case #14041

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion python/mxnet/gluon/contrib/nn/basic_layers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,10 @@ class SyncBatchNorm(BatchNorm):

Standard BN [1]_ implementation only normalize the data within each device.
SyncBN normalizes the input within the whole mini-batch.
We follow the sync-onece implmentation described in the paper [2]_.
We follow the implementation described in the paper [2]_.

Note: Current implementation of SyncBN does not support FP16 training.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So does it not support training or inference?
You say training in this line, and refer to inference in the next line.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SyncBN does not have FP16 support for both training and inference. But for FP16 inference, SyncBN can be replaced with nn.BatchNorm as they have similar functionality.

For FP16 inference, use standard nn.BatchNorm instead of SyncBN.

Parameters
----------
Expand Down