Dist testing backend fixes, etc.#708

Merged

jeffra merged 9 commits intomasterfrom

jeffra/dist_changes

Jan 29, 2021

Commits on Jan 28, 2021

comment out some backward tests (based on discussion with @RezaYazdaniAminabadi)
jeffra
committed
remove empty_grad=True, if zero-2 enabled this will cause a hang
jeffra
committed

Commits on Jan 29, 2021

refactor to address comment
jeffra
committed