Cleaning up tensor/pipe parallel accounting.#1252

Merged

ShadenSmith merged 4 commits intodeepspeedai:big-sciencefrom

ShadenSmith:shaden/pipe_replicated

Jul 26, 2021

Commits on Jul 26, 2021

removes repeated overflow log
Shaden Smith
committed
pipe_replicated
Shaden Smith
committed
_pipe_replicated -> ds_pipe_replicated
Shaden Smith
committed
Adds send/recv fallback to bcast when torch version <= 1.8
Shaden Smith
committed