Merge Changes from DS Master#30

Merged

sdtblck merged 81 commits intomainfrom

Apr 22, 2021

Commits on Mar 11, 2021

test sparse self_attn fix
sid
committed

Commits on Mar 12, 2021

Commits on Mar 14, 2021

[doc] pipeline doc typos/improvements (#659)
stas00
authored

Commits on Mar 15, 2021

Commits on Mar 18, 2021

consistent checkpoint filenaming (#865)

stas00
and
tjruwase
authored
[doc] launcher (#868)

stas00
and
jeffra
authored

Commits on Mar 24, 2021

[doc] pipeline (#888)
stas00
authored

Commits on Mar 25, 2021

[debug utils] see_memory_usage fixes (#890)
stas00
authored

Commits on Mar 26, 2021

full fp32 weights reconstruction for zero 2+3 (#892)
stas00
authored

Commits on Mar 27, 2021

Commits on Mar 29, 2021

mlperf attn initial commit
sid
committed

Commits on Mar 30, 2021

Commits on Mar 31, 2021

Commits on Apr 1, 2021

zero.Init() clarification (#880)

stas00
and
tjruwase
authored

Commits on Apr 2, 2021

Commits on Apr 6, 2021

add inference_batch fn
sid
committed

Commits on Apr 7, 2021

Commits on Apr 8, 2021

Commits on Apr 13, 2021

Delete check of pdsh (#941)
tma15
authored

Commits on Apr 16, 2021

Fix ZeRO-3 UnboundLocalError (#968)
tjruwase
authored

Commits on Apr 19, 2021

ZeRO-Infinity (#976)

authored
revert zero-inf change to launcher
jeffra
committed
[docs] zero-inf updates
jeffra
committed
bump to 0.3.15
jeffra
committed
ZeRO-Infinity tutorial additions (#978)
Shaden Smith
authored
[docs] add ZeRO-Inf news items
jeffra
committed
refactor
sid
committed
ZeRO-Infinity docs (#979)
Shaden Smith
authored
[docs] zero-inf updates
jeffra
committed
assert no Z2/Z3 with pipeline and fix some docs links (#980)
Shaden Smith
authored

Commits on Apr 20, 2021

Commits on Apr 22, 2021