[squash] Big science v1#1406

Merged

jeffra merged 41 commits intobig-science-v2from

Sep 27, 2021

Commits on Jun 6, 2021

Minor tweaks to support Megatron 2.4 + DS 3D
Shaden Smith
committed
pipe partitioning
Shaden Smith
committed

Commits on Jun 9, 2021

unit test for bugfix #1135 (#1148)

Shaden Smith
and
jeffra
authored

Commits on Jun 10, 2021

Add Windows support in README, use c++17 on Windows to support latest VC & cuda build tool (#1151)

eltonzheng
and
jeffra
authored

Commits on Jun 11, 2021

Commits on Jun 13, 2021

send/recv
Shaden Smith
committed
isend/irecv missing wait
Shaden Smith
committed

Commits on Jun 14, 2021

turn off async ops
Shaden Smith
committed
Merge branch 'megatron2.4-3d-sendrecv' into megatron2.4-3d
Shaden Smith
committed

Commits on Jun 15, 2021

Fix bugs in the tutorial documentation (#1157)
hyunwoongko
authored

Commits on Jun 17, 2021

add assert to make sure zero isn't enabled with 1bit-* (#1169)
jeffra
authored

Commits on Jun 18, 2021

Avoid partitioning small activations (#1154)
tjruwase
authored

Commits on Jun 21, 2021

remove torchvision dependency (#1178)
jeffra
authored

Commits on Jun 23, 2021

Commits on Jun 24, 2021

Fix bugs about non-contiguous tensor broadcasting (#1168)

authored

Commits on Jun 26, 2021

undo noise (#1191)
stas00
authored
less verbose load
Shaden Smith
committed

Commits on Jun 28, 2021

clean up logging (#1190)

stas00
and
jeffra
authored

Commits on Jun 30, 2021

Merge branch 'master' into megatron2.4-3d
jeffra
authored

Commits on Jul 13, 2021

added shaden's set_train_batch_size patches, plus formatting
jeffra
committed

Commits on Jul 26, 2021

Adds engine.was_step_applied() (#1251)
Shaden Smith
authored
Cleaning up tensor/pipe parallel accounting. (#1252)
Shaden Smith
authored

Commits on Jul 30, 2021

Correctness fix PP+ZeRO for gradient accumulation + updates from master (#1263)

jeffra
and
tjruwase
authored

Commits on Jul 31, 2021

Commits on Aug 2, 2021

fix empty grad zero tests
jeffra
committed

Commits on Aug 9, 2021

Use mpu in DeepSpeedConfig() call (#1271)
tjruwase
authored
API for obtaining global gradient norm (#1292)

tjruwase
and
Shaden Smith
authored

Commits on Aug 11, 2021

turn excessive noise off (#1293)
stas00
authored

Commits on Aug 25, 2021

Commits on Aug 30, 2021

update for cuda-11.4 (#1329)

stas00
and
jeffra
committed