Big science#1382

Closed

jeffra wants to merge 24 commits intomasterfrom

Commits on Jun 6, 2021

Minor tweaks to support Megatron 2.4 + DS 3D
Shaden Smith
committed
pipe partitioning
Shaden Smith
committed

Commits on Jun 11, 2021

Commits on Jun 13, 2021

send/recv
Shaden Smith
committed
isend/irecv missing wait
Shaden Smith
committed

Commits on Jun 14, 2021

turn off async ops
Shaden Smith
committed
Merge branch 'megatron2.4-3d-sendrecv' into megatron2.4-3d
Shaden Smith
committed

Commits on Jun 26, 2021

less verbose load
Shaden Smith
committed

Commits on Jun 30, 2021

Merge branch 'master' into megatron2.4-3d
jeffra
authored

Commits on Jul 13, 2021

added shaden's set_train_batch_size patches, plus formatting
jeffra
committed

Commits on Jul 26, 2021

Adds engine.was_step_applied() (#1251)
Shaden Smith
authored
Cleaning up tensor/pipe parallel accounting. (#1252)
Shaden Smith
authored

Commits on Jul 30, 2021

Correctness fix PP+ZeRO for gradient accumulation + updates from master (#1263)

jeffra
and
tjruwase
authored

Commits on Jul 31, 2021

Commits on Aug 2, 2021

fix empty grad zero tests
jeffra
committed

Commits on Aug 9, 2021

Use mpu in DeepSpeedConfig() call (#1271)
tjruwase
authored
API for obtaining global gradient norm (#1292)

tjruwase
and
Shaden Smith
authored

Commits on Aug 11, 2021

turn excessive noise off (#1293)
stas00
authored

Commits on Aug 25, 2021

Commits on Aug 30, 2021

update for cuda-11.4 (#1329)

stas00
and
jeffra
committed