Code

[Zerobubble] merge main.#6142

Code

Merged

duanjunwen merged 132 commits intohpcaitech:mainhpcaitech/ColossalAI:mainfrom duanjunwen:dev/zero_bubbleduanjunwen/ColossalAI:dev/zero_bubbleCopy head branch name to clipboardNov 19, 2024

+3,682-123

Commits

Commits on Aug 22, 2024

• [feat] add zerobubble pp (just a frame now); add POC test for dx_dw; add test for zerobubble;

  duanjunwen

  committed
  ee9baed

  Copy full SHA for ee9baed

Commits on Aug 23, 2024

• [feat] add dw test;

  duanjunwen

  committed
  c18ef06

  Copy full SHA for c18ef06

• [fix] fix weight not close;

  duanjunwen

  committed
  203033e

  Copy full SHA for 203033e

Commits on Aug 26, 2024

• [update] update text;

  duanjunwen

  committed
  107230d

  Copy full SHA for 107230d

• Merge branch 'main' into dev/zero_bubble

  duanjunwen

  committed
  fd5526b

  Copy full SHA for fd5526b

• [feat] add test run_fwd_bwd automatic scheduling;

  duanjunwen

  committed
  1d75045

  Copy full SHA for 1d75045

Commits on Aug 27, 2024

• [feat] split communication and calculation; fix pop empty send_bwd_buffer error;

  duanjunwen

  committed
  5e09c8b

  Copy full SHA for 5e09c8b

• [feat] add test for p & p grad;

  duanjunwen

  committed
  f1c1a87

  Copy full SHA for f1c1a87

• [feat] add comments for ZBV func;

  duanjunwen

  committed
  1b4bb2b

  Copy full SHA for 1b4bb2b

• [fix] rm useless assign and comments;

  duanjunwen

  committed
  283c9ff

  Copy full SHA for 283c9ff

• [fix] fix ci test; add pytest;

  duanjunwen

  committed
  9e0bd1a

  Copy full SHA for 9e0bd1a

• [feat] add run_fwd_bwd_with_microbatch (replace input) & test; add p&p.grad assert close test & all pass;

  duanjunwen

  committed
  8b37323

  Copy full SHA for 8b37323

• [feat] add apply v_schedule graph; p & p.grad assert err exist;

  duanjunwen

  committed
  fe20916

  Copy full SHA for fe20916

Commits on Aug 28, 2024

• [fix] update

  duanjunwen

  committed
  29383b2

  Copy full SHA for 29383b2

• [feat] fix ci; add assert;

  duanjunwen

  committed
  d6e3d7d

  Copy full SHA for d6e3d7d

• [feat] fix poc format

  duanjunwen

  committed
  b5f7b4d

  Copy full SHA for b5f7b4d

• [feat] fix func name & ci; add comments;

  duanjunwen

  committed
  582ba0d

  Copy full SHA for 582ba0d

• [fix] fix poc test; add comments in poc;

  duanjunwen

  committed
  b1419ef

  Copy full SHA for b1419ef

Commits on Aug 29, 2024

• [feat] add optim backward_b_by_grad

  duanjunwen

  committed
  4c4b01b

  Copy full SHA for 4c4b01b

• [feat] fix optimizer bwd b & w; support return accum loss & output

  duanjunwen

  committed
  48ba22d

  Copy full SHA for 48ba22d

Commits on Aug 30, 2024

• [feat] add fwd_bwd_step, run_fwd_only;

  duanjunwen

  committed
  6af81d8

  Copy full SHA for 6af81d8

• [fix] fix optim bwd; add license for v_schedule; remove redundant attributes; fix schedule loop "while"--> "for"; add communication dict;

  duanjunwen

  committed
  8eb6eac

  Copy full SHA for 8eb6eac

• [fix] fix communication_map;

  duanjunwen

  committed
  a7b767b

  Copy full SHA for a7b767b

Commits on Sep 2, 2024

• [feat] update test; rm comments;

  duanjunwen

  committed
  6d18d38

  Copy full SHA for 6d18d38

• [fix] rm zbv in hybridplugin

  duanjunwen

  committed
  77fe442

  Copy full SHA for 77fe442

• [fix] fix optim bwd;

  duanjunwen

  committed
  591a13b

  Copy full SHA for 591a13b

Commits on Sep 3, 2024

• [fix] fix optim bwd;

  duanjunwen

  committed
  a48afc4

  Copy full SHA for a48afc4

• [fix] rm output.data after send fwd;

  duanjunwen

  committed
  ab643c9

  Copy full SHA for ab643c9

• [fix] fix bwd step if condition; remove useless comments and format info;

  duanjunwen

  committed
  4c1f81c

  Copy full SHA for 4c1f81c

• [fix] fix detach output & release output;

  duanjunwen

  committed
  b4103f1

  Copy full SHA for b4103f1

• [fix] rm requir_grad for output;

  duanjunwen

  committed
  20503cd

  Copy full SHA for 20503cd

Commits on Sep 4, 2024

• [fix] fix requir grad position and detach position and input&output local buffer append position;

  duanjunwen

  committed
  e6e1a97

  Copy full SHA for e6e1a97

• [feat] add memory assertation;

  duanjunwen

  committed
  2f09c37

  Copy full SHA for 2f09c37

• [fix] fix mem check;

  duanjunwen

  committed
  4a35834

  Copy full SHA for 4a35834

Commits on Sep 9, 2024

• [fix] mem assertation'

  duanjunwen

  committed
  400e5e5

  Copy full SHA for 400e5e5

• [fix] fix mem assertation

  duanjunwen

  committed
  35a7b63

  Copy full SHA for 35a7b63

• [fix] fix mem; use a new model shape; only assert mem less and equal than theo;

  duanjunwen

  committed
  a5ec3d4

  Copy full SHA for a5ec3d4

• [fix] fix model zoo import;

  duanjunwen

  committed
  fed8b15

  Copy full SHA for fed8b15

• [fix] fix redundant detach & clone; add buffer assertation in the end;

  duanjunwen

  committed
  7568b34

  Copy full SHA for 7568b34

• [fix] add output_obj_grad assert None at bwd b step; replace input_obj.require_grad_ with treemap;

  duanjunwen

  committed
  ce58d8e

  Copy full SHA for ce58d8e

• [fix] update optim state dict assert (include param group & state); fix mem assert after add optim;

  duanjunwen

  committed
  8366a78

  Copy full SHA for 8366a78

• [fix] add testcase with microbatch 4;

  duanjunwen

  committed
  6c2a120

  Copy full SHA for 6c2a120

Commits on Sep 10, 2024

• [zerobubble]Support ZeroBubble Pipeline (hpcaitech#6034)

  Show description for 11ae684
  duanjunwen

  authored
  11ae684

  Copy full SHA for 11ae684

Commits on Sep 12, 2024

• [feat] moehybrid support zerobubble;

  duanjunwen

  committed
  9bc3b6e

  Copy full SHA for 9bc3b6e

Commits on Sep 18, 2024

• [fix] fix zerobubble pp for shardformer type input;

  duanjunwen

  committed
  3dbad10

  Copy full SHA for 3dbad10

• [feat] add more test;

  duanjunwen

  committed
  af2c2f8

  Copy full SHA for af2c2f8

Commits on Sep 19, 2024

• Merge remote-tracking branch 'upstream/feature/zerobubble' into dev/zero_bubble

  duanjunwen

  committed
  1f5c725

  Copy full SHA for 1f5c725

• [fix] fix require_grad & deallocate call;

  duanjunwen

  committed
  6ee9584

  Copy full SHA for 6ee9584

• [fix] updatw bwd b&w input; dict --> list[torch.Tensor]

  duanjunwen

  committed
  349272c

  Copy full SHA for 349272c

• [fix] fix bwd w input;

  duanjunwen

  committed
  a115106

  Copy full SHA for a115106

• [fix] fix mem assert;

  duanjunwen

  committed
  4753bf7

  Copy full SHA for 4753bf7

Commits on Sep 20, 2024

• [fix] fix input_tensors buffer append input_obj(dict) --> Tuple (microbatch, input_obj) , and all bwd b related cal logic;

  duanjunwen

  committed
  2678377

  Copy full SHA for 2678377

• [fix] use tree_flatten replace dict traverse;

  duanjunwen

  committed
  c6d6ee3

  Copy full SHA for c6d6ee3

• [fix] rm comments;

  duanjunwen

  committed
  b6616f5

  Copy full SHA for b6616f5

• [fix] fix fwd branch, fwd pass both micro_batch & internal_inputs'

  duanjunwen

  committed
  1739df4

  Copy full SHA for 1739df4

• [fix] fix pipeline util func deallocate --> release_tensor_data; fix bwd_b loss bwd branch;

  duanjunwen

  committed
  da3220f

  Copy full SHA for da3220f

Commits on Sep 23, 2024

• [fix] fix detach clone release order;

  duanjunwen

  committed
  c114d14

  Copy full SHA for c114d14

• [fix] fix ci --> oom in 4096 hidden dim;

  duanjunwen

  committed
  a875212

  Copy full SHA for a875212

• [fix] fix dumb clone;

  duanjunwen

  committed
  6c1e155

  Copy full SHA for 6c1e155

Commits on Sep 24, 2024

• [fix] fix detach_output_obj clone;

  duanjunwen

  committed
  7e6f793

  Copy full SHA for 7e6f793

• Merge pull request hpcaitech#6065 from duanjunwen/dev/zero_bubble

  Show description for 8501202
  duanjunwen

  authored
  8501202

  Copy full SHA for 8501202

Commits on Sep 25, 2024

• [fix] fix stage_indices;

  duanjunwen

  committed
  fc8b016

  Copy full SHA for fc8b016

• [fix] fix traverse; traverse dict --> traverse tensor List;

  duanjunwen

  committed
  83163fa

  Copy full SHA for 83163fa

Commits on Sep 26, 2024

• [fix] fix zerobubble; support shardformer model type;

  duanjunwen

  committed
  a92e167

  Copy full SHA for a92e167

• [fix] rm comments;

  duanjunwen

  committed
  45f17fc

  Copy full SHA for 45f17fc

• [fix] fix test_pipeline_utils ci;

  duanjunwen

  committed
  c5503b0

  Copy full SHA for c5503b0

• [fix] remove duplicate arg; rm comments;

  duanjunwen

  committed
  bb0390c

  Copy full SHA for bb0390c

• [fix] remove chunk 0 stage 0 bwd b; u don't have to cal micrbatch's dx;

  duanjunwen

  committed
  64ceea7

  Copy full SHA for 64ceea7

• [fix] rm print & comments;

  duanjunwen

  committed
  1342a98

  Copy full SHA for 1342a98

Commits on Sep 27, 2024

• Merge pull request hpcaitech#6069 from duanjunwen/dev/zero_bubble

  Show description for b804fdc
  duanjunwen

  authored
  b804fdc

  Copy full SHA for b804fdc

• [plugin] hybrid support zero bubble pipeline (hpcaitech#6060)

  Show description for af6aa9e
  

  3 peopleauthored
  af6aa9e

  Copy full SHA for af6aa9e

Commits on Sep 29, 2024

• [feat] zerobubble support moehybridplugin;

  duanjunwen

  committed
  d634795

  Copy full SHA for d634795

• [feat] update optimizer bwd; ä¸

  duanjunwen

  committed
  5c8bbf6

  Copy full SHA for 5c8bbf6

Commits on Sep 30, 2024

• [fix] fix build ci;

  duanjunwen

  committed
  6975c50

  Copy full SHA for 6975c50

Commits on Oct 8, 2024

• [zerobubble] rebase main (hpcaitech#6075)

  Show description for 295dd2d
  

  21 peopleauthored
  295dd2d

  Copy full SHA for 295dd2d

• Merge branch 'feature/zerobubble' of github.com:hpcaitech/ColossalAI into dev/zero_bubble

  duanjunwen

  committed
  f4d023c

  Copy full SHA for f4d023c

• [fix] fix mixtral policy;

  duanjunwen

  committed
  292a504

  Copy full SHA for 292a504

• [fix] fix mixtral policy;

  duanjunwen

  committed
  cc500b3

  Copy full SHA for cc500b3

Commits on Oct 9, 2024

• Merge pull request hpcaitech#6077 from duanjunwen/dev/zero_bubble

  Show description for 531773f
  duanjunwen

  authored
  531773f

  Copy full SHA for 531773f

• [feat] support zbv in mixtral benchmark;

  duanjunwen

  committed
  3f5bec8

  Copy full SHA for 3f5bec8

Commits on Oct 10, 2024

• [fix] MixtralForCausalLMPolicy get_held_layer support zbv;

  duanjunwen

  committed
  9ee80fc

  Copy full SHA for 9ee80fc

• [feat] update MixtralPipelineForwards --> mixtral_model_forward; support zbv;

  duanjunwen

  committed
  72b507a

  Copy full SHA for 72b507a

• [feat] support MixtralPipelineForwards--> mixtral_for_causal_lm_forward for zbv

  duanjunwen

  committed
  e234dfa

  Copy full SHA for e234dfa

Commits on Oct 11, 2024

• [zero bubble] support zero (hpcaitech#6080)

  Show description for dac0e07
  

  21 peopleauthored
  dac0e07

  Copy full SHA for dac0e07

• [fix] fix llama, mixtral benchmark zbv loss none bug; update mixtral & llama policy and modeling;

  duanjunwen

  committed
  0ca16d5

  Copy full SHA for 0ca16d5

Commits on Oct 14, 2024

• [feat] Linear1D_COL/ROW support zbv WeightGradStore;

  duanjunwen

  committed
  cfade4c

  Copy full SHA for cfade4c

• [feat] support use_zbv in llama, mixtral modeling; only replace Linear1D_Col/Row policy;

  duanjunwen

  committed
  a11b4b5

  Copy full SHA for a11b4b5

• [fix] fix test case; moe error in second iter

  duanjunwen

  committed
  abd4551

  Copy full SHA for abd4551

• [feat]EPMixtralSparseMoeBlock (op in MOE) support zbv;

  duanjunwen

  committed
  160e9a4

  Copy full SHA for 160e9a4

Commits on Oct 15, 2024

• [fix] fix bwd b; now bwd w only for Layer replaced by Linear1D_Col/Row; other layer perform a fully bwd;

  duanjunwen

  committed
  9912cc8

  Copy full SHA for 9912cc8

• Merge branch 'feature/zerobubble' of github.com:hpcaitech/ColossalAI into dev/zero_bubble

  duanjunwen

  committed
  52dcc73

  Copy full SHA for 52dcc73

• [fix] debug zbv llama test;

  duanjunwen

  committed
  90939b7

  Copy full SHA for 90939b7

Commits on Oct 16, 2024

• [fix] rm use_zbv flag in Shardconfig; rm debug info;

  duanjunwen

  committed
  e76308c

  Copy full SHA for e76308c

• [fix] add & fix llama test

  duanjunwen

  committed
  705b18e

  Copy full SHA for 705b18e

Commits on Oct 24, 2024

• [feat] support meta cache, meta_grad_send, meta_tensor_send; fix runtime too long in Recv Bwd; benchmark for llama + Hybrid(tp+pp);

  duanjunwen

  committed
  2eca112

  Copy full SHA for 2eca112

Commits on Oct 25, 2024

• [fix\ fix fail case test_shard_llama

  duanjunwen

  committed
  d0ec221

  Copy full SHA for d0ec221

• [fix] fix test_shard_llama

  duanjunwen

  committed
  cc0dfdd

  Copy full SHA for cc0dfdd

• [fix] fix llama modeling policy;

  duanjunwen

  committed
  03fa79a

  Copy full SHA for 03fa79a

Commits on Oct 28, 2024

• [fix] fix test_shard_llama ci;

  duanjunwen

  committed
  6377aa0

  Copy full SHA for 6377aa0

• [fix] fix test zerobubble

  duanjunwen

  committed
  5aee426

  Copy full SHA for 5aee426

Commits on Oct 29, 2024

• [fix] fix handle name; rm useless comments;

  duanjunwen

  committed
  fafe049

  Copy full SHA for fafe049

• [fix] fix send recv signature;

  duanjunwen

  committed
  fa3ccda

  Copy full SHA for fa3ccda

• [fix] fix comment in llama & benchmark

  duanjunwen

  committed
  982e4ee

  Copy full SHA for 982e4ee

Commits on Oct 30, 2024

• [feat] support no tensor parallel Linear in shardformer; Add test for use weightGradStore and not use WeightGradStore

  duanjunwen

  committed
  d2e05a9

  Copy full SHA for d2e05a9

Commits on Oct 31, 2024

• [fix] fix linear (no tp) ops func name;

  duanjunwen

  committed
  5f09243

  Copy full SHA for 5f09243

• [feat] support zbv in mixtral benchmark; (hpcaitech#6083)

  Show description for aed20fb
  duanjunwen

  authored
  aed20fb

  Copy full SHA for aed20fb

Commits on Nov 1, 2024

• Merge branch 'main' into dev/zero_bubble

  duanjunwen

  committed
  1d328ff

  Copy full SHA for 1d328ff

• Merge branch 'feature/zerobubble' of github.com:hpcaitech/ColossalAI into dev/zero_bubble

  duanjunwen

  committed
  c82c75a

  Copy full SHA for c82c75a

• [fix] fix fp8 args in HybridParallel

  duanjunwen

  committed
  3b5c314

  Copy full SHA for 3b5c314

• [fix] fix hybridparall use_fp8 config

  duanjunwen

  committed
  5b5fbcf

  Copy full SHA for 5b5fbcf

• [fix] fix use_fp8 flag

  duanjunwen

  committed
  0218e67

  Copy full SHA for 0218e67

• [fix] fix model zoo init

  duanjunwen

  committed
  8e40087

  Copy full SHA for 8e40087

Commits on Nov 5, 2024

• [feat] support no_tp Linear for sharderformer.llama

  duanjunwen

  committed
  4fc92aa

  Copy full SHA for 4fc92aa

Commits on Nov 6, 2024

• [fix] fix zbv llama pp4

  duanjunwen

  committed
  0d6d40c

  Copy full SHA for 0d6d40c

Commits on Nov 11, 2024

• [fix] fix send_tensor_metadata & send_grad_metadata;

  duanjunwen

  committed
  12919de

  Copy full SHA for 12919de

• [feat] fix testcase;

  duanjunwen

  committed
  337debc

  Copy full SHA for 337debc

Commits on Nov 12, 2024

• [feat] support mixtral policy with zbv tp_Linear & non_tp_Linear

  duanjunwen

  committed
  80b04d7

  Copy full SHA for 80b04d7

Commits on Nov 14, 2024

• [feat] update mixtral policy & bert policy for zerobubble

  duanjunwen

  committed
  b6d5e61

  Copy full SHA for b6d5e61

• [fix] fix p2p error in zbv

  duanjunwen

  committed
  1bc4dba

  Copy full SHA for 1bc4dba

• [fix] fix attn

  duanjunwen

  committed
  014afbd

  Copy full SHA for 014afbd

Commits on Nov 15, 2024

• [fix] fix mixtral modeling & policy; update wait handles; doing benchmarking for llama hybrid;

  duanjunwen

  committed
  5c2ebbf

  Copy full SHA for 5c2ebbf

• [fix] fix zbv wait_handle

  duanjunwen

  committed
  cf86c1b

  Copy full SHA for cf86c1b

• [fix] rm debug info; update llama policy; update wait handle

  duanjunwen

  committed
  0fb500c

  Copy full SHA for 0fb500c

• [fix] fix test_lora

  duanjunwen

  committed
  2980da5

  Copy full SHA for 2980da5

• [fix] fix test_lora in llama policy

  duanjunwen

  committed
  f48a85e

  Copy full SHA for f48a85e

Commits on Nov 18, 2024

• [fix] fix wait handle in run_fwd_bwd

  duanjunwen

  committed
  9a21f87

  Copy full SHA for 9a21f87

• [fix] remove debug info;

  duanjunwen

  committed
  dafda0f

  Copy full SHA for dafda0f

• [fix] rm unused comments

  duanjunwen

  committed
  41fdd21

  Copy full SHA for 41fdd21

• Merge branch 'main' into dev/zero_bubble

  duanjunwen

  committed
  cb9e5cc

  Copy full SHA for cb9e5cc

Commits on Nov 19, 2024

• [fix] fix fp8 overlap code

  duanjunwen

  committed
  8a0bad9

  Copy full SHA for 8a0bad9

• [fix] fix yml file & v_schedule comments

  duanjunwen

  committed
  9aa4c67

  Copy full SHA for 9aa4c67

• [fix] rm fwd only meta cache comments;

  duanjunwen

  committed
  e4488b1

  Copy full SHA for e4488b1