Code

Tensor Parallel#153

Code

Merged

zkh2016 merged 32 commits intoOpenBMB:devOpenBMB/BMTrain:devfrom zkh2016:tensor_parallelzkh2016/BMTrain:tensor_parallelCopy head branch name to clipboardAug 29, 2023

+804-191

Commits

Commits on Aug 22, 2023

• fix is_first_layer

  

  zhangkaihuo

  committed
  0d4ea37

  Copy full SHA for 0d4ea37

Commits on Aug 23, 2023

• tensor parallel

  

  zhangkaihuo

  committed
  3063afb

  Copy full SHA for 3063afb

• Merge branch 'fix_first_layer' into tensor_parallel

  

  zhangkaihuo

  committed
  bdc1ed9

  Copy full SHA for bdc1ed9

• rm unused code

  

  zhangkaihuo

  committed
  8648f5b

  Copy full SHA for 8648f5b

Commits on Aug 24, 2023

• refactor nccl group; remove partition_modules in pipe_layer.py

  

  zhangkaihuo

  committed
  763b408

  Copy full SHA for 763b408

• fix by review comment

  

  zhangkaihuo

  committed
  4c50567

  Copy full SHA for 4c50567

• fix topology

  

  zhangkaihuo

  committed
  825139c

  Copy full SHA for 825139c

• fix topology

  

  zhangkaihuo

  committed
  4ff0f41

  Copy full SHA for 4ff0f41

• fix

  

  zhangkaihuo

  committed
  a5d7ba6

  Copy full SHA for a5d7ba6

• use ParallelEmbedding

  

  zhangkaihuo

  committed
  2951d70

  Copy full SHA for 2951d70

• overlap parallel linear backward

  

  zhangkaihuo

  committed
  39319e1

  Copy full SHA for 39319e1

• add tp_comm_stream

  

  zhangkaihuo

  committed
  df3fd8f

  Copy full SHA for df3fd8f

• fix tp

  Achazwl

  committed
  99efba3

  Copy full SHA for 99efba3

• Merge branch 'tensor_parallel' into tp

  Achazwl

  authored
  85dd5ab

  Copy full SHA for 85dd5ab

• Merge pull request #1 from Achazwl/tp

  Show description for 76abcb4
  

  zhangkaihuo

  authored
  76abcb4

  Copy full SHA for 76abcb4

Commits on Aug 25, 2023

• fix load_state_dict

  

  zhangkaihuo

  committed
  f1b4fd7

  Copy full SHA for f1b4fd7

• test parallel linear

  

  zhangkaihuo

  committed
  677a316

  Copy full SHA for 677a316

• mv zero_level to CheckpointBlock

  

  zhangkaihuo

  committed
  743253e

  Copy full SHA for 743253e

• merge dev

  

  zhangkaihuo

  committed
  4e8c462

  Copy full SHA for 4e8c462

• fix overlap

  

  zhangkaihuo

  committed
  604ddfe

  Copy full SHA for 604ddfe

• gather once in atten

  

  zhangkaihuo

  committed
  0aee817

  Copy full SHA for 0aee817

• fix sub grad_input in parallel linear

  

  zhangkaihuo

  committed
  bd0bad0

  Copy full SHA for bd0bad0

Commits on Aug 26, 2023

• Merge branch 'dev' into tensor_parallel

  zkh2016

  authored
  50cdcaf

  Copy full SHA for 50cdcaf

• fix gather_output

  

  zhangkaihuo

  committed
  15460b6

  Copy full SHA for 15460b6

• Merge branch 'tensor_parallel' of https://github.com/zkh2016/BMTrain into tensor_parallel

  

  zhangkaihuo

  committed
  0e0e05c

  Copy full SHA for 0e0e05c

• fix train.py

  

  zhangkaihuo

  committed
  b44a62e

  Copy full SHA for b44a62e

• fused q,k,v

  

  zhangkaihuo

  committed
  100cd55

  Copy full SHA for 100cd55

• fix row parallel linear

  

  zhangkaihuo

  committed
  fa09468

  Copy full SHA for fa09468

• fix cross entropy

  

  zhangkaihuo

  committed
  37bc403

  Copy full SHA for 37bc403

Commits on Aug 28, 2023

• Update setup.py

  zkh2016

  authored
  15c2c48

  Copy full SHA for 15c2c48

• overlap send communication in pipeline

  

  zhangkaihuo

  committed
  42663c8

  Copy full SHA for 42663c8

Commits on Aug 29, 2023

• Merge branch 'tensor_parallel' of https://github.com/zkh2016/BMTrain into tensor_parallel

  

  zhangkaihuo

  committed
  207912b

  Copy full SHA for 207912b