New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tensor Parallel #153

Merged

zkh2016 merged 32 commits into OpenBMB:dev from zkh2016:tensor_parallel

Aug 29, 2023

Collaborator

zkh2016 commented Aug 23, 2023 •

edited

Loading

PR主要修改点：

添加tensor parallel模式：Support Tensor Parallel #149
修改topology以支持PP，TP，ZERO组合
修改parameter相关代码，适配TP模式
PP去除单独切分参数的逻辑，复用CheckpointBlock的参数切分
save/load适配TP模式

TODO：

优化linear，反向通信可以overlap

zhangkaihuo added 2 commits

August 22, 2023 15:35


          fix is_first_layer

0d4ea37


          tensor parallel

3063afb

zkh2016 requested a review from Achazwl

August 23, 2023 11:37

zhangkaihuo added 2 commits

August 23, 2023 20:05


          Merge branch 'fix_first_layer' into tensor_parallel

bdc1ed9


          rm unused code

8648f5b

Achazwl reviewed

View reviewed changes

bmtrain/nn/__init__.py Outdated Show resolved Hide resolved

bmtrain/nn/column_parallel_linear.py Outdated Show resolved Hide resolved

bmtrain/nn/embedding.py Outdated Show resolved Hide resolved

Achazwl reviewed

View reviewed changes

bmtrain/parameter.py Outdated Show resolved Hide resolved

Achazwl reviewed

View reviewed changes

bmtrain/param_init.py Outdated Show resolved Hide resolved

Achazwl reviewed

View reviewed changes

bmtrain/block_layer.py Outdated Show resolved Hide resolved

Achazwl reviewed

View reviewed changes

bmtrain/block_layer.py Outdated Show resolved Hide resolved

Achazwl reviewed

View reviewed changes

bmtrain/init.py Outdated Show resolved Hide resolved

Achazwl reviewed

View reviewed changes

bmtrain/init.py Outdated Show resolved Hide resolved

Achazwl reviewed

View reviewed changes

bmtrain/init.py Outdated Show resolved Hide resolved

zhangkaihuo added 3 commits

August 24, 2023 09:46


          refactor nccl group; remove partition_modules in pipe_layer.py

763b408


          fix by review comment

4c50567


          fix topology

825139c

zkh2016 changed the title ~~[WIP]Tensor Parallel~~ Tensor Parallel


          fix topology

4ff0f41

Achazwl requested changes

View reviewed changes

bmtrain/init.py Outdated Show resolved Hide resolved

zhangkaihuo and others added 11 commits

August 24, 2023 15:43

fix

a5d7ba6


          use ParallelEmbedding

2951d70


          overlap parallel linear backward

39319e1


          add tp_comm_stream

df3fd8f


          fix tp

99efba3


          Merge branch 'tensor_parallel' into tp

85dd5ab


          Merge pull request #1 from Achazwl/tp

76abcb4

Fix tp


          fix load_state_dict

f1b4fd7


          test parallel linear

677a316


          mv zero_level to CheckpointBlock

743253e


          merge dev

4e8c462

zhangkaihuo added 3 commits

August 25, 2023 16:53


          fix overlap

604ddfe


          gather once in atten

0aee817


          fix sub grad_input in parallel linear

bd0bad0

Achazwl reviewed

View reviewed changes

bmtrain/nn/parallel_linear_func.py Outdated Show resolved Hide resolved

zkh2016 and others added 10 commits

August 26, 2023 09:45


          Merge branch 'dev' into tensor_parallel

50cdcaf


          fix gather_output

15460b6


          Merge branch 'tensor_parallel' of https://github.com/zkh2016/BMTrain …

0e0e05c

…into tensor_parallel


          fix train.py

b44a62e


          fused q,k,v

100cd55


          fix row parallel linear

fa09468


          fix cross entropy

37bc403


          Update setup.py

15c2c48


          overlap send communication in pipeline

42663c8


          Merge branch 'tensor_parallel' of https://github.com/zkh2016/BMTrain …

207912b

…into tensor_parallel

Achazwl approved these changes

View reviewed changes

zkh2016 merged commit df43d6d into OpenBMB:dev

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet