[LLM] Add tools for parameters #9137
Conversation
|
Thanks for your contribution! |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## develop #9137 +/- ##
===========================================
- Coverage 53.34% 51.98% -1.36%
===========================================
Files 652 657 +5
Lines 105401 110297 +4896
===========================================
+ Hits 56222 57342 +1120
- Misses 49179 52955 +3776 ☔ View full report in Codecov by Sentry. |
|
emm，你们搞得太复杂了。safetensor本来就支持多文件直接加载，还支持mmap切分load。不需要这么麻烦的。 |
|
对safetensors格式的权重做TP切分，我们在 https://github.com/PaddlePaddle/PaddleNLP/blob/develop/paddlenlp/transformers/model_utils.py#L2770 这个函数中已经做了实现，直接调用该函数就行，不需要对safetensors合并后再拆分。 @Hanyonggong |
llm/tools/convert_gqa_to_mha.py
Outdated
| config_file = open(config_path, "r") | ||
| config_json = json.load(config_file) | ||
|
|
||
| model = paddle.load(gqa_model_path) |
There was a problem hiding this comment.
如果对于safetensors格式的权重，就不支持了？
llm/tools/merge_satetensors.py
Outdated
| @@ -0,0 +1,65 @@ | |||
| # Copyright (c) 2024 PaddlePaddle Authors. All Rights Reserved. | |||
| # | |||
llm/tools/split_weights.py
Outdated
|
|
||
| model_state_dict = paddle.load(model_path) | ||
|
|
||
| state_dict = model.convert_tensor_parallel( |
There was a problem hiding this comment.
There was a problem hiding this comment.
感觉你没有理解我的意思。我的意思是切分TP权重的这个函数，convert_tensor_parallel这个东西，不需要你自己调用了，前面的load_tp_checkpoint已经做了这个功能，拿到的state_dict就是切分后的。
|
|
PR types
Others
PR changes
Others
Description
add some llm tools: