support transfer llama hf weight to megatron weight

[uygnef]

Hi there,

I hope this message finds you well. I would like to request the availability of the pretrained checkpoint for the pretrain and SFT stages of the project. Currently, there is no corresponding checkpoint available for llama2 in the Megatron repository.

To address this issue, I have modify a script from that facilitates the conversion from hf (Hugging Face) format to Megatron format. This script will enable the usage of llama2's pretrained checkpoint in the Megatron framework.

Please let me know if there are any further steps required or if you need any additional information from my end to proceed with this request.

Thank you for your attention and assistance.

Best regards,

[uygnef]

@microsoft-github-policy-service agree

[uygnef]

there must be some better solution to init model without init distibute group. please help me ..

[tjruwase]

The distributed initialization only occurs for args.zero_stage==3. Have you tried with different stage value on command line?

[uygnef]

The distributed initialization only occurs for args.zero_stage==3. Have you tried with different stage value on command line?

The problem is mpu.get_sequence_data_parallel_group(). How can I solve this problem?

  File "/mnt/megatron-deepspeed/pretrain_gpt.py", line 48, in model_provider
    with deepspeed.zero.Init(sequence_data_parallel_group=mpu.get_sequence_data_parallel_group(),
  File "/mnt/megatron-deepspeed/megatron/core/parallel_state.py", line 369, in get_sequence_data_parallel_group
    assert _SEQUENCE_DATA_PARALLEL_GROUP is not None, \
AssertionError: sequence data parallel group is not initialized

[conglongli]

@uygnef it seems that you are still working on this PR? When you finish, please ping my teammate @tjruwase who agreed to review your PR.

[uygnef]

hi @tjruwase. The code has been completed. Could you please take some time to review this pull request?

[cdj0311]

How to convert megatron model to deepspeed?

[tjruwase]

@uygnef, thanks for the PR. Will review now.

[uygnef]

hello @tjruwase
I have made the necessary changes. Please review it whenever you have the time.

[uygnef]

How to convert megatron model to deepspeed?
@cdj0311 this link might help you
https://github.com/uygnef/Megatron-DeepSpeed/blob/main/tools/convert_checkpoint/README.md

[cdj0311]

How to convert megatron model to deepspeed?
@cdj0311 this link might help you
https://github.com/uygnef/Megatron-DeepSpeed/blob/main/tools/convert_checkpoint/README.md

hi,
I convert megatron to deepspeed with
python3 tools/checkpoint_util.py
--target-tensor-parallel-size 4
--target-pipeline-parallel-size 2
--load-dir /path/to/Megatron-Deepspeed/checkpoint/
--save-dir /path/to/Megatron-Deepspeed/distribute_checkpoint/
--model-type GPT
but get an error:

File "tools/checkpoint_util.py", line 149, in main
    loader.load_checkpoint(queue, args)
  File "/ossfs/workspace/LLaMA2/Megatron-DeepSpeed-LLaMa2-v3/tools/checkpoint_loader_megatron.py", line 340, in load_checkpoint
    _load_checkpoint(queue, args)
  File "/ossfs/workspace/LLaMA2/Megatron-DeepSpeed-LLaMa2-v3/tools/checkpoint_loader_megatron.py", line 205, in _load_checkpoint
    all_models = [get_models(tp_size, md.params_dtype)]
  File "/ossfs/workspace/LLaMA2/Megatron-DeepSpeed-LLaMa2-v3/tools/checkpoint_loader_megatron.py", line 141, in get_models
    load_checkpoint(model_, None, None)
  File "/ossfs/workspace/LLaMA2/Megatron-DeepSpeed-LLaMa2-v3/megatron/checkpointing.py", line 610, in load_checkpoint
    model[0].load_state_dict(state_dict['model'], strict=strict)
  File "/ossfs/workspace/LLaMA2/Megatron-DeepSpeed-LLaMa2-v3/megatron/model/gpt_model.py", line 170, in load_state_dict
    self.language_model.load_state_dict(state_dict, strict=strict)
  File "/ossfs/workspace/LLaMA2/Megatron-DeepSpeed-LLaMa2-v3/megatron/model/language_model.py", line 691, in load_state_dict
    self.encoder.load_state_dict(state_dict_, strict=strict)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 2041, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for ParallelTransformer:
        Missing key(s) in state_dict: "layers.0.self_attention.query.weight", "layers.0.self_attention.key_value.weight", "layers.1.self_attention.query.weight", "layers.1.self_attention.key_value.weight", "layers.2.self_attention.query.weight", "layers.2.self_attention.key_value.weight", "layers.3.self_attention.query.weight", "layers.3.self_attention.key_value.weight", "layers.4.self_attention.query.weight", "layers.4.self_attention.key_value.weight", "layers.5.self_attention.query.weight", "layers.5.self_attention.key_value.weight", "layers.6.self_attention.query.weight", "layers.6.self_attention.key_value.weight", "layers.7.self_attention.query.weight", "layers.7.self_attention.key_value.weight", "layers.8.self_attention.query.weight", "layers.8.self_attention.key_value.weight", "layers.9.self_attention.query.weight", "layers.9.self_attention.key_value.weight", "layers.10.self_attention.query.weight", "layers.10.self_attention.key_value.weight", "layers.11.self_attention.query.weight", "layers.11.self_attention.key_value.weight", "layers.12.self_attention.query.weight", "layers.12.self_attention.key_value.weight", "layers.13.self_attention.query.weight", "layers.13.self_attention.key_value.weight", "layers.14.self_attention.query.weight", "layers.14.self_attention.key_value.weight", "layers.15.self_attention.query.weight", "layers.15.self_attention.key_value.weight", "layers.16.self_attention.query.weight", "layers.16.self_attention.key_value.weight", "layers.17.self_attention.query.weight", "layers.17.self_attention.key_value.weight", "layers.18.self_attention.query.weight", "layers.18.self_attention.key_value.weight", "layers.19.self_attention.query.weight", "layers.19.self_attention.key_value.weight", "layers.20.self_attention.query.weight", "layers.20.self_attention.key_value.weight", "layers.21.self_attention.query.weight", "layers.21.self_attention.key_value.weight", "layers.22.self_attention.query.weight", "layers.22.self_attention.key_value.weight", "layers.23.self_attention.query.weight", "layers.23.self_attention.key_value.weight", "layers.24.self_attention.query.weight", "layers.24.self_attention.key_value.weight", "layers.25.self_attention.query.weight", "layers.25.self_attention.key_value.weight", "layers.26.self_attention.query.weight", "layers.26.self_attention.key_value.weight", "layers.27.self_attention.query.weight", "layers.27.self_attention.key_value.weight", "layers.28.self_attention.query.weight", "layers.28.self_attention.key_value.weight", "layers.29.self_attention.query.weight", "layers.29.self_attention.key_value.weight", "layers.30.self_attention.query.weight", "layers.30.self_attention.key_value.weight", "layers.31.self_attention.query.weight", "layers.31.self_attention.key_value.weight", "layers.32.self_attention.query.weight", "layers.32.self_attention.key_value.weight", "layers.33.self_attention.query.weight", "layers.33.self_attention.key_value.weight", "layers.34.self_attention.query.weight", "layers.34.self_attention.key_value.weight", "layers.35.self_attention.query.weight", "layers.35.self_attention.key_value.weight", "layers.36.self_attention.query.weight", "layers.36.self_attention.key_value.weight", "layers.37.self_attention.query.weight", "layers.37.self_attention.key_value.weight", "layers.38.self_attention.query.weight", "layers.38.self_attention.key_value.weight", "layers.39.self_attention.query.weight", "layers.39.self_attention.key_value.weight", "layers.40.self_attention.query.weight", "layers.40.self_attention.key_value.weight", "layers.41.self_attention.query.weight", "layers.41.self_attention.key_value.weight", "layers.42.self_attention.query.weight", "layers.42.self_attention.key_value.weight", "layers.43.self_attention.query.weight", "layers.43.self_attention.key_value.weight", "layers.44.self_attention.query.weight", "layers.44.self_attention.key_value.weight", "layers.45.self_attention.query.weight", "layers.45.self_attention.key_value.weight", "layers.46.self_attention.query.weight", "layers.46.self_attention.key_value.weight", "layers.47.self_attention.query.weight", "layers.47.self_attention.key_value.weight". 
        Unexpected key(s) in state_dict: "layers.0.self_attention.query_key_value.weight", "layers.1.self_attention.query_key_value.weight", "layers.2.self_attention.query_key_value.weight", "layers.3.self_attention.query_key_value.weight", "layers.4.self_attention.query_key_value.weight", "layers.5.self_attention.query_key_value.weight", "layers.6.self_attention.query_key_value.weight", "layers.7.self_attention.query_key_value.weight", "layers.8.self_attention.query_key_value.weight", "layers.9.self_attention.query_key_value.weight", "layers.10.self_attention.query_key_value.weight", "layers.11.self_attention.query_key_value.weight", "layers.12.self_attention.query_key_value.weight", "layers.13.self_attention.query_key_value.weight", "layers.14.self_attention.query_key_value.weight", "layers.15.self_attention.query_key_value.weight", "layers.16.self_attention.query_key_value.weight", "layers.17.self_attention.query_key_value.weight", "layers.18.self_attention.query_key_value.weight", "layers.19.self_attention.query_key_value.weight", "layers.20.self_attention.query_key_value.weight", "layers.21.self_attention.query_key_value.weight", "layers.22.self_attention.query_key_value.weight", "layers.23.self_attention.query_key_value.weight", "layers.24.self_attention.query_key_value.weight", "layers.25.self_attention.query_key_value.weight", "layers.26.self_attention.query_key_value.weight", "layers.27.self_attention.query_key_value.weight", "layers.28.self_attention.query_key_value.weight", "layers.29.self_attention.query_key_value.weight", "layers.30.self_attention.query_key_value.weight", "layers.31.self_attention.query_key_value.weight", "layers.32.self_attention.query_key_value.weight", "layers.33.self_attention.query_key_value.weight", "layers.34.self_attention.query_key_value.weight", "layers.35.self_attention.query_key_value.weight", "layers.36.self_attention.query_key_value.weight", "layers.37.self_attention.query_key_value.weight", "layers.38.self_attention.query_key_value.weight", "layers.39.self_attention.query_key_value.weight", "layers.40.self_attention.query_key_value.weight", "layers.41.self_attention.query_key_value.weight", "layers.42.self_attention.query_key_value.weight", "layers.43.self_attention.query_key_value.weight", "layers.44.self_attention.query_key_value.weight", "layers.45.self_attention.query_key_value.weight", "layers.46.self_attention.query_key_value.weight", "layers.47.self_attention.query_key_value.weight". 

[uygnef]

How to convert megatron model to deepspeed?
@cdj0311 this link might help you
https://github.com/uygnef/Megatron-DeepSpeed/blob/main/tools/convert_checkpoint/README.md

hi, I convert megatron to deepspeed with python3 tools/checkpoint_util.py --target-tensor-parallel-size 4 --target-pipeline-parallel-size 2 --load-dir /path/to/Megatron-Deepspeed/checkpoint/ --save-dir /path/to/Megatron-Deepspeed/distribute_checkpoint/ --model-type GPT but get an error:

File "tools/checkpoint_util.py", line 149, in main
    loader.load_checkpoint(queue, args)
  File "/ossfs/workspace/LLaMA2/Megatron-DeepSpeed-LLaMa2-v3/tools/checkpoint_loader_megatron.py", line 340, in load_checkpoint
    _load_checkpoint(queue, args)
  File "/ossfs/workspace/LLaMA2/Megatron-DeepSpeed-LLaMa2-v3/tools/checkpoint_loader_megatron.py", line 205, in _load_checkpoint
    all_models = [get_models(tp_size, md.params_dtype)]
  File "/ossfs/workspace/LLaMA2/Megatron-DeepSpeed-LLaMa2-v3/tools/checkpoint_loader_megatron.py", line 141, in get_models
    load_checkpoint(model_, None, None)
  File "/ossfs/workspace/LLaMA2/Megatron-DeepSpeed-LLaMa2-v3/megatron/checkpointing.py", line 610, in load_checkpoint
    model[0].load_state_dict(state_dict['model'], strict=strict)
  File "/ossfs/workspace/LLaMA2/Megatron-DeepSpeed-LLaMa2-v3/megatron/model/gpt_model.py", line 170, in load_state_dict
    self.language_model.load_state_dict(state_dict, strict=strict)
  File "/ossfs/workspace/LLaMA2/Megatron-DeepSpeed-LLaMa2-v3/megatron/model/language_model.py", line 691, in load_state_dict
    self.encoder.load_state_dict(state_dict_, strict=strict)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 2041, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for ParallelTransformer:
        Missing key(s) in state_dict: "layers.0.self_attention.query.weight", "layers.0.self_attention.key_value.weight", "layers.1.self_attention.query.weight", "layers.1.self_attention.key_value.weight", "layers.2.self_attention.query.weight", "layers.2.self_attention.key_value.weight", "layers.3.self_attention.query.weight", "layers.3.self_attention.key_value.weight", "layers.4.self_attention.query.weight", "layers.4.self_attention.key_value.weight", "layers.5.self_attention.query.weight", "layers.5.self_attention.key_value.weight", "layers.6.self_attention.query.weight", "layers.6.self_attention.key_value.weight", "layers.7.self_attention.query.weight", "layers.7.self_attention.key_value.weight", "layers.8.self_attention.query.weight", "layers.8.self_attention.key_value.weight", "layers.9.self_attention.query.weight", "layers.9.self_attention.key_value.weight", "layers.10.self_attention.query.weight", "layers.10.self_attention.key_value.weight", "layers.11.self_attention.query.weight", "layers.11.self_attention.key_value.weight", "layers.12.self_attention.query.weight", "layers.12.self_attention.key_value.weight", "layers.13.self_attention.query.weight", "layers.13.self_attention.key_value.weight", "layers.14.self_attention.query.weight", "layers.14.self_attention.key_value.weight", "layers.15.self_attention.query.weight", "layers.15.self_attention.key_value.weight", "layers.16.self_attention.query.weight", "layers.16.self_attention.key_value.weight", "layers.17.self_attention.query.weight", "layers.17.self_attention.key_value.weight", "layers.18.self_attention.query.weight", "layers.18.self_attention.key_value.weight", "layers.19.self_attention.query.weight", "layers.19.self_attention.key_value.weight", "layers.20.self_attention.query.weight", "layers.20.self_attention.key_value.weight", "layers.21.self_attention.query.weight", "layers.21.self_attention.key_value.weight", "layers.22.self_attention.query.weight", "layers.22.self_attention.key_value.weight", "layers.23.self_attention.query.weight", "layers.23.self_attention.key_value.weight", "layers.24.self_attention.query.weight", "layers.24.self_attention.key_value.weight", "layers.25.self_attention.query.weight", "layers.25.self_attention.key_value.weight", "layers.26.self_attention.query.weight", "layers.26.self_attention.key_value.weight", "layers.27.self_attention.query.weight", "layers.27.self_attention.key_value.weight", "layers.28.self_attention.query.weight", "layers.28.self_attention.key_value.weight", "layers.29.self_attention.query.weight", "layers.29.self_attention.key_value.weight", "layers.30.self_attention.query.weight", "layers.30.self_attention.key_value.weight", "layers.31.self_attention.query.weight", "layers.31.self_attention.key_value.weight", "layers.32.self_attention.query.weight", "layers.32.self_attention.key_value.weight", "layers.33.self_attention.query.weight", "layers.33.self_attention.key_value.weight", "layers.34.self_attention.query.weight", "layers.34.self_attention.key_value.weight", "layers.35.self_attention.query.weight", "layers.35.self_attention.key_value.weight", "layers.36.self_attention.query.weight", "layers.36.self_attention.key_value.weight", "layers.37.self_attention.query.weight", "layers.37.self_attention.key_value.weight", "layers.38.self_attention.query.weight", "layers.38.self_attention.key_value.weight", "layers.39.self_attention.query.weight", "layers.39.self_attention.key_value.weight", "layers.40.self_attention.query.weight", "layers.40.self_attention.key_value.weight", "layers.41.self_attention.query.weight", "layers.41.self_attention.key_value.weight", "layers.42.self_attention.query.weight", "layers.42.self_attention.key_value.weight", "layers.43.self_attention.query.weight", "layers.43.self_attention.key_value.weight", "layers.44.self_attention.query.weight", "layers.44.self_attention.key_value.weight", "layers.45.self_attention.query.weight", "layers.45.self_attention.key_value.weight", "layers.46.self_attention.query.weight", "layers.46.self_attention.key_value.weight", "layers.47.self_attention.query.weight", "layers.47.self_attention.key_value.weight". 
        Unexpected key(s) in state_dict: "layers.0.self_attention.query_key_value.weight", "layers.1.self_attention.query_key_value.weight", "layers.2.self_attention.query_key_value.weight", "layers.3.self_attention.query_key_value.weight", "layers.4.self_attention.query_key_value.weight", "layers.5.self_attention.query_key_value.weight", "layers.6.self_attention.query_key_value.weight", "layers.7.self_attention.query_key_value.weight", "layers.8.self_attention.query_key_value.weight", "layers.9.self_attention.query_key_value.weight", "layers.10.self_attention.query_key_value.weight", "layers.11.self_attention.query_key_value.weight", "layers.12.self_attention.query_key_value.weight", "layers.13.self_attention.query_key_value.weight", "layers.14.self_attention.query_key_value.weight", "layers.15.self_attention.query_key_value.weight", "layers.16.self_attention.query_key_value.weight", "layers.17.self_attention.query_key_value.weight", "layers.18.self_attention.query_key_value.weight", "layers.19.self_attention.query_key_value.weight", "layers.20.self_attention.query_key_value.weight", "layers.21.self_attention.query_key_value.weight", "layers.22.self_attention.query_key_value.weight", "layers.23.self_attention.query_key_value.weight", "layers.24.self_attention.query_key_value.weight", "layers.25.self_attention.query_key_value.weight", "layers.26.self_attention.query_key_value.weight", "layers.27.self_attention.query_key_value.weight", "layers.28.self_attention.query_key_value.weight", "layers.29.self_attention.query_key_value.weight", "layers.30.self_attention.query_key_value.weight", "layers.31.self_attention.query_key_value.weight", "layers.32.self_attention.query_key_value.weight", "layers.33.self_attention.query_key_value.weight", "layers.34.self_attention.query_key_value.weight", "layers.35.self_attention.query_key_value.weight", "layers.36.self_attention.query_key_value.weight", "layers.37.self_attention.query_key_value.weight", "layers.38.self_attention.query_key_value.weight", "layers.39.self_attention.query_key_value.weight", "layers.40.self_attention.query_key_value.weight", "layers.41.self_attention.query_key_value.weight", "layers.42.self_attention.query_key_value.weight", "layers.43.self_attention.query_key_value.weight", "layers.44.self_attention.query_key_value.weight", "layers.45.self_attention.query_key_value.weight", "layers.46.self_attention.query_key_value.weight", "layers.47.self_attention.query_key_value.weight". 

Do you need transfer it to hf ckpt? this script can help you. https://github.com/epfLLM/Megatron-LLM/blob/main/weights_conversion/megatron_to_hf.py Some weight name should be change

[tjruwase]

I did not notice --ckpt_transfer in the argument parsing code. How is this attribute added to args?

[uygnef]

When the ckpt splitting program loads the model, it actually doesn't initialize the parallel_state, so running parallel_state.get_sequence_parallel_world_size() will cause an error.

  File "/mnt/megatron-deepspeed/megatron/core/parallel_state.py", line 362, in get_sequence_parallel_group
    assert _SEQUENCE_PARALLEL_GROUP is not None, \
AssertionError: sequence parallel group is not initialized

Therefore, I used ckpt_transfer to skip getting get_sequence_parallel_world_size.
I also think this modification is not good, do you have any suggestions?

[uygnef]

I did not notice --ckpt_transfer in the argument parsing code. How is this attribute added to args?

I understand that you are likely busy with many responsibilities, but I would greatly appreciate your feedback on this PR when you get a chance.

[SefaZeng]

I did not notice --ckpt_transfer in the argument parsing code. How is this attribute added to args?

I understand that you are likely busy with many responsibilities, but I would greatly appreciate your feedback on this PR when you get a chance.

Hi, @uygnef , thank you for great your work! I am trying to use this script for convert HF LLAMA to Megatron-Deepspeed format and I met the same error AssertionError: sequence parallel group is not initialized. Do you solve this issue?

[SefaZeng]

I did not notice --ckpt_transfer in the argument parsing code. How is this attribute added to args?

I understand that you are likely busy with many responsibilities, but I would greatly appreciate your feedback on this PR when you get a chance.

Hi, @uygnef, I changed ckpt_transfer parameter so it works. But it seems the output format is Megatron-LM format not Megatron-DeepSpeed format?

[inkcherry]

Hi @uygnef, thank you so much for this pr! Would it be possible for you to provide an example of a launch script(pretrain or finetune) for it?

[zdaiot]

@SefaZeng @cdj0311 Hello, have you solved it?