When use one Gpu do model training, met one issue.

[yxy123]

bash training/finetune_RedPajama-INCITE-Chat-3B-v1.sh
My configurations changes as below:
--lr 1e-5 --seq-length 2048 --batch-size 8 --micro-batch-size 1 --gradient-accumulate-step 1
--num-layers 2 --embedding-dim 2560
--world-size 1 --pipeline-group-size 1 --data-group-size 1
(trap 'kill 0' SIGINT;
python ${DIR}/dist_clm_train.py $(echo ${ARGS}) --cuda-id 0 --rank 0
&
My environment:
GPU: NVIDIA RTX A4000
Graphics card memory:16 GB
Number of CPUs available for use: 8
Memory : 60 GB
Free space: 200 GB

error log:
Rank 0 node forward pass 0/1 takes 1.84s
{'loss': 16.892578125, 'lr': 1e-05}
Rank 0 node backward pass 0/1 takes 1.09s
cuda:0 cuda:0 cuda:0
!!! Warning: find inf in fp16 optimizer-step() !!!
/root/miniconda3/envs/myconda/lib/python3.8/site-packages/torch/optim/lr_scheduler.py:138: UserWarning: Detected call of lr_scheduler.step() before optimizer.step(). In PyTorch 1.1.0 and later, you should call them in the opposite order: optimizer.step() before lr_scheduler.step(). Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
warnings.warn("Detected call of lr_scheduler.step() before optimizer.step(). "
after cuda sync 0
Rank 0 node optimizer step takes 0.07s
Rank 0 node whole iteration takes 3.00s

Rank 0 node forward pass 0/1 takes 0.36s
{'loss': 16.744140625, 'lr': 9e-06}

With this error log, it seems also produce one model locate in model_ckpts/redpajama-incite-chat-3b-sample/checkpoint_10/, I don't if it's model is right?

[yxy123]

And then execute converting weights to Huggingface format, met the dimension and target size of the extension operation do not match.
(myconda) root@ZaodV6:/mnt/tet/OpenChatKit-main# mkdir huggingface_models \

&& python tools/convert_to_hf_gptneox.py
--config-name EleutherAI/pythia-6.9b-deduped
--ckpt-path model_ckpts/redpajama-incite-chat-3b-sample/checkpoint_10
--save-path huggingface_models/Pythia-Chat-Base-3B
--n-stages 4
--n-layer-per-stage 8
--fp16
loading config...
loaded config.
loading tokenizer...
loaded tokenizer.
creating empty model...
created empty model.
loading model ckpt...
loading stage 0
Traceback (most recent call last):
File "tools/convert_to_hf_gptneox.py", line 123, in
load_decentralized_checkpoint(
File "tools/convert_to_hf_gptneox.py", line 48, in load_decentralized_checkpoint
model.gpt_neox.embed_in.weight.data[:] = _tmp['embed_in.weight']
RuntimeError: The expanded size of the tensor (4096) must match the existing size (2560) at non-singleton dimension 1. Target sizes: [50432, 4096]. Tensor sizes: [50432, 2560]

[ChengYen-Tang]

bash training/finetune_RedPajama-INCITE-Chat-3B-v1.sh My configurations changes as below: --lr 1e-5 --seq-length 2048 --batch-size 8 --micro-batch-size 1 --gradient-accumulate-step 1 --num-layers 2 --embedding-dim 2560 --world-size 1 --pipeline-group-size 1 --data-group-size 1 (trap 'kill 0' SIGINT; python DIR/distclmtrain.py(echo ${ARGS}) --cuda-id 0 --rank 0 & My environment: GPU: NVIDIA RTX A4000 Graphics card memory:16 GB Number of CPUs available for use: 8 Memory : 60 GB Free space: 200 GB

error log:

Rank 0 node forward pass 0/1 takes 1.84s
{'loss': 16.892578125, 'lr': 1e-05}
Rank 0 node backward pass 0/1 takes 1.09s
cuda:0 cuda:0 cuda:0
!!! Warning: find inf in fp16 optimizer-step() !!!
/root/miniconda3/envs/myconda/lib/python3.8/site-packages/torch/optim/lr_scheduler.py:138: UserWarning: Detected call of before . In PyTorch 1.1.0 and later, you should call them in the opposite order: before . Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
warnings.warn("Detected call of before . "
after cuda sync 0
Rank 0 node optimizer step takes 0.07s
Rank 0 node whole iteration takes 3.00slr_scheduler.step()``optimizer.step()``optimizer.step()``lr_scheduler.step()``lr_scheduler.step()``optimizer.step()
Rank 0 node forward pass 0/1 takes 0.36s {'loss': 16.744140625, 'lr': 9e-06}

With this error log, it seems also produce one model locate in model_ckpts/redpajama-incite-chat-3b-sample/checkpoint_10/, I don't if it's model is right?

This model has 32 layers, so if you only have one GPU, --num-layers must be 32
https://huggingface.co/togethercomputer/RedPajama-INCITE-Chat-3B-v1/blob/main/config.json#L16

[ChengYen-Tang]

And then execute converting weights to Huggingface format, met the dimension and target size of the extension operation do not match. (myconda) root@ZaodV6:/mnt/tet/OpenChatKit-main# mkdir huggingface_models \

&& python tools/convert_to_hf_gptneox.py
--config-name EleutherAI/pythia-6.9b-deduped
--ckpt-path model_ckpts/redpajama-incite-chat-3b-sample/checkpoint_10
--save-path huggingface_models/Pythia-Chat-Base-3B
--n-stages 4
--n-layer-per-stage 8
--fp16
loading config...
loaded config.
loading tokenizer...
loaded tokenizer.
creating empty model...
created empty model.
loading model ckpt...
loading stage 0
Traceback (most recent call last):
File "tools/convert_to_hf_gptneox.py", line 123, in
load_decentralized_checkpoint(
File "tools/convert_to_hf_gptneox.py", line 48, in load_decentralized_checkpoint
model.gpt_neox.embed_in.weight.data[:] = _tmp['embed_in.weight']
RuntimeError: The expanded size of the tensor (4096) must match the existing size (2560) at non-singleton dimension 1. Target sizes: [50432, 4096]. Tensor sizes: [50432, 2560]

#86 (comment)