Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix kunlunxin-glm training. #242

Merged
merged 3 commits into from
Sep 11, 2023
Merged

Fix kunlunxin-glm training. #242

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b62c34c
Fix kunlunxin GLM training configs
KungYork Sep 8, 2023
add614f
Relocate xacc install logic
KungYork Sep 8, 2023
2198cad
Modify max_steps for config 1x1 and 2x8
KungYork Sep 8, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions training/kunlunxin/docker_image/pytorch/pytorch_install.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,5 @@ set -xe

pip install https://bd.bcebos.com/klx-pytorch-ipipe-bd/flagperf/latest/xacc-0.1.0-cp38-cp38-linux_x86_64.whl
pip install https://bd.bcebos.com/klx-pytorch-ipipe-bd/flagperf/latest/xmlir-0.0.1-cp38-cp38-linux_x86_64.whl

python -m xacc.install
2 changes: 1 addition & 1 deletion training/kunlunxin/glm-pytorch/config/config_R300x1x1.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,4 @@
lr_decay_iters = 4338
log_freq = 1
seed = 4096
max_samples_termination = 925510
max_samples_termination = 4000
2 changes: 1 addition & 1 deletion training/kunlunxin/glm-pytorch/config/config_R300x2x8.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,4 @@
lr_decay_iters = 4338
log_freq = 1
seed = 4096
max_samples_termination = 2776540
max_samples_termination = 20000
2 changes: 2 additions & 0 deletions training/kunlunxin/glm-pytorch/config/environment_variables.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,3 +17,5 @@ export ALLREDUCE_FUSION=0

export XMLIR_F_XPU_FC_GEMM_MODE=float
export XMLIR_F_FAST_INDEX_PUT=true

export XACC_ENABLE=1
1 change: 1 addition & 0 deletions training/kunlunxin/glm-pytorch/config/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,4 @@ h5sparse
boto3
h5py
numpy>=1.15.4
psutil
5 changes: 3 additions & 2 deletions training/kunlunxin/glm-pytorch/extern/trainer_adapter.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
import torch
import config

from torch import nn
import torch.distributed as dist

import config
from optimizers import get_optimizer_param_groups
from optimizers.loss_scaler import DynamicLossScaler
from driver.dist_pytorch import main_proc_print

import torch_xmlir
import torch_xmlir.core.xpu_model as xm
from torch_xmlir.optimizer import AdamW as Adam
from torch_xmlir.nn.clip_grad import clip_grad_norm
Expand Down Expand Up @@ -79,4 +79,5 @@ def _clip_grad():
if DynamicLossScaler._has_inf_or_nan(reduced_loss):
main_proc_print("Found NaN loss, skip backward")

torch_xmlir.xpu.empty_cache()
return reduced_loss