关于lora微调训练问题

[tianyi25]

显示“没有找到Nvidia显卡”，显卡驱动是最新的，wsl也升级到了2

############################################################################

RWKV-5 BF16 on 1x1 GPU, bsz 1x1x1=1, deepspeed_stage_2

Data = ./finetune/json2binidx_tool/data/sample_text_document (binidx), ProjDir = lora-models

Epoch = 0 to 19, save every 1 epoch

Each "epoch" = 200 steps, 200 samples, 30000 tokens

Model = 24 n_layer, 2048 n_embd, 150 ctx_len

Adam = lr 5e-05 to 5e-05, warmup 0 steps, beta (0.9, 0.999), eps 1e-08

Found torch 2.2.0+cu121, recommend 1.13.1+cu117 or newer

Found deepspeed 0.12.6, recommend 0.7.0 (faster than newer versions)

Found pytorch_lightning 1.9.5, recommend 1.9.5

############################################################################
INFO:pytorch_lightning.utilities.rank_zero:{'load_model': 'models/RWKV-x060-World-3B-v2-20240228-ctx4096.pth', 'wandb': '', 'proj_dir': 'lora-models', 'random_seed': -1, 'data_file': './finetune/json2binidx_tool/data/sample_text_document', 'data_type': 'binidx', 'vocab_size': 65536, 'ctx_len': 150, 'epoch_steps': 200, 'epoch_count': 20, 'epoch_begin': 0, 'epoch_save': 1, 'micro_bsz': 1, 'n_layer': 24, 'n_embd': 2048, 'dim_att': 2048, 'dim_ffn': 7168, 'pre_ffn': 0, 'head_qk': 0, 'tiny_att_dim': 0, 'tiny_att_layer': -999, 'lr_init': 5e-05, 'lr_final': 5e-05, 'warmup_steps': 0, 'beta1': 0.9, 'beta2': 0.999, 'adam_eps': 1e-08, 'grad_cp': 0, 'dropout': 0, 'weight_decay': 0, 'weight_decay_final': -1, 'my_pile_version': 1, 'my_pile_stage': 0, 'my_pile_shift': -1, 'my_pile_edecay': 0, 'layerwise_lr': 1, 'ds_bucket_mb': 2, 'my_sample_len': 0, 'my_ffn_shift': 1, 'my_att_shift': 1, 'head_size_a': 64, 'head_size_divisor': 8, 'my_pos_emb': 0, 'load_partial': 0, 'magic_prime': 0, 'my_qa_mask': 0, 'my_random_steps': 0, 'my_testing': '', 'my_exit': 99999999, 'my_exit_tokens': 0, 'emb': False, 'lora': True, 'lora_load': '', 'lora_r': 8, 'lora_alpha': 32.0, 'lora_dropout': 0.01, 'lora_parts': 'att,ffn,time,ln', 'logger': False, 'enable_checkpointing': False, 'default_root_dir': None, 'gradient_clip_val': 1.0, 'gradient_clip_algorithm': None, 'num_nodes': 1, 'num_processes': None, 'devices': '1', 'gpus': None, 'auto_select_gpus': None, 'tpu_cores': None, 'ipus': None, 'enable_progress_bar': True, 'overfit_batches': 0.0, 'track_grad_norm': -1, 'check_val_every_n_epoch': 100000000000000000000, 'fast_dev_run': False, 'accumulate_grad_batches': 8, 'max_epochs': 20, 'min_epochs': None, 'max_steps': -1, 'min_steps': None, 'max_time': None, 'limit_train_batches': None, 'limit_val_batches': None, 'limit_test_batches': None, 'limit_predict_batches': None, 'val_check_interval': None, 'log_every_n_steps': 100000000000000000000, 'accelerator': 'gpu', 'strategy': 'deepspeed_stage_2', 'sync_batchnorm': False, 'precision': 'bf16', 'enable_model_summary': True, 'num_sanity_val_steps': 0, 'resume_from_checkpoint': None, 'profiler': None, 'benchmark': None, 'reload_dataloaders_every_n_epochs': 0, 'auto_lr_find': False, 'replace_sampler_ddp': False, 'detect_anomaly': False, 'auto_scale_batch_size': False, 'plugins': None, 'amp_backend': None, 'amp_level': None, 'move_metrics_to_cpu': False, 'multiple_trainloader_mode': 'max_size_cycle', 'inference_mode': True, 'my_timestamp': '2025-01-25-14-28-40', 'betas': (0.9, 0.999), 'real_bsz': 1, 'run_name': '65536 ctx150 L24 D2048'}
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
Using /root/.cache/torch_extensions/py310_cu121 as PyTorch extensions root...
Creating extension directory /root/.cache/torch_extensions/py310_cu121/wkv5...
Detected CUDA files, patching ldflags
Emitting ninja build file /root/.cache/torch_extensions/py310_cu121/wkv5/build.ninja...
RWKV_MY_TESTING
Traceback (most recent call last):
File "/mnt/d/rwkv/./finetune/lora/v5/train.py", line 308, in
from src.trainer import train_callback, generate_init_weight
File "/mnt/d/rwkv/finetune/lora/v5/src/trainer.py", line 6, in
from .model import LORA_CONFIG
File "/mnt/d/rwkv/finetune/lora/v5/src/model.py", line 56, in
wkv5_cuda = load(
File "/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py", line 1306, in load
return _jit_compile(
File "/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py", line 1710, in _jit_compile
_write_ninja_file_and_build_library(
File "/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py", line 1810, in _write_ninja_file_and_build_library
_write_ninja_file_to_build_library(
File "/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py", line 2195, in _write_ninja_file_to_build_library
cuda_flags = common_cflags + COMMON_NVCC_FLAGS + _get_cuda_arch_flags()
File "/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py", line 1976, in _get_cuda_arch_flags
arch_list[-1] += '+PTX'
IndexError: list index out of range