RuntimeError: mat1 and mat2 shapes cannot be multiplied (1024x5 and 4x4096)

[thelongestusernameofall]

╰─$ python finetune.py

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: /home/jovyan/simon-data/home/miniconda3/envs/alpaca39 did not contain libcudart.so as expected! Searching further paths...
warn(msg)
/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//simon-gpt-0'), PosixPath('8888/notebook/prdsafe/simon-gpt'), PosixPath('http')}
warn(msg)
/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/home/jovyan/.local/share/autojump/errors.log')}
warn(msg)
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 7.0
CUDA SETUP: Detected CUDA version 111
/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: Compute capability < 7.5 detected! Only slow 8-bit matmul is supported for your GPU!
warn(msg)
CUDA SETUP: Loading binary /home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda111_nocublaslt.so...
Overriding torch_dtype=None with torch_dtype=torch.float16 due to requirements of bitsandbytes to enable model loading in mixed int8. Either pass torch_dtype=torch.float16 or don't pass this argument at all to remove this warning.
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 33/33 [03:23<00:00, 6.16s/it]
Found cached dataset json (/home/jovyan/.cache/huggingface/datasets/json/default-b21b7af1932b77a9/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00, 1.68s/it]
Loading cached split indices for dataset at /home/jovyan/.cache/huggingface/datasets/json/default-b21b7af1932b77a9/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51/cache-514db5b55ef8a533.arrow and /home/jovyan/.cache/huggingface/datasets/json/default-b21b7af1932b77a9/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51/cache-a2d8efccb09d7efe.arrow
/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/transformers/optimization.py:391: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set no_deprecation_warning=True to disable this warning
warnings.warn(
0%| | 0/585 [00:00<?, ?it/s]Traceback (most recent call last):
File "/home/jovyan/simon-data/research/AI/gpt/facebook/japanese-alpaca-lora/finetune.py", line 118, in
trainer.train()
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/transformers/trainer.py", line 1628, in train
return inner_training_loop(
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/transformers/trainer.py", line 1895, in _inner_training_loop
tr_loss_step = self.training_step(model, inputs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/transformers/trainer.py", line 2637, in training_step
loss = self.compute_loss(model, inputs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/transformers/trainer.py", line 2669, in compute_loss
outputs = model(**inputs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/nn/parallel/data_parallel.py", line 171, in forward
outputs = self.parallel_apply(replicas, inputs, kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/nn/parallel/data_parallel.py", line 181, in parallel_apply
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/nn/parallel/parallel_apply.py", line 89, in parallel_apply
output.reraise()
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/_utils.py", line 644, in reraise
raise exception
RuntimeError: Caught RuntimeError in replica 1 on device 1.
Original Traceback (most recent call last):
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/nn/parallel/parallel_apply.py", line 64, in _worker
output = module(*input, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/peft/peft_model.py", line 529, in forward
return self.base_model(
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 770, in forward
outputs = self.model(
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 619, in forward
layer_outputs = decoder_layer(
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 316, in forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 217, in forward
key_states = self.k_proj(hidden_states).view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/bitsandbytes/nn/modules.py", line 242, in forward
out = bnb.matmul(x, self.weight, bias=self.bias, state=self.state)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/bitsandbytes/autograd/_functions.py", line 488, in matmul
return MatMul8bitLt.apply(A, B, out, bias, state)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/autograd/function.py", line 506, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/bitsandbytes/autograd/_functions.py", line 397, in forward
output += torch.matmul(subA, state.subB)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (1024x5 and 4x4096)

how to solve