Token indices sequence length is longer than the specified maximum sequence length for this model (4158 > 2048)

[tginart]

Describe the bug
Running the Pythia-7B fine-tune script on 4 x A10 (24GB each).

Seems like issue with seq len:

_```
Token indices sequence length is longer than the specified maximum sequence length for this model (4158 > 2048). Running this sequence through the model will result in indexing errors
Traceback (most recent call last):
File "/home/ec2-user/OpenChatKit/training/dist_clm_train.py", line 358, in
main()
File "/home/ec2-user/OpenChatKit/training/dist_clm_train.py", line 332, in main
train_loop(args, pipe, device, train_data_loader, test_data_loader)
File "/home/ec2-user/OpenChatKit/training/dist_clm_train.py", line 151, in train_loop
get_data_parallel_comm().recv(
File "/home/ec2-user/OpenChatKit/training/comm/nccl_backend.py", line 79, in recv
self.comm.recv(
File "cupy_backends/cuda/libs/nccl.pyx", line 477, in cupy_backends.cuda.libs.nccl.NcclCommunicator.recv
File "cupy_backends/cuda/libs/nccl.pyx", line 129, in cupy_backends.cuda.libs.nccl.check_status
cupy_backends.cuda.libs.nccl.NcclError: NCCL_ERROR_UNHANDLED_CUDA_ERROR: unhandled cuda error
Traceback (most recent call last):
File "/home/ec2-user/OpenChatKit/training/dist_clm_train.py", line 358, in
main()
File "/home/ec2-user/OpenChatKit/training/dist_clm_train.py", line 332, in main
train_loop(args, pipe, device, train_data_loader, test_data_loader)
File "/home/ec2-user/OpenChatKit/training/dist_clm_train.py", line 117, in train_loop
get_data_parallel_comm().send(
File "/home/ec2-user/OpenChatKit/training/comm/nccl_backend.py", line 65, in send
self.comm.send(
File "cupy_backends/cuda/libs/nccl.pyx", line 468, in cupy_backends.cuda.libs.nccl.NcclCommunicator.send
File "cupy_backends/cuda/libs/nccl.pyx", line 129, in cupy_backends.cuda.libs.nccl.check_status
cupy_backends.cuda.libs.nccl.NcclError: NCCL_ERROR_UNHANDLED_CUDA_ERROR: unhandled cuda error

**To Reproduce**
Steps to reproduce the behavior:
Run Pythia train script with following modifications:

`--num-layers 16 --embedding-dim 4096 \
--world-size 4 --pipeline-group-size 2 --data-group-size 2 \`


**Expected behavior**
Training should work? 
Using standard AWS deep learning AMI with cuda

[yangnicejin]

This problem also occurred when I reproduced the 20B model

[haifaksh]

Hi ,
did you find a solution ?
I am facing the same problem. I am trying to test alpa for distributed parallel training.
NVIDIA-SMI 470.182.03 Driver Version: 470.182.03 CUDA Version: 11.4

nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Sun_Feb_14_21:12:58_PST_2021
Cuda compilation tools, release 11.2, V11.2.152
Build cuda_11.2.r11.2/compiler.29618528_0

When I run python3 -m alpa.test_install:
File "cupy_backends/cuda/libs/nccl.pyx", line 283, in cupy_backends.cuda.libs.nccl.NcclCommunicator.init
File "cupy_backends/cuda/libs/nccl.pyx", line 129, in cupy_backends.cuda.libs.nccl.check_status
cupy_backends.cuda.libs.nccl.NcclError: NCCL_ERROR_UNHANDLED_CUDA_ERROR: unhandled cuda error

Any help would be really appreciated. I tried different versions of cuda , same error every time..

[huhuiqi7]

also same error...