RuntimeError: Mask tensor can take 0 and 1 values only #6
Comments
|
I originally installed the torch according to the required version, but reported the same error as that in #3.According to your comments below, I installed the torch version matching my CUDA, and then reported the above error.Now,my CUDA is 11.7. |
Hello, thanks for your interest. Firstly, "You are using 0 GPUs to train!!" may show that your cuda is unavailable. Secondly, I do not encounter this RuntimeError problem, I guess it may also be due to the torch version. |
|
Yes, at the beginning, I configured the environment according to your requirements. I could use 3 GPU for training, but the error in #3 would be reported. According to the answer in that question, I upgraded my pytorch version to adapt to my CUDA version（11.7）, and then ran it again, it would display 0 GPU for training |
Hello, the author, I encountered some problems in reproducing your project:
When I run the pre training command "bash run/retrain. bash", an error occurs after several steps. The situation is as follows：
(hop) wuyifei@dell-Precision-7920-Tower:~/code/HOP-VLN$ bash run/pretrain.bash
You are using 0 GPUs to train!!
Loading image features from img_features/ResNet-152-imagenet.tsv
read_num 0
feature_store: True
Loading image features from img_features/ResNet-152-imagenet.tsv
read_num 0
feature_store: True
you have loaded 1069620 time steps
Loading image features from img_features/ResNet-152-imagenet.tsv
read_num 0
feature_store: True
Loading image features from img_features/ResNet-152-imagenet.tsv
read_num 0
feature_store: True
you have loaded 17420 time steps
/home/wuyifei/code/HOP-VLN/tasks/pretrain/batch_loader.py:137: UserWarning: masked_fill_ received a mask with dtype torch.uint8, this behavior is now deprecated,please use a mask with dtype torch.bool instead. (Triggered internally at /opt/conda/conda-bld/pytorch_1670525496686/work/aten/src/ATen/native/TensorAdvancedIndexing.cpp:1646.)
probability_matrix.masked_fill_(torch.tensor(special_tokens_mask, dtype=torch.uint8), value=0.0)
/home/wuyifei/code/HOP-VLN/tasks/pretrain/batch_loader.py:141: UserWarning: masked_fill_ received a mask with dtype torch.uint8, this behavior is now deprecated,please use a mask with dtype torch.bool instead. (Triggered internally at /opt/conda/conda-bld/pytorch_1670525496686/work/aten/src/ATen/native/TensorAdvancedIndexing.cpp:1646.)
attention_mask = torch.full(labels.shape, 1).masked_fill_(torch.tensor(att_mask, dtype=torch.uint8), value=0)
/home/wuyifei/code/HOP-VLN/tasks/pretrain/batch_loader.py:137: UserWarning: masked_fill_ received a mask with dtype torch.uint8, this behavior is now deprecated,please use a mask with dtype torch.bool instead. (Triggered internally at /opt/conda/conda-bld/pytorch_1670525496686/work/aten/src/ATen/native/TensorAdvancedIndexing.cpp:1646.)
probability_matrix.masked_fill_(torch.tensor(special_tokens_mask, dtype=torch.uint8), value=0.0)
/home/wuyifei/code/HOP-VLN/tasks/pretrain/batch_loader.py:141: UserWarning: masked_fill_ received a mask with dtype torch.uint8, this behavior is now deprecated,please use a mask with dtype torch.bool instead. (Triggered internally at /opt/conda/conda-bld/pytorch_1670525496686/work/aten/src/ATen/native/TensorAdvancedIndexing.cpp:1646.)
attention_mask = torch.full(labels.shape, 1).masked_fill_(torch.tensor(att_mask, dtype=torch.uint8), value=0)
Iteration: 0%|| 0/8357 [00:00<?, ?it/s]/home/wuyifei/code/HOP-VLN/tasks/pretrain/batch_loader.py:137: UserWarning: masked_fill_ received a mask with dtype torch.uint8, this behavior is now deprecated,please use a mask with dtype torch.bool instead. (Triggered internally at /opt/conda/conda-bld/pytorch_1670525496686/work/aten/src/ATen/native/TensorAdvancedIndexing.cpp:1646.)
probability_matrix.masked_fill_(torch.tensor(special_tokens_mask, dtype=torch.uint8), value=0.0)
/home/wuyifei/code/HOP-VLN/tasks/pretrain/batch_loader.py:137: UserWarning: masked_fill_ received a mask with dtype torch.uint8, this behavior is now deprecated,please use a mask with dtype torch.bool instead. (Triggered internally at /opt/conda/conda-bld/pytorch_1670525496686/work/aten/src/ATen/native/TensorAdvancedIndexing.cpp:1646.)
probability_matrix.masked_fill_(torch.tensor(special_tokens_mask, dtype=torch.uint8), value=0.0)
/home/wuyifei/code/HOP-VLN/tasks/pretrain/batch_loader.py:137: UserWarning: masked_fill_ received a mask with dtype torch.uint8, this behavior is now deprecated,please use a mask with dtype torch.bool instead. (Triggered internally at /opt/conda/conda-bld/pytorch_1670525496686/work/aten/src/ATen/native/TensorAdvancedIndexing.cpp:1646.)
probability_matrix.masked_fill_(torch.tensor(special_tokens_mask, dtype=torch.uint8), value=0.0)
/home/wuyifei/code/HOP-VLN/tasks/pretrain/batch_loader.py:141: UserWarning: masked_fill_ received a mask with dtype torch.uint8, this behavior is now deprecated,please use a mask with dtype torch.bool instead. (Triggered internally at /opt/conda/conda-bld/pytorch_1670525496686/work/aten/src/ATen/native/TensorAdvancedIndexing.cpp:1646.)
attention_mask = torch.full(labels.shape, 1).masked_fill_(torch.tensor(att_mask, dtype=torch.uint8), value=0)
/home/wuyifei/code/HOP-VLN/tasks/pretrain/batch_loader.py:141: UserWarning: masked_fill_ received a mask with dtype torch.uint8, this behavior is now deprecated,please use a mask with dtype torch.bool instead. (Triggered internally at /opt/conda/conda-bld/pytorch_1670525496686/work/aten/src/ATen/native/TensorAdvancedIndexing.cpp:1646.)
attention_mask = torch.full(labels.shape, 1).masked_fill_(torch.tensor(att_mask, dtype=torch.uint8), value=0)
/home/wuyifei/code/HOP-VLN/tasks/pretrain/batch_loader.py:137: UserWarning: masked_fill_ received a mask with dtype torch.uint8, this behavior is now deprecated,please use a mask with dtype torch.bool instead. (Triggered internally at /opt/conda/conda-bld/pytorch_1670525496686/work/aten/src/ATen/native/TensorAdvancedIndexing.cpp:1646.)
probability_matrix.masked_fill_(torch.tensor(special_tokens_mask, dtype=torch.uint8), value=0.0)
/home/wuyifei/code/HOP-VLN/tasks/pretrain/batch_loader.py:141: UserWarning: masked_fill_ received a mask with dtype torch.uint8, this behavior is now deprecated,please use a mask with dtype torch.bool instead. (Triggered internally at /opt/conda/conda-bld/pytorch_1670525496686/work/aten/src/ATen/native/TensorAdvancedIndexing.cpp:1646.)
attention_mask = torch.full(labels.shape, 1).masked_fill_(torch.tensor(att_mask, dtype=torch.uint8), value=0)
/home/wuyifei/code/HOP-VLN/tasks/pretrain/batch_loader.py:141: UserWarning: masked_fill_ received a mask with dtype torch.uint8, this behavior is now deprecated,please use a mask with dtype torch.bool instead. (Triggered internally at /opt/conda/conda-bld/pytorch_1670525496686/work/aten/src/ATen/native/TensorAdvancedIndexing.cpp:1646.)
attention_mask = torch.full(labels.shape, 1).masked_fill_(torch.tensor(att_mask, dtype=torch.uint8), value=0)
/home/wuyifei/code/HOP-VLN/tasks/pretrain/batch_loader.py:137: UserWarning: masked_fill_ received a mask with dtype torch.uint8, this behavior is now deprecated,please use a mask with dtype torch.bool instead. (Triggered internally at /opt/conda/conda-bld/pytorch_1670525496686/work/aten/src/ATen/native/TensorAdvancedIndexing.cpp:1646.)
probability_matrix.masked_fill_(torch.tensor(special_tokens_mask, dtype=torch.uint8), value=0.0)
/home/wuyifei/code/HOP-VLN/tasks/pretrain/batch_loader.py:141: UserWarning: masked_fill_ received a mask with dtype torch.uint8, this behavior is now deprecated,please use a mask with dtype torch.bool instead. (Triggered internally at /opt/conda/conda-bld/pytorch_1670525496686/work/aten/src/ATen/native/TensorAdvancedIndexing.cpp:1646.)
attention_mask = torch.full(labels.shape, 1).masked_fill_(torch.tensor(att_mask, dtype=torch.uint8), value=0)
/home/wuyifei/code/HOP-VLN/tasks/pretrain/batch_loader.py:137: UserWarning: masked_fill_ received a mask with dtype torch.uint8, this behavior is now deprecated,please use a mask with dtype torch.bool instead. (Triggered internally at /opt/conda/conda-bld/pytorch_1670525496686/work/aten/src/ATen/native/TensorAdvancedIndexing.cpp:1646.)
probability_matrix.masked_fill_(torch.tensor(special_tokens_mask, dtype=torch.uint8), value=0.0)
/home/wuyifei/code/HOP-VLN/tasks/pretrain/batch_loader.py:141: UserWarning: masked_fill_ received a mask with dtype torch.uint8, this behavior is now deprecated,please use a mask with dtype torch.bool instead. (Triggered internally at /opt/conda/conda-bld/pytorch_1670525496686/work/aten/src/ATen/native/TensorAdvancedIndexing.cpp:1646.)
attention_mask = torch.full(labels.shape, 1).masked_fill_(torch.tensor(att_mask, dtype=torch.uint8), value=0)
/home/wuyifei/code/HOP-VLN/tasks/pretrain/batch_loader.py:142: UserWarning: masked_fill_ received a mask with dtype torch.uint8, this behavior is now deprecated,please use a mask with dtype torch.bool instead. (Triggered internally at /opt/conda/conda-bld/pytorch_1670525496686/work/aten/src/ATen/native/TensorAdvancedIndexing.cpp:1646.)
labels[~masked_indices] = -1 # We only compute loss on masked tokens
/home/wuyifei/code/HOP-VLN/tasks/pretrain/batch_loader.py:142: UserWarning: masked_fill_ received a mask with dtype torch.uint8, this behavior is now deprecated,please use a mask with dtype torch.bool instead. (Triggered internally at /opt/conda/conda-bld/pytorch_1670525496686/work/aten/src/ATen/native/TensorAdvancedIndexing.cpp:1646.)
labels[~masked_indices] = -1 # We only compute loss on masked tokens
Iteration: 0%|| 0/8357 [00:00<?, ?it/s]
Traceback (most recent call last):
File "./tasks/pretrain/main.py", line 762, in
main()
File "./tasks/pretrain/main.py", line 727, in main
global_step, tr_loss = trainval(params, train_dataset, eval_dataset, model, tokenizer)
File "./tasks/pretrain/main.py", line 179, in trainval
for step, batch in epoch_iterator:
File "/home/wuyifei/anaconda3/envs/hop/lib/python3.7/site-packages/tqdm/std.py", line 1185, in iter
for obj in iterable:
File "/home/wuyifei/anaconda3/envs/hop/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 628, in next
data = self._next_data()
File "/home/wuyifei/anaconda3/envs/hop/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1333, in _next_data
return self._process_data(data)
File "/home/wuyifei/anaconda3/envs/hop/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1359, in _process_data
data.reraise()
File "/home/wuyifei/anaconda3/envs/hop/lib/python3.7/site-packages/torch/_utils.py", line 543, in reraise
raise exception
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/wuyifei/anaconda3/envs/hop/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
data = fetcher.fetch(index)
File "/home/wuyifei/anaconda3/envs/hop/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 58, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/wuyifei/anaconda3/envs/hop/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 58, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/wuyifei/code/HOP-VLN/tasks/pretrain/batch_loader.py", line 212, in getitem
output = self.getQuery(query, self.nag_trajs, self.nag_trajs_seq, self.nag_trajs_bnb)
File "/home/wuyifei/code/HOP-VLN/tasks/pretrain/batch_loader.py", line 460, in getQuery
masked_text_seq, masked_text_label, attention_mask = mask_tokens(text_seq, self.tok, self.args)
File "/home/wuyifei/code/HOP-VLN/tasks/pretrain/batch_loader.py", line 142, in mask_tokens
labels[~masked_indices] = -1 # We only compute loss on masked tokens
RuntimeError: Mask tensor can take 0 and 1 values only
Can you help me solve this problem? I would be grateful！
The text was updated successfully, but these errors were encountered: