KeyError: 'boxes' in training my own model

[JingyunLiang]

I followed instruction of Train your own model and got vgg16.pth. When training using ./experiments/scripts/train_faster_rcnn.sh 0 pascal_voc vgg16, error occurred:

Loaded dataset `voc_2007_trainval` for training
Set proposal method: gt
Appending horizontally-flipped training examples...
voc_2007_trainval gt roidb loaded from /home/ljy/pytorch-examples-master/pytorch-faster-rcnn-master/data/cache/voc_2007_trainval_gt_roidb.pkl
Traceback (most recent call last):
  File "./tools/trainval_net.py", line 104, in <module>
    imdb, roidb = combined_roidb(args.imdb_name)
  File "./tools/trainval_net.py", line 75, in combined_roidb
    roidbs = [get_roidb(s) for s in imdb_names.split('+')]
  File "./tools/trainval_net.py", line 75, in <listcomp>
    roidbs = [get_roidb(s) for s in imdb_names.split('+')]
  File "./tools/trainval_net.py", line 72, in get_roidb
    roidb = get_training_roidb(imdb)
  File "/home/ljy/pytorch-examples-master/pytorch-faster-rcnn-master/tools/../lib/model/train_val.py", line 302, in get_training_roidb
    imdb.append_flipped_images()
  File "/home/ljy/pytorch-examples-master/pytorch-faster-rcnn-master/tools/../lib/datasets/imdb.py", line 113, in append_flipped_images
    boxes = self.roidb[i]['boxes'].copy()
KeyError: 'boxes'
Command exited with non-zero status 1
7.08user 0.19system 0:07.29elapsed 99%CPU (0avgtext+0avgdata 331752maxresident)k
0inputs+40outputs (0major+99561minor)pagefaults 0swaps

BTW, I am confused about modification on vgg16, why do we need following modification ?

import torch
from torch.utils.model_zoo import load_url
from torchvision import models
sd = load_url("https://s3-us-west-2.amazonaws.com/jcjohns-models/vgg16-00b39a1b.pth")
sd['classifier.0.weight'] = sd['classifier.1.weight']
sd['classifier.0.bias'] = sd['classifier.1.bias']
del sd['classifier.1.weight']
del sd['classifier.1.bias']
sd['classifier.3.weight'] = sd['classifier.4.weight']
sd['classifier.3.bias'] = sd['classifier.4.bias']
del sd['classifier.4.weight']
del sd['classifier.4.bias']
torch.save(sd, "vgg16.pth")

[ruotianluo]

About vgg: jcjohnson/pytorch-vgg#3
About Keyerror, this sounds weird to me. Did you download the dataset correctly?
Can you check if this function is working properly

pytorch-faster-rcnn/lib/datasets/imdb.py

Line 216 in 19021d4

def create_roidb_from_box_list(self, box_list, gt_roidb):

.

[JingyunLiang]

Updates: resnet101-caffe.pth downloaded from your Gdrive also failed with the same problem. And I am sure that Voc 2007 is downloaded correctly.

When I want to debug and see what's wrong, error occured as longcw/faster_rcnn_pytorch#47 How do you debug your python code? Do you have the same problem in direct-run and debug mode?

[ruotianluo]

Your error is related to the data loading part. So, can you do what I said? Does roidb has boxes key?

[JingyunLiang]

Problem solved! I used your cache according Setup data https://github.com/ruotianluo/pytorch-faster-rcnn/blob/master/README.md#setup-data in your instruction. Maybe it is kind of confusing for us novices. In fact, everyone should generate his own cache.

By the way, the file type should be .pth in this line

pytorch-faster-rcnn/experiments/scripts/train_faster_rcnn.sh

Line 56 in 95996c9

NET_FINAL=output/${NET}/${TRAIN_IMDB}/default/${NET}_faster_rcnn_iter_${ITERS}.ckpt

[ruotianluo]

training script fixed.
For the cache file, this is actually from tf-faster-rcnn (I copy most part of his readme, I should make it clear). But that file worked for me. So, this is weird.