About world_expo'10 dataset

[liwenxi]

Recently, I try to use world_expo'10 dataset, but I found I'm hard to reduce the MAE of scene4. When MAE of scene1 is 1.5 and scene5 is 2.6, scene4 is 36.75.

I use train_frame to train and test_frame to test. ROI is used on both train and test. After ROI, I use
transform = transforms.Compose([transforms.ToTensor(),transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])])to normalize. For density, I also add ROI and set sigma=0.2M(x).

Is there anything else that needs attention?

[gjy3035]

@liwenxi
Scene 4 is harder than other scenes. Thus, the performance on it is poorer than that of others.

I have two suggestions that may improve your performance on it.

• In fact, there are many error annotations in the word expo data set. You can visualize the GT (key points) in the original image. Then you will find some heads outside the ROI are labeled. These mislabeled key points should be removed.
• Do you set the black for the region out of ROIs during the training process? Setting black may be not a good strategy. You can try to blur them or maintain the original image.

Besides, according to my experience and the results of experiments, I find that the small models perform better than other big models (such as pre-trained model, VGG, ResNet and so on).

For the detailed setting on WorldExpo, I will provide some code for it. Please watch it.

[liwenxi]

Thank you for reply!

• I found this problem. So I used ROI to mask the density map. I will try to delete the key points before generating density map.

• I compared MAE with ROI and MAE without ROI, the results are similar. I will use blur and original image to do more test.

Now I used CSRNet to test. This model is modified from VGG16. When I train model with pre-trained on world_expo'10, the MAE of scene4 is about 10 at first, but other scene is not good. With the increase of epoch, scene4 is getting worse, but others are getting better.

I also show out the outputs and density map with matplot. There are more errors in the upper part of the outputs. Maybe the body is recognized as person, but in fact head is out of ROI. So in density map there is no point.

I will try more test to solve the problem. Thanks again.

[gjy3035]

I does not conduct the reproduction of CSRNet on WorldExpo. However, I train other pretrained models on it, such as ResNet, VGG, deeplab v3 and so on. Their results can not outperform that of some small models from scratch, for example, ACSCP and SANet.

I am very much looking forward to your progress on WorldExpo.

[aachenhang]

I found some of the label file have no position message, like:
500674_E05-03-S20100717083000000E20100717233000000_5_clip1_2.mat.

[aachenhang]

And the image have lots of people indeed.

[liwenxi]

Yeah, so I deleted these images before training.
I list the wrong samples here.
Training set:

wrong_data = ['200778_C10-03-S20100717083000000E20100717233000000_4_clip1_2.jpg',
              ' 200778_C10-03-S20100717083000000E20100717233000000_4_clip1_3.jpg',
              '200778_C10-03-S20100717083000000E20100717233000000_clip1_3.jpg',
              '500674_E05-03-S20100717083000000E20100717233000000_5_clip1_2.jpg',
              '600079_E06-02-S20100717083000000E20100717233000000_7_clip1_2.jpg',
              '100270_A02HiRD36-01-S20100626080000000E20100626233000000_new.split.157_2.jpg']

Test set:

wrong_data = ['104207/104207_1-04-S20100821071000000E20100821120000000_034550.jpg']

[gjy3035]

@liwenxi Thanks for your summary!

[yxxxxxxxx]

@liwenxi Hello, have you ever reproduced the results of WorldExpo by using CSRNet. I delete the wrong images but still can't get the same result. I will appreciate it if you can help me.

[liwenxi]

@yxxxxxxxx Sorry, I couldn't reproduce the results.

[Wyt-ong]

Recently, I found another wrong image in this dataset.
Training set : {102267_01-S20100821090000000E20100821233000000_2_clip1_3.jpg}