This repository is the official implementation of Cloze Test Helps: Effective Video Anomaly Detection via Learning to Complete Video Events (oral paper In ACM Multimedia 2020) by Guang Yu, Siqi Wang, Zhiping Cai, En Zhu, Chuanfu Xu, Jianping Yin, Marius Kloft.
- python 3.6
- PyTorch 1.1.0 (0.3.0 for calculating optical flow)
- torchvision 0.3.0
- cuda 9.0.176
- cudnn 7.0.5
- mmcv 0.2.14 (might use
pip install mmcv==0.2.14
to install old version) - mmdetection 1.0rc0 (might use
git clone -b v1.0rc0 https://github.com/open-mmlab/mmdetection.git
to clone old version) - numpy 1.17.2
- scikit-learn 0.21.3
Refer to the full environment in issue. Note that our project is based on mmdet v1.0rc0. Run the program strictly according to our environment, or might try the newer versions of mmdet, PyTorch and mmcv.
Recently (2021.1) the interface of mmdet v1.0rc0 seems to have changed. If you install mmdet v1.0rc0 and get "No module named 'mmdet.datasets.pipelines' " when running the program, please refer to issue to fix the bug.
Download UCSDped2 from official website , avenue and Shanghaitech from OneDrive or BaiduYunPan (code:i9b3, provided by StevenLiuWen) , and ground truth of avenue from official website. Create a folder named raw_datasets
in root directory to store the downloaded datasets. The directory structure should be organized to match vad_datasets.py
as follows (Refer to the entire project directory structure in directory_structure.txt
):
.
├── ...
├── raw_datasets
│ ├── avenue
│ │ ├── bboxes_test_obj_det_with_motion.npy
│ │ ├── bboxes_train_obj_det_with_motion.npy
│ │ ├── ground_truth_demo
│ │ ├── testing
│ │ └── training
│ ├── ShanghaiTech
│ │ ├── bboxes_test_obj_det_with_motion.npy
│ │ ├── bboxes_train_obj_det_with_motion.npy
│ │ ├── extract_frames.py
│ │ ├── Testing
│ │ ├── training
│ │ └── training.zip
│ ├── UCSDped2
│ │ ├── bboxes_test_obj_det_with_motion.npy
│ │ ├── bboxes_train_obj_det_with_motion.npy
│ │ ├── Test
│ │ └── Train
├── calc_optical_flow.py
├── ...
Note: (1) To facilitate testing and training, extracted foreground bounding boxes (bboxes_test_obj_det_with_motion.npy
, bboxes_train_obj_det_with_motion.npy
) have been uploaded to the directories of each dataset. Please set train_bbox_saved=True
and test_bbox_saved=True
in config.cfg
to load the extracted bboxes directly if you don't want to extract bboxes using mmdet. (2) ShanghaiTech's training set provides videos rather than video frames, which need to be extracted manually. extract_frames.py
have been uploaded to ./raw_datasets/ShanghaiTech
for video frame extraction. After downloading and unzipping ShanghaiTech, run extract_frames.py
to get the video frames of ShanghaiTech training set.
(1) Follow the instructions to install FlowNet2, then download the pretrained model flownet2, and move the downloaded model FlowNet2_checkpoint.pth.tar
into ./FlowNet2_src/pretrained
(create a folder named pretrained).
(2) Run calc_optical_flow.py
(in PyTorch 0.3.0): python calc_optical_flow.py
. This will generate a new folder named optical_flow
containing the optical flow of the different datasets. The optical_flow
folder has basically the same directory structure as the raw_datasets folder.
(1) Follow the instructions to install mmdet (might use git clone -b v1.0rc0 https://github.com/open-mmlab/mmdetection.git
to clone old version of mmdetection). Then download the pretrained object detector Cascade R-CNN, and move it to fore_det/obj_det_checkpoints
(create a folder named obj_det_checkpoints).
(2) Select the model in ./data/raw2flow
, and move the files in the model folder (such as avenue_model_5raw1of_auc0.902
) into ./data/raw2flow
.
(3) Edit the file config.cfg
: i. Set the dataset_name
(UCSDped2
, avenue
and ShanghaiTech
are optional) of [shared_parameters]
for the selected model in step (2). ii. Set the context_of_num
(4 and 0 are optional, 4 corresponds to the model with 5of
and 0 corresponds to 1of
) in [SelfComplete]
.
(4) Run test.py
: python test.py
.
Edit the file config.cfg
according to Experimental Settings in our paper or your requirements, and run train.py
: python train.py
.
Dataset | UCSDped2 | avenue | ShanghaiTech |
---|---|---|---|
AUROC | 97.3% | 90.2% | 74.8% |
@inproceedings{yu2020cloze,
title={Cloze Test Helps: Effective Video Anomaly Detection via Learning to Complete Video Events},
author={Yu, Guang and Wang, Siqi and Cai, Zhiping and Zhu, En and Xu, Chuanfu and Yin, Jianping and Kloft, Marius},
booktitle={Proceedings of the 28th ACM International Conference on Multimedia},
pages={583--591},
year={2020}
}