Graph2Net

The Official implementation for Graph2Net: Perceptually-Enriched Graph Learning for Skeleton-Based Action Recognition (TCSVT 2021).

Also, on the basis of this method, we won the first place in Multi-Modal Video Reasoning and Analyzing Competition (MMVRAC, Track 2 Skeleton-based Action Recognition) from ICCV Workshop.

• Prerequisite
• Data
• Training&Testing
• Ensemble
• Citation
• Acknowledgement
• Contact

Prerequisite

• Python 3.7
• Pytorch 1.5
• Other Python libraries can be installed with pip install -r requirements.txt.

Data

Generate the Joint data

Ntu-RGB+D 60 & 120

• Download the raw data of NTU-RGB+D. Put NTU-RGB+D 60 data under the directory ./data/nturgbd_raw. Put NTU-RGB+D-120 data under the directory ./data/nturgbd120_raw.
• For NTU-RGB+D 60, preprocess data with python data_gen/ntu_gendata.py.
• For NTU-RGBD+120, preprocess data with python data_gen/ntu120_gendata.py.

Kinetics-400 Skeleton

• Download the raw data of Kinetics-400 Skeleton. Put Kinetics-400 Skeleton data under the directory ./data/kinetics_raw/.
• Preprocess data with python data_gen/kinetics_gendata.py.

Northwestern-UCLA

The preprocess of Northwestern-UCLA dataset is borrow from kchengiva/Shift-GCN.

• Download the raw data of Northwestern-UCLA. Put Northwestern-UCLA data under the directory ./data/nw_ucla_raw/.

Generate the Bone data

• Generate the bone data with python data_gen/gen_bone_data.py.

Training&Testing

Training

We provided several examples to train Graph2Net with this repo:

• To train on NTU-RGB+D 60 under Cross-View evaluation, you can run

  python main.py --config ./config/nturgbd-cross-view/train_joint.yaml

  python main.py --config ./config/nturgbd-cross-view/train_bone.yaml

• To train on Mini-Kinetics-Skeleton, you can run

  python main.py --config ./config/kinetics-skeleton/train_joint.yaml

  python main.py --config ./config/kinetics-skeleton/train_bone.yaml

• To train on Northwestern-UCLA, you can run

  python main_nw_ucla.py --config ./config/northwestern-ucla/train_joint.yaml

  python main_nw_ucla.py --config ./config/northwestern-ucla/train_bone.yaml

Testing

We also provided several examples to test Graph2Net with this repo:

• To test on NTU-RGB+D 60 under Cross-View evaluation, you can run

  python main.py --config ./config/nturgbd-cross-view/test_joint.yaml

  python main.py --config ./config/nturgbd-cross-view/test_bone.yaml

• To test on Mini-Kinetics-Skeleton, you can run

  python main.py --config ./config/kinetics-skeleton/test_joint.yaml

  python main.py --config ./config/kinetics-skeleton/test_bone.yaml

• To test on Northwestern-UCLA, you can run

  python main_nw_ucla.py --config ./config/northwestern-ucla/test_joint.yaml

  python main_nw_ucla.py --config ./config/northwestern-ucla/test_bone.yaml

The corresponding result of the above command is as follows,

	NTU-RGB+D 60 (Cross-View)	Mini-Kinetics-Skeleton	Northwestern-UCLA
Joint	95.2	42.3	94.4
Bone	94.6	42.1	92.5

In the save_models folder, we also provide the trained model parameters.

Please refer to the config folder for other training and testing commands. You can also freely change the train or test config file according to your needs.

Ensemble

To ensemble the results of joints and bones, run the test command we provided to generate the scores of the softmax layer. Then combine the generated scores with:

• NTU-RGB+D 60

  python ensemble.py --datasets ntu/xview

• Mini-Kinetics-Skeleton

  python ensemble.py --datasets kinetics_min_skeleton

• Northwestern-UCLA

  python ensemble_nw_ucla.py

The corresponding result of the above command is as follows,

	NTU-RGB+D 60 (Cross-View)	Mini-Kinetics-Skeleton	Northwestern-UCLA
Ensemble	96.0	44.9	95.3

Citation

If you find this model useful for your research, please use the following BibTeX entry.

@ARTICLE{9446181,
  author={Wu, Cong and Wu, Xiao-Jun and Kittler, Josef},
  journal={IEEE Transactions on Circuits and Systems for Video Technology}, 
  title={Graph2Net: Perceptually-Enriched Graph Learning for Skeleton-Based Action Recognition}, 
  year={2021},
  volume={},
  number={},
  pages={1-1},
  doi={10.1109/TCSVT.2021.3085959}
}

Acknowledgement

Thanks for the framework provided by 2s-AGCN, which is source code of the published work Two-Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition in CVPR 2019.

Contact

For any questions, feel free to contact: congwu@stu.jiangnan.edu.cn.