Facial Expression Recognition using Residual Masking Network

The code for my undergraduate thesis.

Inference:

1. Install from pip

pip install rmn

# or build from source

git clone git@github.com:phamquiluan/ResidualMaskingNetwork.git
cd ResidualMaskingNetwork
pip install -e .

2. Run demo in Python (with webcam available)

from rmn import RMN
m = RMN()
m.video_demo()

3. Detect emotions from an image

image = cv2.imread("some-image-path.png")
results = m.detect_emotion_for_single_frame(image)
print(results)
image = m.draw(image, results)
cv2.imwrite("output.png", image)

Table of Contents

• Recent Update
• Benchmarking on FER2013
• Benchmarking on ImageNet
• Installation
• Download datasets
• Training on FER2013
• Training on ImageNet
• Evaluation results
• Download dissertation and slide

       

Recent Update

• [07/03/2023] Re-structure, update Readme
• [05/05/2021] Release ver 2, add colab
• [27/02/2021] Add paper
• [14/01/2021] Packaging Project and publish rmn on Pypi
• [27/02/2020] Update Tensorboard visualizations and Overleaf source
• [22/02/2020] Test-time augmentation implementation.
• [21/02/2020] Imagenet training code and trained weights released.
• [21/02/2020] Imagenet evaluation results released.
• [10/01/2020] Checking demo stuff and training procedure works on another machine
• [09/01/2020] First time upload

Benchmarking on FER2013

We benchmark our code thoroughly on two datasets: FER2013 and VEMO. Below are the results and trained weights:

Model	Accuracy
VGG19	70.80
EfficientNet_b2b	70.80
Googlenet	71.97
Resnet34	72.42
Inception_v3	72.72
Bam_Resnet50	73.14
Densenet121	73.16
Resnet152	73.22
Cbam_Resnet50	73.39
ResMaskingNet	74.14
ResMaskingNet + 6	76.82

Results in VEMO dataset could be found in my thesis or slide (attached below)

Benchmarking on ImageNet

We also benchmark our model on ImageNet dataset.

Model	Top-1 Accuracy	Top-5 Accuracy
Resnet34	72.59	90.92
CBAM Resnet34	73.77	91.72
ResidualMaskingNetwork	74.16	91.91

Installation

• Install PyTorch by selecting your environment on the website and running the appropriate command.
• Clone this repository and install package prerequisites below.
• Then download the dataset by following the instructions below.

Datasets

• FER2013 Dataset (locate it in saved/data/fer2013 like saved/data/fer2013/train.csv)
• ImageNet 1K Dataset (ensure it can be loaded by torchvision.datasets.Imagenet)

Training on FER2013

• To train the networks, you need to specify the model name and other hyperparameters in the config file (located at configs/*) then ensure it is loaded in main file, then run training procedure by simply run main file, for example:

python main_fer.py  # Example for fer2013_config.json file

• The best checkpoints will chosen at term of best validation accuracy, located at saved/checkpoints
• The TensorBoard training logs are located at saved/logs, to open it, use tensorboard --logdir saved/logs/

• By default, it will train alexnet model, you can switch to another model by edit configs/fer2013\_config.json file (to resnet18 or cbam\_resnet50 or my network resmasking\_dropout1.

Training on the Imagenet dataset

To perform training resnet34 on 4 V100 GPUs on a single machine:

python ./main_imagenet.py -a resnet34 --dist-url 'tcp://127.0.0.1:12345' --dist-backend 'nccl' --multiprocessing-distributed --world-size 1 --rank 0

Evaluation

For students, who should take care of the font family of the confusion matrix and would like to write things in LaTeX, below is an example for generating a striking confusion matrix.

(Read this article for more information, there will be some bugs if you blindly run the code without reading).

python cm_cbam.py

Ensemble method

I used the no-weighted sum average ensemble method to fuse 7 different models together, to reproduce results, you need to do some steps:

1. Download all needed trained weights and locate them on the ./saved/checkpoints/ directory. The link to download can be found in the Benchmarking section.
2. Edit file gen_results and run it to generate result offline for each model.
3. Run the gen_ensemble.py file to generate accuracy for example methods.

Dissertation and Slide

• Dissertation PDF (in Vietnamese)
• Dissertation Overleaf Source
• Presentation slide PDF (in English) with full appendix
• Presentation slide Overleaf Source
• ICPR Paper
• ICPR Poster Overleaf Source

Authors

• Luan Pham
• Tuan Anh Tran

Citation

Pham Luan, The Huynh Vu, and Tuan Anh Tran. "Facial Expression Recognition using Residual Masking Network". In: Proc. ICPR. 2020.

@inproceedings{pham2021facial,
  title={Facial expression recognition using residual masking network},
  author={Pham, Luan and Vu, The Huynh and Tran, Tuan Anh},
  booktitle={2020 25th International Conference on Pattern Recognition (ICPR)},
  pages={4513--4519},
  year={2021},
  organization={IEEE}
}

Star History