Joint Image Emotion Classification and Distribution Learning via Deep Convolutional Neural Network

By Jufeng Yang, Dongyu She, Ming Sun

Introduction

We develop a multi-task deep framework by jointly optimizing classification and distribution prediction

• It achieves state-of-the-art performance on emotion classification, and LDL prediction tasks.
• Our code is based on Caffe.

The paper has been accepted by IJCAI 2017. For more details, please refer to our paper.

Architecture

License

Our framework is released under the MIT License (refer to the LICENSE file for details).

Citing

If you find our framework useful in your research, please consider citing:

@inproceedings{ijcai2018joint,
	Author = {Jufeng Yang, Dongyu She, Ming Sun},
	Title = {Joint Image Emotion Classification and Distribution Learning via Deep Convolutional Neural Network},
	booktitle = {IJCAI},
	Year = {2017}
}

Contents

1. Requirements: software
2. Requirements: hardware
3. Basic installation
4. Extra Downloads (dataset lmdb)
5. Usage
6. Trained models

Requirements: software

1. Requirements for Caffe (see: Caffe installation instructions)

# This is required only if you will compile the matlab interface.

Requirements: hardware

1. NVIDIA GTX TITANX (~12G of memory)

Installation

1. Clone the repository

git clone https://github.com/sherleens/EmotionDistributionLearning.git

2. Change the label number in $ROOT/caffe_KL/src/caffe/layers/data_layer.cpp

line52:    int labelNum = 7; //change to the corresponding label number with the LMDB file
...
line110:   int labelNum = 7; //for example, change to 7 for the Emotion6 dataset, 8 for the Flickr_LDL dataset 

3. Build Caffe with KLloss

cd $ROOT/caffe_KL
# Now follow the Caffe installation instructions here
#   http://caffe.berkeleyvision.org/installation.html
# If you're experienced with Caffe and have all of the requirements installed
# and your Makefile.config in place, then simply do:
make all -j 8 && make matcaffe

Download dataset lmdb

LMDB file is generated by modified code (caffe_KL/tool/convert_imageset_r) to support multiple ground-truth labels, which can also be downloaded from Baidu, password:b83b.

Contents:

• dataset_config

  • emotion6_config.mat
  • flickrldl_config.mat
  • twitterldl_config.mat
  • readme

• lmdb_file

  • train_twitterldl_split1_lmdb.tar 1.5G
  • train_flickrldl_split1_lmdb.tar 1.67G
  • train_emotion6_lmdb.tar 265.4M
  • test_twitterldl_split1_lmdb.tar 384.7M
  • test_flickrldl_split1_lmdb.tar 426.8M
  • test_emotion6_lmdb.tar 113.7M

Usage

Train a deep network. For example, train a VGG16 network on distribution datasets.

cd prototxt && bash ./train_vgg_gs.sh

Our trained models

The models trained on the distribution datasets can be downloaded from here.

coming soon...