NN Time Sequence Prediction

Environment

• OS
  Ubuntu 16.04.3 LTS (Xenial Xerus)　

• GPU
  NVIDIA Tesla K40　

• CUDA
  cuda driver 375.74　
  cuda 8.0

• Python
  Python 3.6.2　
  Pytorch 0.3.0

I used a qrnn library of Salesforce in my code (./models/torchqrnn/*).

@article{bradbury2016quasi,
  title={{Quasi-Recurrent Neural Networks}},
  author={Bradbury, James and Merity, Stephen and Xiong, Caiming and Socher, Richard},
  journal={International Conference on Learning Representations (ICLR 2017)},
  year={2017}
}

Usage

python main.py [-h] [--model {all,rnn,lstm,qrnn,cnn}] [--epoch EPOCH]
               [--batch_size BATCH_SIZE] [--n_iter N_ITER] [--seq_len SEQ_LEN]
               [--hidden_size HIDDEN_SIZE] [--num_layers NUM_LAYERS]
               [--result_path RESULT_PATH] [--cuda CUDA]

optional arguments:
  -h, --help                        show this help message and exit
  --model {all,rnn,lstm,qrnn,cnn}   The type of model (default: all)
  --epoch EPOCH                     The number of epochs to run (default: 300)
  --batch_size BATCH_SIZE           The number of batch (default: 200)
  --n_iter N_ITER                   The number of iteration (default: 5)
  --seq_len SEQ_LEN                 The length of sequence (default: 50)
  --hidden_size HIDDEN_SIZE         The number of features in the hidden state h (default: 20)
  --num_layers NUM_LAYERS           The number of layers (default: 2)
  --result_path RESULT_PATH         Result path (default: ./result)
  --cuda CUDA                       set CUDA_VISIBLE_DEVICES (default: None)

example
If you set --model all(default), all models (rnn, lstm, cnn, qrnn) are used. If you cannot use GPU, set --cuda None(default).

python main.py --cuda 0 --model all

You can also run one model.

python main.py --cuda 0 --model rnn 

Datasets

I used summed up sin wave.

You can generate the datasets as follows:

python generate_data.py [-h] [--sample SAMPLE] [--path PATH]
                        [--filename FILENAME]

optional arguments:
  -h, --help           show this help message and exit
  --sample SAMPLE      The number of samples (default: 10000)
  --path PATH          Data path (default: ./data)
  --filename FILENAME  File name (default: data.pkl)

Result

Train & Test Error	Time

Prediction results are as follows:

RNN	LSTM
CNN	QRNN

Thesis

Overall

• Recent Advances in Recurrent Neural Networks

RNN

• Finding Structure in Time
• SERIAL ORDER: A PARALLEL DISTRIBUTED PROCESSING APPROACH

LSTM

• LONG SHORT-TERM MEMORY
• Learning to Forget: Continual Prediction with LSTM
• Recurrent Nets that Time and Count

CNN（TDNN）

• Phoneme Recognition Using Time-Delay Neural Networks
• Review of TDNN Architectures for Speech Recognition
• TDNN architecture for efficient modeling of longtemporal contexts

QRNN

• Quasi-Recurrent Neural Networks