Skip to content

PyTorch Implementation for INTERSPEECH'20 "An Effective Domain Adaptive Post-Training Method for BERT in Response Selection"

Notifications You must be signed in to change notification settings

taesunwhang/BERT-ResSel

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

17 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

An Effective Domain Adaptive Post-Training Method for BERT in Response Selection

Implements the model described in the following paper An Effective Domain Adaptive Post-Training Method for BERT in Response Selection.

@inproceedings{whang2020domain,
  author={Whang, Taesun and Lee, Dongyub and Lee, Chanhee and Yang, Kisu and Oh, Dongsuk and Lim, HeuiSeok},
  title="An Effective Domain Adaptive Post-Training Method for BERT in Response Selection",
  year=2020,
  booktitle={Proc. Interspeech 2020}
}

This code is reimplemented as a fork of huggingface/transformers.

Data Creation

  1. Download ubuntu_train.pkl, ubuntu_valid.pkl, ubuntu_test.pkl here or you can create pkl files to train response selection model based on BERT model. If you wish to create pkl, download ubuntu_corpus_v1 dataset here provided by Xu et al. (2016) and keep the files under data/ubuntu_corpus_v1 directory.
  2. Ubuntu corpus for domain post trianing will be created by running:
python data/data_utils.py

Post Training Data Creation

Download ubuntu_post_training.txt corpus here and simply run

python data/create_bert_post_training_data.py

After creating post_training data, keep ubuntu_post_training.hdf5 file under data/ubuntu_corpus_v1directory.

Domain Post Training BERT

To domain post-train BERT, simply run

python main.py --model bert_ubuntu_pt --train_type post_training --bert_pretrained bert-base-uncased --data_dir ./data/ubuntu_corpus_v1/ubuntu_post_training.hdf5

BERT Fine-tuning (Response Selection)

Training

Train a response selection model based on BERT_base:

python main.py --model bert_base_ft --train_type fine_tuning --bert_pretrained bert-base-uncased

Train a response selection model based on Domain post-trained BERT. If you wish to get the domain post trained BERT, download model checkpoint (bert-post-uncased-pytorch_model.pth) here, and keep checkpoint under resources/bert-post-uncased directory:

python main.py --model bert_dpt_ft --train_type fine_tuning --bert_pretrained bert-post-uncased

Evaluation

To evaluate bert_base,bert_dpt models, set a model checkpoint path and simply run

python main.py --model bert_dpt_ft --train_type fine_tuning --bert_pretrained bert-post-uncased --evaluate /path/to/checkpoint.pth

If you wish to get the pre-trained response selection model, we provide the model checkpoints below.

Model R@1 R@2 R@5 MRR
BERT_base 0.8115 0.9003 0.9768 0.8809
BERT_DPT 0.8515 0.9272 0.9851 0.9081

Acknowledgements

  • This work was supported by Institute for Information & communications Technology Promotion (IITP) grant funded by the Korea government (MSIT) (no. 2016-0-00010-003, Digital Centent InHouse R&D)
  • Work in collaboration with Kakao Corp.

About

PyTorch Implementation for INTERSPEECH'20 "An Effective Domain Adaptive Post-Training Method for BERT in Response Selection"

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published