ULMFit Language Modeling, Text Feature Extraction and Text Classification in Thai Language. Created as part of pyThaiNLP with ULMFit implementation from fast.ai
Models and word embeddings can also be downloaded via Dropbox.
We pretrained a language model with 60,005 embeddings on Thai Wikipedia Dump (perplexity of 28.71067) and text classification (micro-averaged F-1 score of 0.60322 on 5-label classification problem. Benchmarked to 0.5109 by fastText and 0.4976 by LinearSVC on Wongnai Challenge: Review Rating Prediction. The language model can also be used to extract text features for other downstream tasks.
- Python>=3.6
- PyTorch>=1.0
- fastai>=1.0.38
- Pretrained language model based on Thai Wikipedia with the perplexity of 46.61
- Pretrained word embeddings (.vec) with 51,556 tokens and 300 dimensions
- Classification benchmark of 94.4% accuracy compared to 65.2% by fastText for 4-label classification of BEST
- Refactored to use
fastai.textinstead oftorchtext - Pretrained word embeddings (.vec and .bin) with 60,000 tokens and 300 dimensions (
word2vec_examples.ipynb) - Classification benchmark of 0.60925 micro-averaged F1 score compared to 0.49366 by fastText and 0.58139 by competition winner for 5-label classification of Wongnai Challenge: Review Rating Prediction (
ulmfit_wongnai.ipynb) - Text feature extraction for other downstream tasks such as clustering (
ulmfit_ec.ipynb)
- Repo name changed to
thai2fitin order to avoid confusion since this is ULMFit not word2vec implementation - Migrate to Pytorch 1.0 and fastai 1.0 API
- Add QRNN-based models; inference time drop by 50% on average
- Pretrained language model based on Thai Wikipedia with the perplexity of 46.04264 (20% validation) and 23.32722 (1% validation) (
pretrain_wiki.ipynb) - Pretrained word embeddings (.vec and .bin) with 60,000 tokens and 400 dimensions (
word2vec_examples.ipynb) based on QRNN - Classification benchmark of 0.60925 micro-averaged F1 score compared to 0.49366 by fastText and 0.58139 by competition winner for 5-label classification of Wongnai Challenge: Review Rating Prediction (
ulmfit_wongnai.ipynb) - LSTM weights are copied from v0.2 according to guideline provided in fastai forum
I remember someone doing a script but I can’t find it. For both, you just have to map the old names of the weights to the new ones. Note that:
in language models, there is a bias in the decoder in fastai v1 that you probably won’t have
in the classifier, the order you see for the layers is artificial (it’s the pytorch representation that takes the things in the order you put them in __init__ when not using Sequential) but the two models (old and new) apply batchnorm, dropout and linear in the same order
tokenizing is done differently in fastai v1, so you may have to fine-tune your models again (we add an xxmaj token for words beginning with a capital for instance)
for weight dropout, you want the weights you have put both in '0.rnns.0.module.weight_hh_l0' and 0.rnns.0.weight_hh_l0_raw (the second one is copied to the first with dropout applied anyway)
- Support fastai>=1.0.38
- Pretrained Thai Wikipedia Dump with the same training scheme as ulmfit-multilingual
- Remove QRNN models due to inferior performance
- Classification benchmarks now include for wongnai-corpus (See
wongnai_cls), prachathai-67k (Seeprachathai_cls), and wisesight-sentiment (Seewisesight_cls)
- Better text cleaning rules resulting in Thai Wikipedia Dump pretrained perplexity of 28.71067.
- Replace AWD-LSTM/QRNN with tranformers-based models
- Named-entity recognition
We trained the ULMFit model implemented bythai2fit for text classification. We use Wongnai Challenge: Review Rating Prediction as our benchmark as it is the only sizeable and publicly available text classification dataset at the time of writing (June 21, 2018). It has 39,999 reviews for training and validation, and 6,203 reviews for testing.
We achieved validation perplexity at 35.75113 and validation micro F1 score at 0.598 for five-label classification. Micro F1 scores for public and private leaderboards are 0.59313 and 0.60322 respectively, which are state-of-the-art as of the time of writing (February 27, 2019). FastText benchmark based on their own pretrained embeddings has the performance of 0.50483 and 0.49366 for public and private leaderboards respectively. See ulmfit_wongnai.ipynb for more details.
The pretrained language model of thai2fit can be used to convert Thai texts into vectors, after which said vectors can be used for various machine learning tasks such as classification, clustering, translation, question answering and so on. The idea is to train a language model that "understands" the texts then extract certain vectors that the model "thinks" represents the texts we want. You can access this functionality easily via pythainlp
from pythainlp.ulmfit import *
document_vector('วันนี้วันดีปีใหม่',learn,data)
>> array([ 0.066298, 0.307813, 0.246051, 0.008683, ..., -0.058363, 0.133258, -0.289954, -1.770246], dtype=float32)
The goal of this notebook is to train a language model using the fast.ai version of AWD LSTM Language Model, with data from Thai Wikipedia Dump last updated February 17, 2019. Using 40M/200k/200k tokens of train-validation-test split, we achieved validation perplexity of 27.81627 with 60,004 embeddings at 400 dimensions, compared to state-of-the-art as of October 27, 2018 at 42.41 for English WikiText-2 by Yang et al (2018). To the best of our knowledge, there is no comparable research in Thai language at the point of writing (February 17, 2019). See thwiki_lm for more details.
We use the embeddings from v0.1 since it was trained specifically for word2vec as opposed to latter versions which garner to classification. The thai2vec.bin 51,556 word embeddings of 300 dimensions, in descending order by their frequencies (See thai2vec.vocab). The files are in word2vec format readable by gensim. Most common applications include word vector visualization, word arithmetic, word grouping, cosine similarity and sentence or document vectors. For sample code, see thwiki_lm/word2vec_examples.ipynb.
You can do simple "arithmetic" with words based on the word vectors such as:
- ผู้หญิง (female) + ราชา (king) - ผู้ชาย (male) = ราชินี (queen)
- หุ้น (stock) - พนัน (gambling) = กิจการ (business)
- อเมริกัน (american) + ฟุตบอล (football) = เบสบอล (baseball)
It can also be used to do word groupings. For instance:
- อาหารเช้า อาหารสัตว์ อาหารเย็น อาหารกลางวัน (breakfast animal-food dinner lunch) - อาหารสัตว์ (animal-food) is type of food whereas others are meals in the day
- ลูกสาว ลูกสะใภ้ ลูกเขย ป้า (duaghter daughter-in-law son-in-law aunt) - ลูกสาว (daughter) is immediate family whereas others are not
- กด กัด กิน เคี้ยว (press bite eat chew) - กด (press) is not verbs for the eating process Note that this could be relying on a different "take" than you would expect. For example, you could have answered ลูกเขย in the second example because it is the one associated with male gender.
Calculate cosine similarity between two word vectors.
- จีน (China) and ปักกิ่ง (Beijing): 0.31359560752667964
- อิตาลี (Italy) and โรม (Rome): 0.42819627065839394
- ปักกิ่ง (Beijing) and โรม (Rome): 0.27347283956785434
- จีน (China) and โรม (Rome): 0.02666692964073511
- อิตาลี (Italy) and ปักกิ่ง (Beijing): 0.17900795797557473
@software{charin_polpanumas_2021_4429691,
author = {Charin Polpanumas and
Wannaphong Phatthiyaphaibun},
title = {thai2fit: Thai language Implementation of ULMFit},
month = jan,
year = 2021,
publisher = {Zenodo},
version = {v0.3},
doi = {10.5281/zenodo.4429691},
url = {https://doi.org/10.5281/zenodo.4429691}
}