Skip to content

Latest commit

 

History

History
63 lines (38 loc) · 2.35 KB

README.md

File metadata and controls

63 lines (38 loc) · 2.35 KB
Raw

NLP for Punjabi

This repository contains State of the Art Language models and Classifier for Punjabi language (spoken in Indian sub-continent)

The models trained here have been used in Natural Language Toolkit for Indic Languages (iNLTK)

Dataset

Created as part of this project

  1. Punjabi Wikipedia Articles

  2. Punjabi News Dataset

Open Source Datasets

  1. IndicNLP News Article Classification Dataset - Punjabi

Results

Language Model Perplexity (on validation set)

Architecture/Dataset Punjabi Wikipedia Articles
ULMFiT 24.40
TransformerXL 14.03

Classification Metrics

ULMFiT
Dataset Accuracy MCC Notebook to Reproduce results
IndicNLP News Article Classification Dataset - Punjabi 97.12 96.17 Link

Visualizations

Word Embeddings
Architecture Visualization
ULMFiT Embeddings projection
TransformerXL Embeddings projection
Sentence Embeddings
Architecture Visualization
ULMFiT Encodings projection

Pretrained Models

Language Models

Download pretrained Language Models from here

Tokenizer

Unsupervised training using Google's sentencepiece

Download the trained model and vocabulary from here