[Suggestion] Add a large dictionary data

[konbraphat51]

Detailed description

I would suggest adding a large (300K+ sized) dictionary for better tokenization performance.

Context

I am currently doing text mining of Pantip. Pantip has a lot of new(?) words and proper-nouns that pythainlp.corpus.common.thai_words() couldn't catch.
But when I added new words from

• Volubilis Dictionary for new words
• Titles of wikipedia-Th articles for proper-nouns

and the performance improved by around 10%. The dictionary became 300K words in total.

I guess that it could be useful if this large dictionary data is easily (just import from pythainlp modules) available for other users too.

Possible implementation

Simply make a dictionary data from the sources above and serve it as pythainlp.corpus.common.thai_words_large() or something (Since dynamically downloading from the sources above could be a burden for the providers)

[wannaphong]

I agree but Titles of wikipedia-Th articles may does not enough good for word list because Thai wikipedia use long text that can segment and many titles are not named entity. Example: https://th.wikipedia.org/wiki/รายชื่อปฏิบัติการทางทหารระหว่างสงครามโลกครั้งที่สอง/

[konbraphat51]

Hmm, true…
Is there any handful pythainlp functions to detect non named entity?

[wannaphong]

Hmm, true… Is there any handful pythainlp functions to detect non named entity?

Yes, PyThaiNLP has named entity recognition but maybe can't detect non named entity for a word. I think you can start by uses deepcut to count words if more than 3 (or more) words, remove the word.

[bact]

I guess some categories (like "รายชื่อxxx") should be easy to filter out by just pattern match.

[konbraphat51]

I will check what patterns of the wikipedia titles exists.
Please wait for a while

[konbraphat51]

Found this: https://th.wikipedia.org/wiki/%E0%B8%A7%E0%B8%B4%E0%B8%81%E0%B8%B4%E0%B8%9E%E0%B8%B5%E0%B9%80%E0%B8%94%E0%B8%B5%E0%B8%A2:%E0%B8%8A%E0%B8%B7%E0%B9%88%E0%B8%AD%E0%B8%9A%E0%B8%97%E0%B8%84%E0%B8%A7%E0%B8%B2%E0%B8%A1

I made a plan to clean the Wikipedia titles data: https://github.com/konbraphat51/Thai_Dictionary_Cleaner/blob/wiki/Wikipedia.ipynb
How this seems?

[bact]

Looks great.

Those titles with parenthesis: "XXX (YYY)" may able to split to two entries: "XXX", "YYY" (where XXX and YYY may duplicated with other existing entries; in that case, just keep one).

[konbraphat51]

I tried to cleaned wikipedia data. How this seems?
https://github.com/konbraphat51/Thai_Dictionary_Cleaner/blob/wiki/Wikipedia/wikipedia_nlp.txt

This is not 100% perfect but could be worth using

https://github.com/konbraphat51/Thai_Dictionary_Cleaner/blob/wiki/Wikipedia/Wikipedia.ipynb

[wannaphong]

I tried to cleaned wikipedia data. How this seems? https://github.com/konbraphat51/Thai_Dictionary_Cleaner/blob/wiki/wikipedia_nlp.txt

This is not 100% perfect but could be worth using

notebook: https://github.com/konbraphat51/Thai_Dictionary_Cleaner/blob/wiki/Wikipedia.ipynb

It look good for me. @bact What do you think?

[konbraphat51]

If seems good, I would make a corpus implementation of these.

How could I handle with the license of the data source?
wikipedia: https://github.com/konbraphat51/Thai_Dictionary_Cleaner/blob/wiki/Wikipedia/LISENCE.md
Volubilis: https://github.com/konbraphat51/Thai_Dictionary_Cleaner/blob/main/Volubilis/LISENCE.md_

[bact]

Great work.
One thing: How we are dealing with digits/numbers at this moment?

I currently see the likes of

• ๑๖กรกฎาคม
• ๓๑มีนาคม
• ๑๘
  in the output word list (wikipedia_nlp.txt)

[wannaphong]

If seems good, I would make a corpus implementation of these.

How could I handle with the license of the data source? wikipedia: https://github.com/konbraphat51/Thai_Dictionary_Cleaner/blob/wiki/Wikipedia/LISENCE.md Volubilis: https://github.com/konbraphat51/Thai_Dictionary_Cleaner/blob/main/Volubilis/LISENCE.md_

I think it can be fair use from USA's law (just use the title that manybe < 1% of all data) for Wikipedia. I think you can change to CC-0 or just CC-BY.

I don't sure about Volubilis.

[bact]

For the Wikipedia title, I don't think we will have any problem to follow the original license. Text in Wikipedia is CC BY-SA. CC BY-SA is an open license and we can use/distribute it as it is.

Volubilis can be tricky. I'm not sure.