Text_LanguageDetect

PHP library to identify human languages from text samples. Returns confidence scores for each.

Installation

PEAR

$ pear install Text_LanguageDetect

Composer

$ composer require pear/text_languagedetect

Usage

Also see the examples in the docs/ directory and the official documentation.

Language detection

Simple language detection:

<?php
require_once 'Text/LanguageDetect.php';

$text = 'Was wäre, wenn ich Ihnen das jetzt sagen würde?';

$ld = new Text_LanguageDetect();
$language = $ld->detectSimple($text);

echo $language;
//output: german

Show the three most probable languages with their confidence score:

<?php
require_once 'Text/LanguageDetect.php';

$text = 'Was wäre, wenn ich Ihnen das jetzt sagen würde?';

$ld = new Text_LanguageDetect();
//3 most probable languages
$results = $ld->detect($text, 3);

foreach ($results as $language => $confidence) {
    echo $language . ': ' . number_format($confidence, 2) . "\n";
}

//output:
//german: 0.35
//dutch: 0.25
//swedish: 0.20
?>

Language code

Instead of returning the full language name, ISO 639-2 two and three letter codes can be returned:

<?php
require_once 'Text/LanguageDetect.php';
$ld = new Text_LanguageDetect();

//will output the ISO 639-1 two-letter language code
// "de"
$ld->setNameMode(2);
echo $ld->detectSimple('Das ist ein kleiner Text') . "\n";

//will output the ISO 639-2 three-letter language code
// "deu"
$ld->setNameMode(3);
echo $ld->detectSimple('Das ist ein kleiner Text') . "\n";
?>

Supported languages

• albanian
• arabic
• azeri
• bengali
• bulgarian
• cebuano
• croatian
• czech
• danish
• dutch
• english
• estonian
• farsi
• finnish
• french
• german
• hausa
• hawaiian
• hindi
• hungarian
• icelandic
• indonesian
• italian
• kazakh
• kyrgyz
• latin
• latvian
• lithuanian
• macedonian
• mongolian
• nepali
• norwegian
• pashto
• pidgin
• polish
• portuguese
• romanian
• russian
• serbian
• slovak
• slovene
• somali
• spanish
• swahili
• swedish
• tagalog
• turkish
• ukrainian
• urdu
• uzbek
• vietnamese
• welsh

Links

Homepage

http://pear.php.net/package/Text_LanguageDetect

Bug tracker

http://pear.php.net/bugs/search.php?cmd=display&package_name[]=Text_LanguageDetect

Documentation

http://pear.php.net/package/Text_LanguageDetect/docs

Unit test status

https://travis-ci.org/pear/Text_LanguageDetect

Notes

Where are the data from?

I don't recall where I got the original data set. It's just the frequencies of 3-letter combinations in each supported language. It could be generated from a few random wikipedia pages from each language.