identify languages

[matyaskopp]

Identify languages of sentences

• input: is sentence-segmented text
• output: add the attribute xml:lang to every sentence <s>

[matyaskopp]

workflow:

1. segment seg to sentences (temporary element tmpSentence), for id use concat(seg/@xml:id,'.sent',position())
2. extract text content of tmpSentence and ids to tsv file: id text
3. annotate texts with language to tsc file: id text language
4. load languages to tmpSentence elements
5. join adjected tmpSentence with the same language to tmpLangSeg, add ids to it concat(seg/@xml:id,'.lang',position())
6. annotate tmpLangSeg with udpipe and nametag
7. remove tmpLangSeg elements (preserve only content), and set seg language to most common language in paragraph (number of words). note: special treat needs join=right which will need to be removed in some cases
8. backpropagate <seg> language to non-annotated version

[matyaskopp]

3. annotate texts with language to tsc file: id text language

@olgakanishcheva
I have written(with chatGPT help :-)) a simple script for language identification that uses a lingua package:
https://github.com/ufal/ParlaMint-UA/blob/21864363f0b9e9622aa081bdd4d202e2c81ad00c/Scripts/lang-ident.py
can be a starting point for you. The result on the testing data is here:
https://github.com/ufal/ParlaMint-UA/tree/data/DEVEL-tsv-sent-lang

I will use this script in my pipeline until you develop something more sophisticated.

[olgakanishcheva]

@matyaskopp Thank you!