🔥 🐛 MLTextTranslatorPipelineItem -- causes issues with the indices in tranlation (batching mode)

[nicolay-r]

Using the following script:

#!/bin/bash
python3 -m arelight.run.infer \
	--sampling-framework "arekit" \
	--ner-model-name "ner_ontonotes_bert_mult" \
	--ner-types "ORG|PERSON|LOC|GPE" \
	--terms-per-context 50 \
	--sentence-parser "nltk:russian" \
	--text-b-type "nli_m" \
	--tokens-per-context 128 \
	--bert-framework "opennre" \
	--batch-size 10 \
	--inference-writer "sqlite3" \
	--stemmer "mystem" \
	--pretrained-bert "DeepPavlov/rubert-base-cased" \
	--bert-torch-checkpoint "ra4-rsr1_DeepPavlov-rubert-base-cased_cls.pth.tar" \
	--backend "d3js_graphs" \
	-o "output" \
	--from-files $1 \
	--log-file "arelight.log.txt"
	#--translate-framework googletrans \
	#--translate-text "en:ru" \

Uncommenting last two lines for launching translator will lead to the incorrect indexing
Google Trans Version:
googletrans==3.1.0a0

There is a need to split translated earlier text parts into words.
For example, here is the representation of the list in which we seek for the related position of indexed named entities:

['Он командовал 50 000 солдатами ', <arelight.pipelines.items.entity.IndexedEntity object at 0x7f6346931760>, ', дислоцированными в провинциях ', <arelight.pipelines.items.entity.IndexedEntity object at 0x7f63469310d0>]

[nicolay-r]

There is a need to refactor the quick fix by adapting flattened iterator and setting up separator for terms

def flatten(xss):
    return [x for xs in xss for x in xs]