Predicts garbage for Bengali input

[hafiz031]

I am trying this lookup_compound | Keep original casing example on a Bengali corpus of unigrams and bigrams. As a separator I have used comma. But it seems to be not working. For any misspelled input it is just outputting garbage string.

[mammothb]

Do you a sample code snippet which can reproduce your error? If possible, can you also provide the dictionary files you have used? You can probably just use a snippet of the dictionary if the file size is too large.

[hafiz031]

@mammothb here you go!
unigrams.txt

bigrams.txt

[hafiz031]

@mammothb for example try to correct: রিসেট (it is already correct btw.) but it will split all the letters and also eliminate some characters and output will be like: র স ট

[mammothb]

Have you tried setting split_by_space=True in lookup_compound? I was able to get the same output as input with the following code

from pathlib import Path

from symspellpy import SymSpell

sym_spell = SymSpell(max_dictionary_edit_distance=2, prefix_length=7)

sym_spell.load_dictionary(
    Path(__file__).resolve().parent / "unigrams.txt",
    term_index=0,
    count_index=1,
    separator=",",
    encoding="utf-8",
)
sym_spell.load_bigram_dictionary(
    Path(__file__).resolve().parent / "bigrams.txt",
    term_index=0,
    count_index=1,
    separator=",",
    encoding="utf-8",
)

input_term = "রিসেট"
suggestions = sym_spell.lookup_compound(
    input_term, max_edit_distance=2, split_by_space=True
)
for suggestion in suggestions:
    print(suggestion)
print(input_term)

Output:

রিসেট, 0, 1485
রিসেট

[hafiz031]

@mammothb no, I just used the example as it is in the documentation... I didn’t change any parameters.

[mammothb]

Try and see if split_by_space=True works for you. The default split uses regex which I think doesn't work well with Bengali

[hafiz031]

@mammothb yes it works, thanks!