Diacritic-Insensitive Search Support (Czech characters)

[janroz]

Hello,

I'm using TNTSearch for a project and encountered an issue with diacritic-insensitive searching. Specifically, the library seems unable to find matches when the search term differs in diacritics (e.g., searching for "OSVC" does not match "OSVČ").

I was wondering if there is a known workaround for this, or if there is a potential for a future update that might address this issue? This functionality is very important for applications dealing with languages using diacritics.

Adding this to loadConfig didn't work:

'charset' => 'utf8'

Thank you for your assistance.

[janroz]

Nobody?

[janroz]

For now I solved it by using EdgeNgramTokenizer, after some tweeks it works well:

'tokenizer' => \TeamTNT\TNTSearch\Support\EdgeNgramTokenizer::class,

[somegooser]

What did you tweek with edgengram?

1. Did you try utf8mb4 as charset?

2. What you also could try is "transliterate". Translate all special characters to default english. Examples:

`
['À' => 'A', 'Á' => 'A', 'Â' => 'A', 'Ã' => 'A', 'È' => 'E', 'É' => 'E']

`

[igor-kamil]

I was also looking for a way to handle searches with and without diacritics, but I didn't find a working solution for now...
So, I tried to make own tokenizer and for now it seems to work nicely:
You can check it out here: AccentInsensitiveTokenizer.php.
To see how it works - check the test AccentInsensitiveTokenizerTest.php.

I hope this might be helpful to others.
(I’m considering opening a PR to add it to the TNTSearch repo, but right now it relies on Laravel helpers.)