Diacritic insensitive search

[igormorgado]

Is possible to do a diacritic insensitive search using tinyDB when data stored contains UTF8 diacritics ?

A test code for better explanation. The idea is a search that handle strings "decoded" using ASCII only representations.

import re
from unidecode import unidecode
from tinydb import TinyDB, Query
from tinydb.storages import MemoryStorage

data = [{ 'name': 'José Munoz',   'age': 1 },
        { 'name': 'josè antunes', 'age': 2 },
        { 'name': 'JOSÉ SILVA',   'age': 3 },
        { 'name': 'jose moreno',  'age': 4 },
        { 'name': 'Jose Cruz',    'age': 5 },
        { 'name': 'Josê Bolanos', 'age': 6 },
        { 'name': 'Josê',         'age': 7 },
        { 'name': 'Luís Cruz',    'age': 8 },
]

def find_by_name(name):
    q = Query()
    return db.search(q.name.matches(name, flags=re.IGNORECASE))


# Create a in-memory database for testing purposes
db = TinyDB(storage=MemoryStorage)
db.insert_multiple(data)

# Finding all 'jose' ignoring cases. Will match 4 and 5
case1 = find_by_name('jose')
# [{'name': 'José Munoz', 'age': 1},
#  {'name': 'JOSÉ SILVA', 'age': 3}]

# Finding all 'josé' ignoring cases. Will match 1 and 3
case2 = find_by_name('josé')
# [{'name': 'José Munoz', 'age': 1},
#  {'name': 'JOSÉ SILVA', 'age': 3}]

# Finding all unidecode('JòsÊ') == 'JosE', leads to case 1 (case insentivity)
case3 = find_by_name(unidecode('JòsÊ'))
# [{'name': 'José Munoz', 'age': 1},
#  {'name': 'JOSÉ SILVA', 'age': 3}]

A way to do it, slightly outside of tinyDB, is to perform the search without the Query mechanism.

def find_by_name_unidecoded(name, db):
    return [r for r in iter(db) if re.search(unidecode(name), unidecode(r['name']), flags=re.IGNORECASE)]

case4 = find_by_name_unidecoded('JòsÊ', db)
# [{'name': 'José Munoz', 'age': 1},
#  {'name': 'josè antunes', 'age': 2},
#  {'name': 'JOSÉ SILVA', 'age': 3},
#  {'name': 'jose moreno', 'age': 4},
#  {'name': 'Jose Cruz', 'age': 5},
#  {'name': 'Josê Bolanos', 'age': 6},
#  {'name': 'Josê', 'age': 7}]

That can be generalized in the following naive (not considering registers without the field).

def match_unidecoded(search, field, db):
    return [r for r in iter(db) if re.search(unidecode(search), unidecode(r[field]), flags=re.IGNORECASE)]

Resulting in

case5 = match_unidecoded('JòsÊ', 'name', db)
# [{'name': 'José Munoz', 'age': 1},
#  {'name': 'josè antunes', 'age': 2},
#  {'name': 'JOSÉ SILVA', 'age': 3},
#  {'name': 'jose moreno', 'age': 4},
#  {'name': 'Jose Cruz', 'age': 5},
#  {'name': 'Josê Bolanos', 'age': 6},
#  {'name': 'Josê', 'age': 7}]

Is possible to add such feature in Query mechanism?

[eugene-eeo]

You can use q.name.test(...), for example:

def normalised_match(value, query):
    return re.search(unidecode(query), unidecode(value), flags=re.IGNORECASE)

q.name.test(normalised_match, 'JòsÊ')

Is possible to add such feature in Query mechanism?

One suggestion maybe is to add a .map() method to the Query object.
That way, you can express these things more naturally:

q.name.map(unidecode).search(...)
# to select where a list field has more than 5 items:
q.list.map(len) > 5

Would that be helpful?
cc @msiemens

[igormorgado]

I think that map() method would be a reasonable approach.

[msiemens]

map would indeed be elegant, but it's not trivial to add as it would work different from regular queries. Regular query functions always return a QueryInstance which can be applied to a document. But map would not work by itself as db.search(q.map(something)) doesn't really make sense - it always needs another query function to be called. In effect it would act like a new kind of query altogether. I'll open an issue to think about how that could work, but until that using q.name.test is how I'd solve this issue.

[msiemens]

See #423

[rewritten]

This might be solved by #445, where arbitrary unary functions can be chained to the query.