Difference between cdist Vs cpdist

[mouhssineB]

hello,
could you please let what's the difference between the 2 functions (cdist , cpdist) ? ( in the documentation they seem's to be the same except the same lenght of the list to compare)
is cpdist supose to be much faster than cdist ?

thank you very much

[maxbachmann]

cdist compares all queries with all choices. So for cdist([a, b], [c, d]) it would compare:

a-c
a-d
b-c
b-d

cpdist on the other hand only compares elements in the same position. So for cpdist([a, b], [c, d]) it would compare:

a-c
b-d

[mouhssineB]

thank you very much.

i have to fuzzy-compare about 15 000 000 the full name person with 10 000 000 full name persons. and keep only couples that distance is greater date 75%.

cdist is very good in this job. but the volume that i progress is huge.

i tried to split my inputs, and several stuff , like normalization , make filter to reduce the volume ...

do you know if is there other tips to accelerate the comparison ? parameters of cdist ? more resources ? other approche ?

[maxbachmann]

In terms of cdist things to do are:

1. you will need to process in chunks, since otherwise it would require to much memory for the results table
2. specify the score_cutoff argument
3. set the workers argument to use multithreading (e.g. workers=-1 to use all available cores or workers=n for a specific thread count)
4. be mindful of the scorer you use. E.g. fuzz.ratio is pretty fast, while something like fuzz.partial_ratio is a lot slower. Obviously this depends on what the requirements for the results are

cdist is really fast at comparisons, but obviously with you amount of comparisons it's still going to take quite a while. This depends a bit on how often you have to perform this task. If it's a one of thing it might be fine for the comparison to take a day. However if you need to perform similar comparisons more often, there are some addition things that can be done, that are currently out of scope for rapidfuzz itself. Some that come to my mind are:

• filter out elements ahead of time that can't be the match
• if you are e.g. only interested in elements with e.g a Levenshtein distance <= 2, Levenshtein automata could be an option

[mouhssineB]

Thank you very much for the advise.
i have already implemented some of them. i will try the others.
have a good day