Add fuzzy prefixing tree walking #23
Conversation
There was a problem hiding this comment.
This is a really interesting idea! I’m impressed with how little code it needed but have a few questions from a quick look through:
How well does it work?
It looks like it will do a fuzzy match, but fuzzy matches in a non-trivial corpus quickly lead to a huge amount of noise which then needs at the very least a basic scoring mechanism to make sure the best (by some definition) are shown first.
As far as I can see (please correct if wrong) there is no concept of scoring here so if you fuzzy match with distance threshold 3 the results for “rabix” would deliver “racer” before “radix”.
It looks like since DistFn is pluggable and can modify the value returned you can custom score by emitting for example a tuple (value, score) and then the WalkFn could collect those into a container and sort by score. Is this the intended use? A doc example showing that would be a great addition.
Have you used this in a real case? Did it work well? What was corpus size like?
| @@ -3,13 +3,19 @@ package iradix | |||
| import ( | |||
| "bytes" | |||
| "sort" | |||
|
|
|||
| "code.sajari.com/go-immutable-radix/pkg/levenshtein" | |||
There was a problem hiding this comment.
This looks like a goimports fail - shouldn’t it be importing the levenstein sub package also added here?
There was a problem hiding this comment.
| "code.sajari.com/go-immutable-radix/pkg/levenshtein" | |
| "github.com/hashicorp/go-immutable-radix/pkg/levenshtein" |
Yeah this is my bad. I'd had a few beers and sent the PR a bit prematurely by accident.
| }) | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
The code looks good to me at a glance but test cases to show the behaviour and edge cases seem important!
There was a problem hiding this comment.
Yeah i'm wondering how best to do this? The surface changes are pretty simple. Most of the complexity is in the levenshtein pkg, which is where i focused on the tests. Anyhow happy to add, let me know your thoughts?
|
@banks yes you are correct. The distance function is pluggable as the leaves only act on the I have used this with around 250,000 items in the tree. For this the existing walk iterator takes ~250usec, the fuzzy walk is ~2msec, roughly 10x slower. Our scoring function is pretty complex though, we basically keep a top X based on a combination of the popularity of the key and the probability of the mistake/s made. There's lots of ways to do this, e.g. https://web.stanford.edu/class/cs124/lec/spelling.pdf but in practice even just using the probability of the key works pretty well, but it's better if you can properly penalise mistakes which is part of the reason to pass through the distance. |
|
|
This allows trees to be walked with non-exact prefixes. The Levenshtein distance is used to determine how far the input can deviate, which is highly useful for tasks such as autocompletion or suggestions where the spelling is often not quite correct