thoughts on clustering

[wooters]

This is a cool project. Thanks for making this public!

I was looking over your clustering notebook and thought of a couple of things that might help to improve the distance calculation between candidate words.

• Include contextual information (e.g. previous word)
• In addition to Levenshtein distance, consider adding some form of a Soundex-based distance metric. (Or maybe double metaphone instead of Soundex.)

Anyway, just a couple of quick thoughts. I apologize if you've already considered these things. I hate to presume, but I didn't see them mentioned in your notebook so I thought I'd throw them out there.

Cheers,
Chuck

[colinmorris]

Thanks for bringing this up. I was totally unaware of the world of phonetic algorithms that are out there - I agree that they could be quite useful here. Though it's worth mentioning that some of the most misspelled words are foreign names and loan words that are tricky to spell precisely because they contradict the rules of English pronunciation (e.g. Sriracha, Shyamalan).

Considering previous word(s) would definitely be useful for identifying multi-word expressions like "___ park", "____ syndrome" etc. I was just afraid of the added complexity that would introduce