Fix the out of alphabet token handling in analyses generation

Solves apertium#45 Consider alphanumeric characters to be part of the vocabulary.
AMR-KELEG · Jun 20, 2019 · 3129368 · 3129368
1 parent 46f5737
commit 3129368
Showing 1 changed file with 1 addition and 1 deletion.
diff --git a/lttoolbox/fst_processor.cc b/lttoolbox/fst_processor.cc
@@ -837,7 +837,7 @@ FSTProcessor::isEscaped(wchar_t const c) const
 bool
 FSTProcessor::isAlphabetic(wchar_t const c) const
 {
- return alphabetic_chars.find(c) != alphabetic_chars.end();
+ return (bool)std::iswalnum(c) || alphabetic_chars.find(c) != alphabetic_chars.end();
 }
 
 void