nersuite_tokenizer produces wrong results on wide character types (Unicode, UTFs)

[fnl]

There is a "Unicode-bug" in nersuite_common/tokenizer.cpp, Tokenizer::find_token_end: If the isalnum(int) test inside that method fails, the token created is always one single byte wide (because it then returns beg + 1, and beg is a size_t).
This means that any multibyte encoded texts, such as all UTFs, cannot be correctly tokenized by this tool (with the logical exception of ASCII-only containing UTF-8, naturally) because it splits [more than one byte] wide characters in two or more tokens.
This is even nastier in the case of UTF-8 encoded text, because the bug is non-obvious and only becomes apparent when special characters like non-ASCII dashes or Greek letters are present in the text.

[priancho]

Thank you for your bug-report.

At the beginning of developing this application, we used a pre-processing program that converts Unicode characters to Ascii characters.
It is not completely same program but you can find one from https://github.com/spyysalo/unicode2ascii

I also would like to make NERsuite to handle multibyte input since non-ascii characters virtually appear in all biomedical texts.
Unfortunately, it will take some time, at least a few months, to make a time for this improvement because I am currently preparing my thesis defense presentation that will be at the beginning of Feb.

Best wishes,

[fnl]

Thanks for the reply and the link to a (hopefully, temporary) work-around.
Good luck with the defense!