Small bugfix for semantic similarity evaluation

[akutuzov]

Handling of non-Latin OOV pairs logging in evaluate_word_pairs function (they were not converted to UTF-8, which caused logger to fail).

[tmylk]

Please ignore Python 2.6 and Python 3.3 build errors. They are due to latest PyEMD update.

The 3.4 and 3.5 errors are due to this PR though.

[akutuzov]

Fixed.

[piskvorky]

This doesn't look right. What "OOV pairs"? pairs is a filesystem path (bad variable name, but that's a separate issue).

What exactly fails and how?

[tmylk]

@akutuzov Python 2.6 and 3.3 failures should be fixed now. Please kindly merge in the changes from develop to this PR.

[akutuzov]

@piskvorky The problem arises when we have non-Latin word pairs in the dataset, and they are absent in the model. The logger prints these pairs on debug level here, and it fails in Python 2. To fix it, I explicitly state that the whole string is Unicode. For Python 3 it's irrelevant, as far as I can see.
The pairs variable indeed is the path to the dataset file (I agree that's not the best variable name). The logger outputs this path in case there are incorrect lines in the dataset. Again, in this PR I explicitly state that this warning is Unicode, to handle rare cases when the path contains non-Latin characters.

[piskvorky]

Ah, I see what you mean -- you're talking about non-ASCII (not non-Latin) characters in line, which is a unicode string.

I'm not very familiar with the .format fancy formatting, but normal "%s" should output them (upcast to unicode) just fine.

I'd prefer the fix to use lazy argument formatting: logger.info("%s", unicode_string). It's easier to read and clearer (and doesn't fail).

Independently: why pairs (the filesystem path) was being converted with .encode('utf8') I don't know. Unless there's an explicit comment that documents this (and there isn't), I'd consider it a bug.

[piskvorky]

@tmylk please review before merging...

[tmylk]

Reviewed and tested simple formatting

[piskvorky]

@akutuzov can you please fix the comments above?

[tmylk]

@akutuzov In particular @piskvorky is referring to lazy formatting.

Lazy: logger.info("%s", unicode_string).

Eager: logger.info("%s" % unicode_string).

[akutuzov]

Fixed in #1084