exception while during frosted

[MirkoDziadzka]

Hi

If I run frosted over a bigger codebase, I get the following exception. Unfortunately it does not tell me which file gives the problem (which it should) but simply excepts.

$ frosted --verbose -i E101 -i W101 -i E307 -i E103 -i E301 -i E302 -r .
Traceback (most recent call last):
File "MYPATH/testenv/bin/frosted", line 9, in
load_entry_point('frosted==1.4.1', 'console_scripts', 'frosted')()
File "MYPATH/testenv/lib/python2.7/site-packages/frosted/main.py", line 38, in main
warnings = check_recursive(file_names, *_arguments)
File "MYPATH/testenv/lib/python2.7/site-packages/frosted/api.py", line 152, in check_recursive
warnings += check_path(source_path, reporter, settings_path=None, *_setting_overrides)
File "MYPATH/testenv/lib/python2.7/site-packages/frosted/api.py", line 133, in check_path
return check(codestr, filename, reporter, settings_path, *_setting_overrides)
File "MYPATH/testenv/lib/python2.7/site-packages/frosted/api.py", line 114, in check
w = checker.Checker(tree, filename, None, ignore_lines=_noqa_lines(codeString), *_active_settings)
File "MYPATH/testenv/lib/python2.7/site-packages/frosted/api.py", line 40, in _noqa_lines
g = tokenize.generate_tokens(StringIO(str(codeString)).readline) # tokenize the string
UnicodeDecodeError: 'utf8' codec can't decode byte 0xf8 in position 6783: invalid start byte

[carlio]

I have run into the same problem. I can't speak for @MirkoDziadzka but my problem comes from a non-utf-8 encoded file.

Specifically this line of this file: https://github.com/carlio/Pillow/blob/master/PIL/WalImageFile.py#L78 which is encoded using iso-8859-1.

You can also reproduce this very easily by creating a file with the following contents:

# ü

Then save it with ISO-8859-1 encoding (if you use vim, you can :w ++enc=iso-8859-1 filename.py) and run frosted filename.py.

This does not happen in pyflakes; I wasn't able to figure out exactly why but I think it's just by accident rather than design - in pyflakes the unicode codeString is never manipulated / str'd.