Spec tests fail with ASCII locale ?

[andychu]

This was reported on the mailing list and also here:

https://lobste.rs/s/obktxo/oil_shell_help_needed_if_you_have_debian/comments/akvbsk#c_akvbsk

sh_spec.py needs to read with a known encoding. *.test.sh should be restricted to utf-8 ?

$ ./spec.sh all
./spec-runner.sh run-cases append 
./spec-runner.sh run-cases arith-context 
./spec-runner.sh run-cases arith 
./spec-runner.sh run-cases array 
./spec-runner.sh run-cases assign 
./spec-runner.sh run-cases assoc-zsh 
./spec-runner.sh run-cases assoc 
./spec-runner.sh run-cases blog1 
Traceback (most recent call last):
  File "./sh_spec.py", line 838, in <module>
    sys.exit(main(sys.argv))
  File "./sh_spec.py", line 766, in main
    tokens = Tokenizer(LineIter(f))
  File "./sh_spec.py", line 125, in __init__
    self.next()
  File "./sh_spec.py", line 129, in next
    self.cursor = self.it.__next__()
  File "./sh_spec.py", line 87, in LineIter
    for i, line in enumerate(f):
  File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 133: ordinal not in range(128)
./spec-runner.sh run-cases brace-expansion 
./spec-runner.sh run-cases bugs 
./spec-runner.sh run-cases builtins 
./spec-runner.sh run-cases case_ 
./spec-runner.sh run-cases command-sub 
./spec-runner.sh run-cases comments 
*** Got 1 allowed osh failures, exit with status 0
./spec-runner.sh run-cases dbracket 
Traceback (most recent call last):
  File "./sh_spec.py", line 838, in <module>
    sys.exit(main(sys.argv))
  File "./sh_spec.py", line 766, in main
    tokens = Tokenizer(LineIter(f))
  File "./sh_spec.py", line 125, in __init__
    self.next()
  File "./sh_spec.py", line 129, in next
    self.cursor = self.it.__next__()
  File "./sh_spec.py", line 87, in LineIter
    for i, line in enumerate(f):
  File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 2750: ordinal not in range(128)
./spec-runner.sh run-cases dparen 

[andychu]

Pretty sure I fixed this recently by using an explicit .encode('utf-8').