tmpdir.join("foo").write(...) doesn't work as expected.

[pytestbot]

Originally reported by: Bjorn Pettersen (BitBucket: thebjorn, GitHub: thebjorn)

---

All of the following testcase

#!python

# -*- coding: utf-8 -*-

def test_1(tmpdir):
    tmpdir.join('foo').write(u'æ')


def test_2(tmpdir):
    tmpdir.join('foo').write(u'æ'.encode('u8'))


def test_3(tmpdir):
    tmpdir.join('foo').write(u'æ'.encode('l1'))

fails with the following errors:

#!python

(dev) w:\>py.test test_tmpdir.py
============================= test session starts =============================
platform win32 -- Python 2.7.8 -- py-1.4.25 -- pytest-2.6.3
plugins: cov, xdist
collected 3 items

test_tmpdir.py FFF

================================== FAILURES ===================================
___________________________________ test_1 ____________________________________

tmpdir = local('c:\\tmp\\pytest-20\\test_10')

    def test_1(tmpdir):
>       tmpdir.join('foo').write(u'æ')

test_tmpdir.py:4:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = local('c:\\tmp\\pytest-20\\test_10\\foo'), data = 'æ', mode = 'w'
ensure = False

    def write(self, data, mode='w', ensure=False):
        """ write data into path.   If ensure is True create
            missing parent directories.
            """
        if ensure:
            self.dirpath().ensure(dir=1)
        if 'b' in mode:
            if not py.builtin._isbytes(data):
                raise ValueError("can only process bytes")
        else:
            if not py.builtin._istext(data):
                if not py.builtin._isbytes(data):
                    data = str(data)
                else:
                    data = py.builtin._totext(data, sys.getdefaultencoding())
        f = self.open(mode)
        try:
>           f.write(data)
E           UnicodeEncodeError: 'ascii' codec can't encode character u'\xe6' in position 0: ordinal not in range(128)

dev\lib\site-packages\py\_path\local.py:476: UnicodeEncodeError
___________________________________ test_2 ____________________________________

tmpdir = local('c:\\tmp\\pytest-20\\test_20')

    def test_2(tmpdir):
>       tmpdir.join('foo').write(u'æ'.encode('u8'))

test_tmpdir.py:8:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = local('c:\\tmp\\pytest-20\\test_20\\foo'), data = '\xc3\xa6', mode = 'w'
ensure = False

    def write(self, data, mode='w', ensure=False):
        """ write data into path.   If ensure is True create
            missing parent directories.
            """
        if ensure:
            self.dirpath().ensure(dir=1)
        if 'b' in mode:
            if not py.builtin._isbytes(data):
                raise ValueError("can only process bytes")
        else:
            if not py.builtin._istext(data):
                if not py.builtin._isbytes(data):
                    data = str(data)
                else:
>                   data = py.builtin._totext(data, sys.getdefaultencoding())
E                   UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)

dev\lib\site-packages\py\_path\local.py:473: UnicodeDecodeError
___________________________________ test_3 ____________________________________

tmpdir = local('c:\\tmp\\pytest-20\\test_30')

    def test_3(tmpdir):
>       tmpdir.join('foo').write(u'æ'.encode('l1'))

test_tmpdir.py:12:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = local('c:\\tmp\\pytest-20\\test_30\\foo'), data = '\xe6', mode = 'w'
ensure = False

    def write(self, data, mode='w', ensure=False):
        """ write data into path.   If ensure is True create
            missing parent directories.
            """
        if ensure:
            self.dirpath().ensure(dir=1)
        if 'b' in mode:
            if not py.builtin._isbytes(data):
                raise ValueError("can only process bytes")
        else:
            if not py.builtin._istext(data):
                if not py.builtin._isbytes(data):
                    data = str(data)
                else:
>                   data = py.builtin._totext(data, sys.getdefaultencoding())
E                   UnicodeDecodeError: 'ascii' codec can't decode byte 0xe6 in position 0: ordinal not in range(128)

dev\lib\site-packages\py\_path\local.py:473: UnicodeDecodeError
========================== 3 failed in 0.19 seconds ===========================

(dev) w:\>

---

• Bitbucket: https://bitbucket.org/pytest-dev/pytest/issue/605

[pytestbot]

Original comment by Alexander Dudko (BitBucket: oktopuz, GitHub: oktopuz):

---

If you call open() in python 2.7, the default 'ascii' codec is used.

According to https://docs.python.org/2/howto/unicode.html:

1. If the code point is < 128, each byte is the same as the value of the code point.
2. If the code point is 128 or greater, the Unicode string can’t be represented in this encoding (Python raises a UnicodeEncodeError exception in this case)

The code point of u'æ' is above 127.

In other words this is expected error that has nothing to do with pytest.

You might want to check https://docs.python.org/2.7/library/codecs.html?highlight=open#codecs.open to work with UTF-8 encoded files.

[pytestbot]

Original comment by Bjorn Pettersen (BitBucket: thebjorn, GitHub: thebjorn):

---

Yes, the code point of æ is above 127, but neither of

#!python


    u'æ'.encode('u8')
    u'æ'.encode('l1')

are Unicode. They've been encoded, so they're byte strings:

#!python

>>> u'æ'.encode('u8') == b'\xc3\xa6'
True
>>> u'æ'.encode('l1') == b'\xe6'
True

Byte strings containing the full 256 possible byte values, are perfectly writable in Python:

#!python


c:\srv>python
Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> open('foo', 'w').write(u'æ')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe6' in position 0: ordinal not in range(128)
>>> open('foo', 'w').write(u'æ'.encode('u8'))
>>> open('foo', 'w').write(u'æ'.encode('l1'))

so the question remains: how do you write a Unicode string to a file in py.test, since the normal method of encoding it to utf-8 or latin-1 does not work?

[pytestbot]

Original comment by Anatoly Bubenkov (BitBucket: bubenkoff, GitHub: bubenkoff):

---

i believe that this problem is already solved in latest py (it's there what was wrong)
also please upgrade pytest to the latest
pip install -U pytest py

[pytestbot]

Original comment by Bjorn Pettersen (BitBucket: thebjorn, GitHub: thebjorn):

---

It looks like this code:

#!python

                    data = str(data)
                else:
                    data = py.builtin._totext(data, sys.getdefaultencoding())
        f = self.open(mode)
        try:
>           f.write(data)
E           UnicodeEncodeError: 'ascii' codec can't encode character u'\xe6' in position 0: ordinal not in range(128)

tries to decode the encoded string back to Unicode using the default encoding, and then writes the newly created Unicode string to f letting the system try to convert (ie. encode) it back to a byte-string before outputting it to disk. In general you shouldn't mess with already encoded data...

[pytestbot]

Original comment by Anatoly Bubenkov (BitBucket: bubenkoff, GitHub: bubenkoff):

---

but did you try latest py first?
your console output tells that you don't have the latest

[pytestbot]

Original comment by Bjorn Pettersen (BitBucket: thebjorn, GitHub: thebjorn):

---

I'm now running py 1.4.26 and pytest 2.6.4 (I believe those are the latest..) and getting the exact same errors.

#!python

(dev) go|c:\srv\tmp\ptb> py.test
================================================== test session starts ==================================================
platform win32 -- Python 2.7.3 -- py-1.4.26 -- pytest-2.6.4
plugins: cov, xdist
collected 3 items

test_foo.py FFF

[pytestbot]

Original comment by Anatoly Bubenkov (BitBucket: bubenkoff, GitHub: bubenkoff):

---

platform win32 -- Python 2.7.8 -- py-1.4.25 -- pytest-2.6.3 so this is outdated in your report?

[pytestbot]

Original comment by Bjorn Pettersen (BitBucket: thebjorn, GitHub: thebjorn):

---

@anatoly yes, that was from my work computer, this is from home.

[pytestbot]

Original comment by Anatoly Bubenkov (BitBucket: bubenkoff, GitHub: bubenkoff):

---

code looks wrong i agree
easy workaround it seems is to pass 'b' mode

[pytestbot]

Original comment by Anatoly Bubenkov (BitBucket: bubenkoff, GitHub: bubenkoff):

---

tmpdir.join('foo').write(u'æ'.encode('u8'), mode='wb')

[pytestbot]

Original comment by Bjorn Pettersen (BitBucket: thebjorn, GitHub: thebjorn):

---

I've also verified the problem on linux:

#!python

(dev)go|~/work/ptb$ py.test
========================================================================================= test session starts ==========================================================================================
platform linux2 -- Python 2.7.3 -- py-1.4.26 -- pytest-2.6.4
plugins: cov, xdist
collected 3 items

test_tdir.py FFF

[pytestbot]

Original comment by Bjorn Pettersen (BitBucket: thebjorn, GitHub: thebjorn):

---

Ah, cool tmpdir.join('foo').write(u'æ'.encode('u8'), mode='wb') works :-)

[pytestbot]

Original comment by Anatoly Bubenkov (BitBucket: bubenkoff, GitHub: bubenkoff):

---

the problem is that it detects your string as not a bytes string, which is strange, we need to fix that detection

[pytestbot]

Original comment by Anatoly Bubenkov (BitBucket: bubenkoff, GitHub: bubenkoff):

---

@oktopuz, default encoding depends on the site.py basically, so your statement is not fully true