Tests using tempfile.NamedTemporaryFile fail on Windows 7

[barsch]

Here is in output of unittests:
======================================================================
ERROR: test_empty_file (test_nbt.BugfixTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "D:/Workspace/bravo/bravo/tests\test_nbt.py", line 36, in test_empty_file
self.assertRaises(MalformedFileError, NBTFile, temp.name)
File "c:\Program Files\Python27\Lib\unittest\case.py", line 456, in assertRaises
callableObj(_args, *_kwargs)
File "d:\workspace\bravo\bravo\nbt.py", line 281, in init
self.file = GzipFile(filename, mode)
File "c:\Program Files\Python27\Lib\gzip.py", line 89, in init
fileobj = self.myfileobj = builtin.open(filename, mode or 'rb')
IOError: [Errno 13] Permission denied: 'c:\users\barsch\appdata\local\temp\tmp2eq3fb'

======================================================================
ERROR: testReadBig (test_nbt.ReadWriteTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:/Workspace/bravo/bravo/tests\test_nbt.py", line 49, in testReadBig
    mynbt = NBTFile(self.f.name)
  File "d:\workspace\bravo\bravo\nbt.py", line 281, in __init__
    self.file = GzipFile(filename, mode)
  File "c:\Program Files\Python27\Lib\gzip.py", line 89, in __init__
    fileobj = self.myfileobj = __builtin__.open(filename, mode or 'rb')
IOError: [Errno 13] Permission denied: 'c:\\users\\barsch\\appdata\\local\\temp\\tmpvnhfsi'

======================================================================
ERROR: testWriteBig (test_nbt.ReadWriteTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:/Workspace/bravo/bravo/tests\test_nbt.py", line 54, in testWriteBig
    mynbt = NBTFile(self.f.name)
  File "d:\workspace\bravo\bravo\nbt.py", line 281, in __init__
    self.file = GzipFile(filename, mode)
  File "c:\Program Files\Python27\Lib\gzip.py", line 89, in __init__
    fileobj = self.myfileobj = __builtin__.open(filename, mode or 'rb')
IOError: [Errno 13] Permission denied: 'c:\\users\\barsch\\appdata\\local\\temp\\tmpp4xikw'

======================================================================
ERROR: testWriteback (test_nbt.ReadWriteTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:/Workspace/bravo/bravo/tests\test_nbt.py", line 60, in testWriteback
    mynbt = NBTFile(self.f.name)
  File "d:\workspace\bravo\bravo\nbt.py", line 281, in __init__
    self.file = GzipFile(filename, mode)
  File "c:\Program Files\Python27\Lib\gzip.py", line 89, in __init__
    fileobj = self.myfileobj = __builtin__.open(filename, mode or 'rb')
IOError: [Errno 13] Permission denied: 'c:\\users\\barsch\\appdata\\local\\temp\\tmpk1tri7'

[barsch]

changing e.g.:

def test_empty_file(self):
    """
    Opening an empty file causes an exception.

    https://github.com/twoolie/NBT/issues/issue/4
    """

    temp = tempfile.NamedTemporaryFile()
    temp.flush()
    self.assertRaises(MalformedFileError, NBTFile, temp.name)

to

def test_empty_file(self):
    """
    Opening an empty file causes an exception.

    https://github.com/twoolie/NBT/issues/issue/4
    """

    temp = tempfile.NamedTemporaryFile(delete=False)
    temp.flush()
    temp.close()
    self.assertRaises(MalformedFileError, NBTFile, temp.name)
    os.unlink(temp.name)

works for Python >= 2.6.x (See also: http://bugs.python.org/issue6357).

However in another project supporting Python 2.5.5 - Python 3.1.3 we use this utility function instead the original class for all tests using temporary files - but file handlers must be closed manually within the tests:

def NamedTemporaryFile(dir=None, suffix='.tmp'):
    """
    Weak replacement for the Python class :class:`tempfile.NamedTemporaryFile`.

    This will work also with Windows Vista's UAC.

    .. warning::
        The calling program is responsible to close the returned file pointer
        after usage.
    """

    class NamedTemporaryFile(object):

        def __init__(self, fd, fname):
            self._fileobj = os.fdopen(fd, 'w+b')
            self.name = fname

        def __getattr__(self, attr):
            return getattr(self._fileobj, attr)

    return NamedTemporaryFile(*tempfile.mkstemp(dir=dir, suffix=suffix))

[MostAwesomeDude]

Hm. The entire point of the tempfile stuff there is to ensure that the OS is not in our way when doing things. However, it looks like Win7 is bound and determined to get in our way. Perhaps we should consider this a Python bug and not our bug.

[mattbasta]

Windows does not allow processes other than the one used to create the NamedTemporaryFile to access the file when using delete=True (the default).

The fix, in this case, is to set delete=False and just remember to do the following to clean up the temp file:

try:
    temp.close()
    os.unlink(temp.name)
except:
    pass

[MostAwesomeDude]

Ugh. This is a trial bug, then? Joy... Will see if Twisted has a bug for this already.

[barsch]

nope - its still a problem in Python tempfile.NamedTemporaryFile

[MostAwesomeDude]

That doesn't seem like a Python bug since Python does not guarantee that temp files will be accessible across process boundaries, and in fact, it seems quite reasonable that this is the case. I'd rather blame trial, because of the way trial sandboxes things. Either way, I'm creating a "Windows" tag for these bugs, since they can't really be easily diagnosed on my end.

I'm going to send a new patch to Twisted for http://twistedmatrix.com/trac/ticket/3931, which will hopefully fix this issue for Twisted 11.0 whenever it does come out. In the meantime, I don't want to patch those tests just for Windows, especially when the temp file usage is only in the tests and not in the actual code. (And also when NBT wouldn't even have this problem if it used FilePath instead of bare file handles.)

[barsch]

You actually don't need trial to retrieve the same errors - just run: python bravo\tests\test_nbt.py. So imho its clearly Python related.

Calling a explicit close() on the file handler of the temp file solves the issue for Python >= 2.6.x and I personally would forget about 2.5.x as this will be replaced sooner or later by more recent versions.

[mattbasta]

It's an issue with Python, but it's not a bug. They clearly document this in the docs:

Whether the name can be used to open the file a second time, while
the named temporary file is still open, varies across platforms (it can
be so used on Unix; it cannot on Windows NT or later).

http://docs.python.org/library/tempfile.html#tempfile.NamedTemporaryFile