io.open patch failing with numpy's genfromtxt

[jamesbraza]

Describe the bug

It seems numpy.genfromtxt is failing with pyfakefs, it's throwing a FileNotFoundError despite the file actually being present.

How To Reproduce

import csv
import pathlib

import numpy as np
import pytest

THIS_DIRECTORY = pathlib.PurePath(__file__).parent


@pytest.fixture(name="thisdir_fs")
def fixture_thisdir_fs(fs):
    fs.add_real_directory(THIS_DIRECTORY, read_only=False)
    return fs


def test_with_numpy_genfromtxt(thisdir_fs) -> None:
    csv_file = THIS_DIRECTORY / "stub.csv"
    with open(csv_file, mode="w", newline="", encoding="utf-8") as f:
        csv_writer = csv.writer(f)
        csv_writer.writerow(["spam", "ham"])
        csv_writer.writerows([(1, 2), (3, 4)])

    with open(csv_file, newline="", encoding="utf-8") as f:
        assert len(f.readlines()) == 3  # Confirm file was actually made in fake fs

    # FileNotFoundError: [Errno 2] No such file or directory: '/path/to/file/stub.csv'
    file_data = np.genfromtxt(str(csv_file), delimiter=",", skip_header=True)

Running this test will yield the following FileNotFoundError:

file.py::test_with_numpy_genfromtxt FAILED
file.py:15 (test_with_numpy_genfromtxt)
thisdir_fs = <pyfakefs.fake_filesystem.FakeFilesystem object at 0x102fda170>

    def test_with_numpy_genfromtxt(thisdir_fs) -> None:
        csv_file = THIS_DIRECTORY / "stub.csv"
        with open(csv_file, mode="w", newline="", encoding="utf-8") as f:
            csv_writer = csv.writer(f)
            csv_writer.writerow(["spam", "ham"])
            csv_writer.writerows([(1, 2), (3, 4)])
    
        with open(csv_file, newline="", encoding="utf-8") as f:
            assert len(f.readlines()) == 3  # Confirm file was actually made in fake fs
    
        # FileNotFoundError: [Errno 2] No such file or directory: '/path/to/file/stub.csv'
>       file_data = np.genfromtxt(str(csv_file), delimiter=",", skip_header=True)

/path/to/file.py:27: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/path/to/venv/lib/python3.10/site-packages/numpy/lib/npyio.py:1977: in genfromtxt
    fid = np.lib._datasource.open(fname, 'rt', encoding=encoding)
/path/to/venv/lib/python3.10/site-packages/numpy/lib/_datasource.py:193: in open
    return ds.open(path, mode, encoding=encoding, newline=newline)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <numpy.DataSource object at 0x10335a800>
path = '/path/to/file/stub.csv', mode = 'rt', encoding = None
newline = None

    def open(self, path, mode='r', encoding=None, newline=None):
        """
        Open and return file-like object.
    
        If `path` is an URL, it will be downloaded, stored in the
        `DataSource` directory and opened from there.
    
        Parameters
        ----------
        path : str
            Local file path or URL to open.
        mode : {'r', 'w', 'a'}, optional
            Mode to open `path`.  Mode 'r' for reading, 'w' for writing,
            'a' to append. Available modes depend on the type of object
            specified by `path`. Default is 'r'.
        encoding : {None, str}, optional
            Open text file with given encoding. The default encoding will be
            what `io.open` uses.
        newline : {None, str}, optional
            Newline to use when reading text file.
    
        Returns
        -------
        out : file object
            File object.
    
        """
    
        # TODO: There is no support for opening a file for writing which
        #       doesn't exist yet (creating a file).  Should there be?
    
        # TODO: Add a ``subdir`` parameter for specifying the subdirectory
        #       used to store URLs in self._destpath.
    
        if self._isurl(path) and self._iswritemode(mode):
            raise ValueError("URLs are not writeable")
    
        # NOTE: _findfile will fail on a new file opened for writing.
        found = self._findfile(path)
        if found:
            _fname, ext = self._splitzipext(found)
            if ext == 'bz2':
                mode.replace("+", "")
>           return _file_openers[ext](found, mode=mode,
                                      encoding=encoding, newline=newline)
E           FileNotFoundError: [Errno 2] No such file or directory: '/path/to/file/stub.csv'

/path/to/venv/lib/python3.10/site-packages/numpy/lib/_datasource.py:530: FileNotFoundError

Your environment

> python -c "import platform; print(platform.platform())"
macOS-12.6-arm64-arm-64bit
> python -c "import sys; print('Python', sys.version)"
Python 3.10.9 (main, Jan  6 2023, 11:19:20) [Clang 14.0.0 (clang-1400.0.29.102)]
> python -c "from pyfakefs import __version__; print('pyfakefs', __version__)"
pyfakefs 5.2.2
> python -c "import pytest; print('pytest', pytest.__version__)"
pytest 7.3.1
> python -c "import numpy; print('numpy', numpy.__version__)"
numpy 1.24.3

[jamesbraza]

Okay, root causing further, in numpy, this line:

return _file_openers[ext](found, mode=mode,
                          encoding=encoding, newline=newline)

ext is None, and _file_openers[ext] is:

In [1]: import io
In [2]: _file_openers[ext]
Out[2]: <function io.open(file, mode='r', buffering=-1, encoding=None, errors=None, newline=None, closefd=True, opener=None)>
In [3]: io.open
Out[3]: <bound method FakeIoModule.open of <pyfakefs.fake_io.FakeIoModule object at 0x115b7ac50>>

So for some reason, pyfakefs is not properly patching out io.open, which is stored here: https://github.com/numpy/numpy/blob/v1.24.3/numpy/lib/_datasource.py#L101

---

Manual Patching

This works, properly patching in FakeIoModule.open:

import io
from unittest.mock import patch

expected_file_data = np.array([[1.0, 2.0], [3.0, 4.0]])
with patch("numpy.lib._datasource._file_openers", {None: io.open}):
    file_data = np.genfromtxt(str(csv_file), delimiter=",", skip_header=True)
np.testing.assert_allclose(file_data, expected_file_data)

[jamesbraza]

Okay, I renamed this issue to more reflect the root cause. I guess the new ask is for pyfakefs to figure out how to patch io.open in a more general manner.

I am unsure if this is possible without investigating how pyfakefs does its patching, so it's okay if this issue gets closed as "we can't fix this".

[mrbean-bremen]

Thanks - I will have a look at this. May take some time, as there are a couple of other issues I want to address first...

[mrbean-bremen]

Ok, I had a closer look, and found the problem. This is a common problem with global variables initialized before patching starts, in this case this happens while importing numpy. On importing numpy, the internal module lib._datasource is also imported, which initializes a global _FileOpeners instance. This sets the default opener to io.open, which at this point is not patched.

There is no really nice way to address this, but there are a couple of possibilities.
Easiest, but most fragile, is ensuring that numpy is only loaded after the patching has started, e.g. inside the test. In your case, you would have to remove the global import and move it into the test:

def test_with_numpy_genfromtxt(thisdir_fs) -> None:
    ...
    import numpy as np
    file_data = np.genfromtxt(str(csv_file), delimiter=",", skip_header=True)

This works if you make sure that numpy is not imported before that, and is awkward if you have many tests using it.

Another possibility is to reload the problematic module:

import importlib

@pytest.fixture(name="thisdir_fs")
def fixture_thisdir_fs(fs):
    importlib.reload(np.lib._datasource)
    fs.add_real_directory(THIS_DIRECTORY, read_only=False)
    return fs

This works, though you have to rely on an internal numpy module, and may break with some future version. Also, you can't completely exclude that this has some unwanted side effects.

The same can also be done if using the modules_to_reload argument:

from pyfakefs.fake_filesystem_unittest import Patcher

@pytest.fixture(name="thisdir_fs")
def fixture_thisdir_fs():
    with Patcher(modules_to_reload=[np.lib._datasource]) as p:
        p.fs.add_real_directory(THIS_DIRECTORY, read_only=False)
        yield p.fs

which would achieve the same effect.

A somewhat similar, but more specific version would be to patch the file openers directly:

from unittest import mock

@pytest.fixture(name="thisdir_fs")
def fixture_thisdir_fs(fs):
    with mock.patch(f"{__name__}.np.lib._datasource._file_openers") as fo:
        fo._file_openers = {None: io.open}
        fs.add_real_directory(THIS_DIRECTORY, read_only=False)
        yield fs

though this depends on even more internals.

I'll move this to discussions, as this is something that could happen elsewhere.

[jamesbraza]

Thank you for the detailed response!

Yeah I am familiar with this scenario of having to import modules within a test case (after patching becomes active) or using importlib.reload. It's also good to know that pyfakefs directly supports reloading via modules_to_reload. I think your explanation was great!

---

Fwiw, it looks like two weeks ago (late May 2023), numpy actually changed this: numpy/numpy@7ca502e

Now the next release after 1.24.3 will use the built-in open, instead of io.open. It doesn't change the issue here, just sharing for posterity. Anyways, thank you, and case closed!