Improve isolation of tests for flit install

[kloczek]

I'm packaging your module as an rpm package so I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

• python3 -sBm build -w --no-isolation
• because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
• install .whl file in </install/prefix>
• run pytest with PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>

Here is pytest output:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-flit-3.8.0-6.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-flit-3.8.0-6.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.15, pytest-7.1.3, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/flit-3.8.0
collected 205 items

flit_core/flit_core/tests/test_build_thyself.py ...                                                                                                                  [  1%]
flit_core/flit_core/tests/test_buildapi.py .........                                                                                                                 [  5%]
flit_core/flit_core/tests/test_common.py ....................                                                                                                        [ 15%]
flit_core/flit_core/tests/test_config.py .............................................                                                                               [ 37%]
flit_core/flit_core/tests/test_sdist.py ......                                                                                                                       [ 40%]
flit_core/flit_core/tests/test_versionno.py .                                                                                                                        [ 40%]
flit_core/flit_core/tests/test_wheel.py .....                                                                                                                        [ 43%]
tests/test_build.py .....                                                                                                                                            [ 45%]
tests/test_command.py ..                                                                                                                                             [ 46%]
tests/test_config.py .                                                                                                                                               [ 47%]
tests/test_find_python_executable.py .......                                                                                                                         [ 50%]
tests/test_init.py ..................                                                                                                                                [ 59%]
tests/test_install.py ...F..F..........F.F........                                                                                                                   [ 73%]
tests/test_sdist.py ............                                                                                                                                     [ 79%]
tests/test_tomlify.py .                                                                                                                                              [ 79%]
tests/test_upload.py .......                                                                                                                                         [ 82%]
tests/test_validate.py ................                                                                                                                              [ 90%]
tests/test_vcs.py .                                                                                                                                                  [ 91%]
tests/test_wheel.py ..................                                                                                                                               [100%]

================================================================================= FAILURES =================================================================================
____________________________________________________________________ InstallTests.test_install_data_dir ____________________________________________________________________

self = <tests.test_install.InstallTests testMethod=test_install_data_dir>

    def test_install_data_dir(self):
>       Installer.from_ini_path(
            core_samples_dir / 'with_data_dir' / 'pyproject.toml',
        ).install_directly()

tests/test_install.py:296:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
flit/install.py:310: in install_directly
    self.install_requirements()
flit/install.py:263: in install_requirements
    check_call(cmd)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

popenargs = (['/usr/bin/python3', '-m', 'pip', 'install', '--user', '-r', ...],), kwargs = {}, retcode = 1
cmd = ['/usr/bin/python3', '-m', 'pip', 'install', '--user', '-r', ...]

    def check_call(*popenargs, **kwargs):
        """Run command with arguments.  Wait for command to complete.  If
        the exit code was zero then return, otherwise raise
        CalledProcessError.  The CalledProcessError object will have the
        return code in the returncode attribute.

        The arguments are the same as for the call function.  Example:

        check_call(["ls", "-l"])
        """
        retcode = call(*popenargs, **kwargs)
        if retcode:
            cmd = kwargs.get("args")
            if cmd is None:
                cmd = popenargs[0]
>           raise CalledProcessError(retcode, cmd)
E           subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'pip', 'install', '--user', '-r', '/tmp/tmp7kwihz9srequirements.txt']' returned non-zero exit status 1.

/usr/lib64/python3.8/subprocess.py:364: CalledProcessError
--------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
Requirement already satisfied: requests>=2.18 in /usr/lib/python3.8/site-packages (from -r /tmp/tmp7kwihz9srequirements.txt (line 1)) (2.28.1)
Requirement already satisfied: docutils in /usr/lib/python3.8/site-packages (from -r /tmp/tmp7kwihz9srequirements.txt (line 2)) (0.18.1)
Requirement already satisfied: idna>=2.5 in /usr/lib/python3.8/site-packages (from requests>=2.18->-r /tmp/tmp7kwihz9srequirements.txt (line 1)) (3.4)
Requirement already satisfied: charset-normalizer>=2 in /usr/lib/python3.8/site-packages (from requests>=2.18->-r /tmp/tmp7kwihz9srequirements.txt (line 1)) (3.0.0)
Requirement already satisfied: urllib3>=1.21.1 in /usr/lib/python3.8/site-packages (from requests>=2.18->-r /tmp/tmp7kwihz9srequirements.txt (line 1)) (1.26.12)
--------------------------------------------------------------------------- Captured stderr call ---------------------------------------------------------------------------
ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/home/tkloczko/.local/lib/python3.8/site-packages'
Check the permissions.

_________________________________________________________________ InstallTests.test_install_module_pep621 __________________________________________________________________

self = <tests.test_install.InstallTests testMethod=test_install_module_pep621>

    def test_install_module_pep621(self):
>       Installer.from_ini_path(
            core_samples_dir / 'pep621_nodynamic' / 'pyproject.toml',
        ).install_directly()

tests/test_install.py:60:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
flit/install.py:310: in install_directly
    self.install_requirements()
flit/install.py:263: in install_requirements
    check_call(cmd)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

popenargs = (['/usr/bin/python3', '-m', 'pip', 'install', '--user', '-r', ...],), kwargs = {}, retcode = 1
cmd = ['/usr/bin/python3', '-m', 'pip', 'install', '--user', '-r', ...]

    def check_call(*popenargs, **kwargs):
        """Run command with arguments.  Wait for command to complete.  If
        the exit code was zero then return, otherwise raise
        CalledProcessError.  The CalledProcessError object will have the
        return code in the returncode attribute.

        The arguments are the same as for the call function.  Example:

        check_call(["ls", "-l"])
        """
        retcode = call(*popenargs, **kwargs)
        if retcode:
            cmd = kwargs.get("args")
            if cmd is None:
                cmd = popenargs[0]
>           raise CalledProcessError(retcode, cmd)
E           subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'pip', 'install', '--user', '-r', '/tmp/tmpfoibwf0erequirements.txt']' returned non-zero exit status 1.

/usr/lib64/python3.8/subprocess.py:364: CalledProcessError
--------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
Requirement already satisfied: requests>=2.18 in /usr/lib/python3.8/site-packages (from -r /tmp/tmpfoibwf0erequirements.txt (line 1)) (2.28.1)
Requirement already satisfied: docutils in /usr/lib/python3.8/site-packages (from -r /tmp/tmpfoibwf0erequirements.txt (line 2)) (0.18.1)
Requirement already satisfied: idna>=2.5 in /usr/lib/python3.8/site-packages (from requests>=2.18->-r /tmp/tmpfoibwf0erequirements.txt (line 1)) (3.4)
Requirement already satisfied: urllib3>=1.21.1 in /usr/lib/python3.8/site-packages (from requests>=2.18->-r /tmp/tmpfoibwf0erequirements.txt (line 1)) (1.26.12)
Requirement already satisfied: charset-normalizer>=2 in /usr/lib/python3.8/site-packages (from requests>=2.18->-r /tmp/tmpfoibwf0erequirements.txt (line 1)) (3.0.0)
--------------------------------------------------------------------------- Captured stderr call ---------------------------------------------------------------------------
ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/home/tkloczko/.local/lib/python3.8/site-packages'
Check the permissions.

---------------------------------------------------------------------------- Captured log call -----------------------------------------------------------------------------
WARNING  flit_core.versionno:versionno.py:124 Version number normalised: '0.03' -> '0.3' (see PEP 440)
____________________________________________________________________ InstallTests.test_symlink_data_dir ____________________________________________________________________

self = <tests.test_install.InstallTests testMethod=test_symlink_data_dir>

    def test_symlink_data_dir(self):
        if os.name == 'nt':
            raise SkipTest("symlink")
>       Installer.from_ini_path(
            core_samples_dir / 'with_data_dir' / 'pyproject.toml', symlink=True
        ).install_directly()

tests/test_install.py:305:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
flit/install.py:310: in install_directly
    self.install_requirements()
flit/install.py:263: in install_requirements
    check_call(cmd)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

popenargs = (['/usr/bin/python3', '-m', 'pip', 'install', '--user', '-r', ...],), kwargs = {}, retcode = 1
cmd = ['/usr/bin/python3', '-m', 'pip', 'install', '--user', '-r', ...]

    def check_call(*popenargs, **kwargs):
        """Run command with arguments.  Wait for command to complete.  If
        the exit code was zero then return, otherwise raise
        CalledProcessError.  The CalledProcessError object will have the
        return code in the returncode attribute.

        The arguments are the same as for the call function.  Example:

        check_call(["ls", "-l"])
        """
        retcode = call(*popenargs, **kwargs)
        if retcode:
            cmd = kwargs.get("args")
            if cmd is None:
                cmd = popenargs[0]
>           raise CalledProcessError(retcode, cmd)
E           subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'pip', 'install', '--user', '-r', '/tmp/tmp1nztui2qrequirements.txt']' returned non-zero exit status 1.

/usr/lib64/python3.8/subprocess.py:364: CalledProcessError
--------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
Requirement already satisfied: requests>=2.18 in /usr/lib/python3.8/site-packages (from -r /tmp/tmp1nztui2qrequirements.txt (line 1)) (2.28.1)
Requirement already satisfied: docutils in /usr/lib/python3.8/site-packages (from -r /tmp/tmp1nztui2qrequirements.txt (line 2)) (0.18.1)
Requirement already satisfied: idna>=2.5 in /usr/lib/python3.8/site-packages (from requests>=2.18->-r /tmp/tmp1nztui2qrequirements.txt (line 1)) (3.4)
Requirement already satisfied: urllib3>=1.21.1 in /usr/lib/python3.8/site-packages (from requests>=2.18->-r /tmp/tmp1nztui2qrequirements.txt (line 1)) (1.26.12)
Requirement already satisfied: charset-normalizer>=2 in /usr/lib/python3.8/site-packages (from requests>=2.18->-r /tmp/tmp1nztui2qrequirements.txt (line 1)) (3.0.0)
--------------------------------------------------------------------------- Captured stderr call ---------------------------------------------------------------------------
ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/home/tkloczko/.local/lib/python3.8/site-packages'
Check the permissions.

_________________________________________________________________ InstallTests.test_symlink_module_pep621 __________________________________________________________________

self = <tests.test_install.InstallTests testMethod=test_symlink_module_pep621>

    def test_symlink_module_pep621(self):
        if os.name == 'nt':
            raise SkipTest("symlink")
>       Installer.from_ini_path(
            core_samples_dir / 'pep621_nodynamic' / 'pyproject.toml', symlink=True
        ).install_directly()

tests/test_install.py:160:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
flit/install.py:310: in install_directly
    self.install_requirements()
flit/install.py:263: in install_requirements
    check_call(cmd)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

popenargs = (['/usr/bin/python3', '-m', 'pip', 'install', '--user', '-r', ...],), kwargs = {}, retcode = 1
cmd = ['/usr/bin/python3', '-m', 'pip', 'install', '--user', '-r', ...]

    def check_call(*popenargs, **kwargs):
        """Run command with arguments.  Wait for command to complete.  If
        the exit code was zero then return, otherwise raise
        CalledProcessError.  The CalledProcessError object will have the
        return code in the returncode attribute.

        The arguments are the same as for the call function.  Example:

        check_call(["ls", "-l"])
        """
        retcode = call(*popenargs, **kwargs)
        if retcode:
            cmd = kwargs.get("args")
            if cmd is None:
                cmd = popenargs[0]
>           raise CalledProcessError(retcode, cmd)
E           subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'pip', 'install', '--user', '-r', '/tmp/tmpkabqdk4prequirements.txt']' returned non-zero exit status 1.

/usr/lib64/python3.8/subprocess.py:364: CalledProcessError
--------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
Requirement already satisfied: requests>=2.18 in /usr/lib/python3.8/site-packages (from -r /tmp/tmpkabqdk4prequirements.txt (line 1)) (2.28.1)
Requirement already satisfied: docutils in /usr/lib/python3.8/site-packages (from -r /tmp/tmpkabqdk4prequirements.txt (line 2)) (0.18.1)
Requirement already satisfied: idna>=2.5 in /usr/lib/python3.8/site-packages (from requests>=2.18->-r /tmp/tmpkabqdk4prequirements.txt (line 1)) (3.4)
Requirement already satisfied: charset-normalizer>=2 in /usr/lib/python3.8/site-packages (from requests>=2.18->-r /tmp/tmpkabqdk4prequirements.txt (line 1)) (3.0.0)
Requirement already satisfied: urllib3>=1.21.1 in /usr/lib/python3.8/site-packages (from requests>=2.18->-r /tmp/tmpkabqdk4prequirements.txt (line 1)) (1.26.12)
--------------------------------------------------------------------------- Captured stderr call ---------------------------------------------------------------------------
ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/home/tkloczko/.local/lib/python3.8/site-packages'
Check the permissions.

---------------------------------------------------------------------------- Captured log call -----------------------------------------------------------------------------
WARNING  flit_core.versionno:versionno.py:124 Version number normalised: '0.03' -> '0.3' (see PEP 440)
========================================================================= short test summary info ==========================================================================
FAILED tests/test_install.py::InstallTests::test_install_data_dir - subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'pip', 'install', '--user', '-r', ...
FAILED tests/test_install.py::InstallTests::test_install_module_pep621 - subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'pip', 'install', '--user', '...
FAILED tests/test_install.py::InstallTests::test_symlink_data_dir - subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'pip', 'install', '--user', '-r', ...
FAILED tests/test_install.py::InstallTests::test_symlink_module_pep621 - subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'pip', 'install', '--user', '...
====================================================================== 4 failed, 201 passed in 6.96s =======================================================================

Here is list of installed modules in build env

Package                       Version
----------------------------- -----------------
alabaster                     0.7.12
appdirs                       1.4.4
attrs                         22.1.0
Babel                         2.11.0
Brlapi                        0.8.3
build                         0.9.0
charset-normalizer            3.0.0
codespell                     2.2.1
contourpy                     1.0.5
cssselect                     1.1.0
cycler                        0.11.0
distro                        1.7.0
dnspython                     2.2.1
docutils                      0.18.1
extras                        1.0.0
fixtures                      4.0.0
fonttools                     4.38.0
gpg                           1.17.1-unknown
idna                          3.4
imagesize                     1.4.1
importlib-metadata            5.0.0
iniconfig                     1.1.1
Jinja2                        3.1.1
kiwisolver                    1.4.4
libcomps                      0.1.19
louis                         3.23.0
lxml                          4.9.1
MarkupSafe                    2.1.1
matplotlib                    3.6.0
numpy                         1.23.1
olefile                       0.46
packaging                     21.3
pbr                           5.9.0
pep517                        0.13.0
Pillow                        9.2.0
pip                           22.2.2
pluggy                        1.0.0
py                            1.11.0
Pygments                      2.13.0
PyGObject                     3.42.2
pyparsing                     3.0.9
pytest                        7.1.3
python-dateutil               2.8.2
pytz                          2022.4
requests                      2.28.1
requests_download             0.1.2
responses                     0.22.0
rpm                           4.17.0
scour                         0.38.2
six                           1.16.0
snowballstemmer               2.2.0
Sphinx                        5.3.0
sphinx-rtd-theme              1.1.0
sphinxcontrib-applehelp       1.0.2.dev20220730
sphinxcontrib-devhelp         1.0.2.dev20220730
sphinxcontrib-htmlhelp        2.0.0
sphinxcontrib-jsmath          1.0.1.dev20220730
sphinxcontrib-qthelp          1.0.3.dev20220730
sphinxcontrib-serializinghtml 1.1.5
testpath                      0.6.0
testtools                     2.5.0
toml                          0.10.2
tomli                         2.0.1
tomli_w                       1.0.0
types-toml                    0.10.8
urllib3                       1.26.12
wheel                         0.37.1
zipp                          3.9.0

[gotmax23]

The tests are all failing with

ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/home/tkloczko/.local/lib/python3.8/site-packages'
Check the permissions.

This seems to be a problem with your system and not a problem with the tests.

[kloczek]

Yep .. and question ios why test wiote is truing to install something?

[pradyunsg]

The test filename is:

tests/test_install.py

That should provide a hint for why its trying to install something. Flit has a flit install command, which is what is being tested in those tests.

[pradyunsg]

Closing this since this is an environment configuration issue.

[takluyver]

Actually, I think there is a genuine issue with the tests here that we hadn't spotted.

The tests do some monkeypatching to redirect installation to temporary directories:

flit/tests/test_install.py

Lines 25 to 30 in f5704ea

	self.get_dirs_patch = patch('flit.install.get_dirs',
	return_value={
	'scripts': os.path.join(td.name, 'scripts'),
	'purelib': os.path.join(td.name, 'site-packages'),
	'data': os.path.join(td.name, 'data'),
	})

But this doesn't affect the _auto_user method, which looks to see if it should do a user install or not:

flit/flit/install.py

Lines 152 to 154 in f5704ea

	if python == sys.executable:
	user_site = site.ENABLE_USER_SITE
	lib_dir = sysconfig.get_path('purelib')

And of course the monkeypatching also won't affect pip when the install code runs that as a subprocess.

So I'm going to reopen this 😉 . I'm not sure yet exactly what's the best way to properly isolate those tests - maybe we have to set up a venv for each test, even if that's slower - but I think they do need some work.