Add a "global_datadir" fixture too?

[pmav99]

When the testsuite has nested directories, shared_datadir tries to find a data directory inside each level. This is useful but in some cases you might also want to have a truly "global" datadir.

I think it would make sense if a global_datadir fixture was added that would always reference the data directory at the top level of the tests directory. If there is a single nest level, then shared_datadir and global_datadir should point to the same dir.

The implementation could be something like this (not thoroughly tested, e.g. I haven't tried it for namespace packages etc):

import importlib

@pytest.fixture
def global_datadir(tmpdir):
    package_name = __name__.split('.')[0]
    package_path = pathlib.Path(importlib.import_module(package_name).__file__).parent
    global_shared_path =  package_path/ "data"
    temp_path = pathlib.Path(str(tmpdir.join('data')))
    shutil.copytree(global_shared_path, temp_path)
    return temp_path

Should I make a PR?

[Madoshakalaka]

This is very nice. I've found myself wanting this in multiple projects

[cognifloyd]

You can simplify a bit by using the request context.

def global_datadir(request, tmpdir):
    package_path = pathlib.Path(request.module.__file__).parent
    ...

note that your package_path only looks up one directory above the current directory so if you either have a test that is already in the root dir or a test that is nested several levels down, this won't work.

I'm looking through the request object for something that could get the data dir in the root tests directory, but it looks like we might need to walk the dirs. hmm... surely pytest can tell us where the root conftest.py is?

    rootdir = request.config.rootdir  # path containing the tests dir...
    fspath = request.fspath  # the path to the file for the current test

[cognifloyd]

I hope there's a way to make this more generic so it can be included in this plugin, but here's what I just put together in conftest.py in the root tests directory of my project.

import os
import pathlib
import pytest
import shutil


@pytest.fixture(scope="session")
def original_global_datadir():
    return pathlib.Path(os.path.realpath(__file__)).parent / "data"


def prep_global_datadir(tmp_path_factory, original_global_datadir):
    temp_dir = tmp_path_factory.mktemp("data") / "datadir"
    shutil.copytree(original_global_datadir, temp_dir)
    return temp_dir


@pytest.fixture(scope="session")
def session_global_datadir(tmp_path_factory, original_global_datadir):
    return prep_global_datadir(tmp_path_factory, original_global_datadir)


@pytest.fixture(scope="module")
def module_global_datadir(tmp_path_factory, original_global_datadir):
    return prep_global_datadir(tmp_path_factory, original_global_datadir)


@pytest.fixture(scope="function")
def global_datadir(tmp_path_factory, original_global_datadir):
    return prep_global_datadir(tmp_path_factory, original_global_datadir)

[pmav99]

note that your package_path only looks up one directory above the current directory so if you either have a test that is already in the root dir or a test that is nested several levels down, this won't work.

I usually put such "global" fixtures in tests/conftest.py and as result the path does work as expected. Anyhow, this was just a proof of concept. Your latest suggestion is definitely more carefully thought/designed.

[BramVer]

Any plans on integrating this functionality?
We tend to use a single data folder inside our tests folder to house everything, and that's currently stopping us from using this plugin.

[igortg]

I just bump into this problem myself.

One addition would be to take the approach suggested by @nicoddemus here, since a shared data folder for an entire project could become very large.

Hence, it would be

def test_foo(global_datadir):
    file = global_datadir.get("file.txt")  # -- only here the file is copied to the tmp folder. 
                                           # --  would tmp folder be the same returned by "datadir" fixture?

About going forward with this: authors are very sensible on adding new features since this is a very mature and slim package (and I mostly agree with this approach). On the other hand, seems a very common request, and having a separate plugin to deal with such similar problem would be a bit annoying.

@nicoddemus @gabrielcnr - could you comment on that?

[marcelotrevisani]

maybe something similar of what @nicoddemus pointed but could be parametrized, in the lines of

def pytest_generate_tests(metafunc):
    for mark in metafunc.definition.iter_markers("data_file"):
        retrieve_data_file(metafunc, mark)

and in the retrieve_data_file we can get the parametrized options to retrieve the data file and use it something in the lines of

@pytest.mark.data_file("file1", "file.txt")
def test_with_mark(file1):
    assert file1.exists()

and it can return a Path object, similar of what parametrize does, and it is also possible to inform other parameters if necessary

[gabrielcnr]

@igortg - I think the idea of a global datadir might be indeed useful. Would you like to contribute with a PR ? That would be very welcome. Otherwise I'm happy to take a look at this at some point in the next few weeks after my holidays :-)