diff --git a/search/search_index.json b/search/search_index.json
index ff910f4..09eb98c 100644
--- a/search/search_index.json
+++ b/search/search_index.json
@@ -1 +1 @@
-{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"pytest-cases","text":"Separate test code from test cases in pytest.
Slides from the pytest-cases presentation at EuroPython 2021 are now available here.
New current_cases fixture to easily know the current case for each parameter ! See below for details.
Did you ever think that most of your test functions were actually the same test code, but with different data inputs and expected results/exceptions ?
-
pytest-cases leverages pytest and its great @pytest.mark.parametrize decorator, so that you can separate your test cases from your test functions.
-
In addition, pytest-cases provides several useful goodies to empower pytest. In particular it improves the fixture mechanism to support \"fixture unions\". This is a major change in the internal pytest engine, unlocking many possibilities such as using fixture references as parameter values in a test function. See here.
pytest-cases is fully compliant with pytest-harvest so you can easily monitor the execution times and created artifacts. With it, it becomes very easy to create a complete data science benchmark, for example comparing various models on various datasets as illustrated below (from the example section):
"},{"location":"#citing","title":"Citing","text":"If pytest-cases helps you with your research work, don't hesitate to spread the word ! For this simply use this Zenodo link to get the proper citation entry (at the bottom right of the page, many formats available including BibTeX).
"},{"location":"#installing","title":"Installing","text":"> pip install pytest-cases\n
Note: Installing pytest-cases has effects on the order of pytest tests execution, even if you do not use its features. One positive side effect is that it fixed pytest#5054. But if you see less desirable ordering please report it.
"},{"location":"#why-pytest-cases","title":"Why pytest-cases ?","text":"pytest philosophy
Let's consider the following foo function under test, located in example.py:
def foo(a, b):\n return a + 1, b + 1\n
If we were using plain pytest to test it with various inputs, we would create a test_foo.py file and use @pytest.mark.parametrize:
import pytest\nfrom example import foo\n\n@pytest.mark.parametrize(\"a,b\", [(1, 2), (-1, -2)])\ndef test_foo(a, b):\n # check that foo runs correctly and that the result is a tuple. \n assert isinstance(foo(a, b), tuple)\n
This is the fastest and most compact thing to do when you have a few number of test cases, that do not require code to generate each test case.
pytest current limitations
Now imagine that instead of (1, 2) and (-1, -2) each of our test cases
- requires a few lines of code to be generated. For example artificial data creation using
numpy and/or pandas:
import numpy as np\nimport pandas as pd\n\n# case 1: non-sorted uniformly sampled timeseries with 2 holes\ncase1 = pd.DataFrame({\"datetime\": pd.date_range(start='20/1/1', periods=20, \n freq='-1d', tz='UTC'),\n \"data1\": np.arange(0, 20),\n \"data2\": np.arange(1, 21),\n \"data3\": np.arange(1, 21)})\ncase1.drop([3, 12], inplace=True)\n
-
requires documentation to explain the other developers the intent of that precise test case
-
requires external resources (data files on the filesystem, databases...), with a variable number of cases depending on what is available on the resource - but of course not all the cases would come from the same resource, that would be too easy :).
-
requires a readable id, such as 'uniformly_sampled_nonsorted_with_holes' for the above example. Of course we could use pytest.param or ids=<list> but that is \"a pain to maintain\" according to pytest doc (I agree!). Such a design does not feel right as the id is detached from the case.
With standard pytest there is no particular pattern to simplify your life here. Investigating a little bit, people usually end up trying to mix parameters and fixtures and asking this kind of question: so1, so2. But by design it is not possible to solve this problem using fixtures, because pytest does not handle \"unions\" of fixtures.
So all in all, the final answer is \"you have to do this yourself\", and have pytest use your handcrafted list of parameters as the list of argvalues in @pytest.mark.parametrize. Typically we would end up creating a get_all_foo_test_cases function, independently from pytest:
@pytest.mark.parametrize(\"a,b\", get_all_foo_test_cases())\ndef test_foo(a, b):\n ...\n
There is also an example in pytest doc with a metafunc hook.
The issue with such workarounds is that you can do anything. And anything is a bit too much: this does not provide any convention / \"good practice\" on how to organize test cases, which is an open door to developing ad-hoc unreadable or unmaintainable solutions.
pytest_cases was created to provide an answer to this precise situation. It proposes a simple framework to separate test cases from test functions. The test cases are typically located in a separate \"companion\" file:
test_foo.py is your usual test file containing the test functions (named test_<id>),test_foo_cases.py contains the test cases, that are also functions (named case_<id> or even <prefix>_<id> if you prefer). Note: an alternate file naming style cases_foo.py is also available if you prefer it.
Test cases can also be provided explicitly, for example in a class container:
And many more as we'll see below.
"},{"location":"#basic-usage","title":"Basic usage","text":""},{"location":"#a-case-functions","title":"a- Case functions","text":"Let's create a test_foo_cases.py file. This file will contain test cases generator functions, that we will call case functions for brevity. In these functions, you will typically either parse some test data files, generate some simulated test data, expected results, etc.
def case_two_positive_ints():\n \"\"\" Inputs are two positive integers \"\"\"\n return 1, 2\n\ndef case_two_negative_ints():\n \"\"\" Inputs are two negative integers \"\"\"\n return -1, -2\n
Case functions do not have any particular requirement, apart from the default name convention case_<id> - but even that can be customized: you can use distinct prefixes to denote distinct kind of parameters, such as data_<id>, user_<id>, model_<id>...
Case functions can return anything that is considered useful to run the associated test. We will see below that you can use all classic pytest mechanism on case functions (id customization, skip/fail marks, parametrization, fixtures injection).
"},{"location":"#b-test-functions","title":"b- Test functions","text":"As usual we write our pytest test functions starting with test_, in a test_foo.py file. The only difference is that we now decorate it with @parametrize_with_cases instead of @pytest.mark.parametrize as we were doing previously:
from example import foo\nfrom pytest_cases import parametrize_with_cases\n\n@parametrize_with_cases(\"a,b\")\ndef test_foo(a, b):\n # check that foo runs correctly and that the result is a tuple. \n assert isinstance(foo(a, b), tuple)\n
As simple as that ! The syntax is basically the same than in pytest.mark.parametrize.
Executing pytest will now run our test function once for every case function:
>>> pytest -s -v\n============================= test session starts =============================\n(...)\n<your_project>/tests/test_foo.py::test_foo[two_positive_ints] PASSED [ 50%]\n<your_project>/tests/test_foo.py::test_foo[two_negative_ints] PASSED [ 100%]\n\n========================== 2 passed in 0.24 seconds ==========================\n
"},{"location":"#tools-for-daily-use","title":"Tools for daily use","text":""},{"location":"#a-cases-collection","title":"a- Cases collection","text":""},{"location":"#alternate-sources","title":"Alternate source(s)","text":"It is not mandatory that case functions should be in a different file than the test functions: both can be in the same file. For this you can use cases='.' or cases=THIS_MODULE to refer to the module in which the test function is located:
from pytest_cases import parametrize_with_cases\n\ndef case_one_positive_int():\n return 1\n\ndef case_one_negative_int():\n return -1\n\n@parametrize_with_cases(\"i\", cases='.')\ndef test_with_this_module(i):\n assert i == int(i)\n
However WARNING: only the case functions defined BEFORE the test function in the module file will be taken into account!
@parametrize_with_cases(cases=...) also accepts explicit list of case functions, classes containing case functions, and modules. See API Reference for details. A typical way to organize cases is to use classes for example:
from pytest_cases import parametrize_with_cases\n\nclass Foo:\n def case_a_positive_int(self):\n return 1\n\n def case_another_positive_int(self):\n return 2\n\n@parametrize_with_cases(\"a\", cases=Foo)\ndef test_foo(a):\n assert a > 0\n
Note that as for pytest, self is recreated for every test and therefore should not be used to store any useful information.
"},{"location":"#alternate-prefix","title":"Alternate prefix","text":"case_ might not be your preferred prefix, especially if you wish to store in the same module or class various kind of case data. @parametrize_with_cases offers a prefix=... argument to select an alternate prefix for your case functions. That way, you can store in the same module or class case functions as diverse as datasets (e.g. data_), user descriptions (e.g. user_), algorithms or machine learning models (e.g. model_ or algo_), etc.
from pytest_cases import parametrize_with_cases, parametrize\n\ndef data_a():\n return 'a'\n\n@parametrize(\"hello\", [True, False])\ndef data_b(hello):\n return \"hello\" if hello else \"world\"\n\ndef case_c():\n return dict(name=\"hi i'm not used\")\n\ndef user_bob():\n return \"bob\"\n\n@parametrize_with_cases(\"data\", cases='.', prefix=\"data_\")\n@parametrize_with_cases(\"user\", cases='.', prefix=\"user_\")\ndef test_with_data(data, user):\n assert data in ('a', \"hello\", \"world\")\n assert user == 'bob'\n
yields
test_doc_filters_n_tags.py::test_with_data[bob-a] PASSED [ 33%]\ntest_doc_filters_n_tags.py::test_with_data[bob-b-True] PASSED [ 66%]\ntest_doc_filters_n_tags.py::test_with_data[bob-b-False] PASSED [ 100%]\n
"},{"location":"#filters-and-tags","title":"Filters and tags","text":"The easiest way to select only a subset of case functions in a module or a class, is to specify a custom prefix instead of the default one ('case_'), as shown above.
However sometimes more advanced filtering is required. In that case, you can also rely on three additional mechanisms provided in @parametrize_with_cases:
- the
glob argument can contain a glob-like pattern for case ids. This can become handy to separate for example good or bad cases, the latter returning an expected error type and/or message for use with pytest.raises or with our alternative assert_exception.
from math import sqrt\nimport pytest\nfrom pytest_cases import parametrize_with_cases\n\n\ndef case_int_success():\n return 1\n\ndef case_negative_int_failure():\n # note that we decide to return the expected type of failure to check it\n return -1, ValueError, \"math domain error\"\n\n\n@parametrize_with_cases(\"data\", cases='.', glob=\"*success\")\ndef test_good_datasets(data):\n assert sqrt(data) > 0\n\n@parametrize_with_cases(\"data, err_type, err_msg\", cases='.', glob=\"*failure\")\ndef test_bad_datasets(data, err_type, err_msg):\n with pytest.raises(err_type, match=err_msg):\n sqrt(data)\n
- the
has_tag argument allows you to filter cases based on tags set on case functions using the @case decorator. See API reference of @case and @parametrize_with_cases.
from pytest_cases import parametrize_with_cases, case\n\nclass FooCases:\n def case_two_positive_ints(self):\n return 1, 2\n\n @case(tags='foo')\n def case_one_positive_int(self):\n return 1\n\n@parametrize_with_cases(\"a\", cases=FooCases, has_tag='foo')\ndef test_foo(a):\n assert a > 0\n
- Finally if none of the above matches your expectations, you can provide a callable to
filter. This callable will receive each collected case function and should return True (or a truth-value convertible object) in case of success. Note that your function can leverage the get_case_id, get_case_marks, get_case_tags etc. helper functions to read the tags, marks and id found on it. See API doc.
from pytest_cases import get_case_id\n\n@parametrize_with_cases(\"data\", cases='.', \n filter=lambda cf: \"success\" in get_case_id(cf))\ndef test_good_datasets2(data):\n ...\n
- An array of default filters is available in the
filters module: has_tag, id_has_prefix, etc. You can use logical operations on them such as \"and\" (&) \"or\" (|) and \"not\" (~) in order to create your own custom filters. See API reference for details.
from pytest_cases import filters as ft\n\n\n@parametrize_with_cases(\"data\", cases='.',\n filter=ft.has_tag(\"success\") & ft.id_has_prefix(\"case_b\")\ndef test_good_datasets3(data):\n ...\n
"},{"location":"#b-case-functions","title":"b- Case functions","text":""},{"location":"#custom-case-name","title":"Custom case name","text":"The id used by pytest for a given case is automatically taken from the case function name by removing the case_ (or other custom) prefix. It can instead be customized explicitly by decorating your case function with the @case(id=<id>) decorator. See API reference.
from pytest_cases import case\n\n@case(id=\"2 positive integers\")\ndef case_two_positive_ints():\n return 1, 2\n
"},{"location":"#pytest-marks-skip-xfail","title":"Pytest marks (skip, xfail...)","text":"pytest marks such as @pytest.mark.skipif can be applied on case functions the same way as with test functions.
import sys\nimport pytest\n\n@pytest.mark.skipif(sys.version_info < (3, 0), reason=\"Not useful on python 2\")\ndef case_two_positive_ints():\n return 1, 2\n
"},{"location":"#case-generators","title":"Case generators","text":"In many real-world usage we want to generate one test case per <something>. The most intuitive way would be to use a for loop to create the case functions, and to use the @case decorator to set their names ; however this would not be very readable.
Instead, case functions can be parametrized the same way as with test functions: simply add the parameter names as arguments in their signature and decorate with @pytest.mark.parametrize. Even better, you can use the enhanced @parametrize from pytest-cases so as to benefit from its additional usability features (see API reference):
from pytest_cases import parametrize, parametrize_with_cases\n\nclass CasesFoo:\n def case_hello(self):\n return \"hello world\"\n\n @parametrize(who=('you', 'there'))\n def case_simple_generator(self, who):\n return \"hello %s\" % who\n\n\n@parametrize_with_cases(\"msg\", cases=CasesFoo)\ndef test_foo(msg):\n assert isinstance(msg, str) and msg.startswith(\"hello\")\n
Yields
test_generators.py::test_foo[hello] PASSED [ 33%]\ntest_generators.py::test_foo[simple_generator-who=you] PASSED [ 66%]\ntest_generators.py::test_foo[simple_generator-who=there] PASSED [100%]\n
"},{"location":"#cases-requiring-fixtures","title":"Cases requiring fixtures","text":"Cases can use fixtures the same way as test functions do: simply add the fixture names as arguments in their signature and make sure the fixture exists or is imported either in the module where @parametrize_with_cases is used, or in a conftest.py file in one of the parent packages.
See pytest documentation on sharing fixturesand this blog.
You can use the experimental @parametrize_with_cases(import_fixtures=True) argument to perform the import automatically for you, see API reference.
Use @fixture instead of @pytest.fixture
If a fixture is used by some of your cases only, then you should use the @fixture decorator from pytest-cases instead of the standard @pytest.fixture. Otherwise you fixture will be setup/teardown for all cases even those not requiring it. See @fixture doc.
from pytest_cases import parametrize_with_cases, fixture, parametrize\n\n@fixture(scope='session')\ndef db():\n return {0: 'louise', 1: 'bob'}\n\ndef user_bob(db):\n return db[1]\n\n@parametrize(id=range(2))\ndef user_from_db(db, id):\n return db[id]\n\n@parametrize_with_cases(\"a\", cases='.', prefix='user_')\ndef test_users(a, db, request):\n print(\"this is test %r\" % request.node.nodeid)\n assert a in db.values()\n
yields
test_fixtures.py::test_users[a_is_bob] \ntest_fixtures.py::test_users[a_is_from_db-id=0] \ntest_fixtures.py::test_users[a_is_from_db-id=1] \n
"},{"location":"#advanced-topics","title":"Advanced topics","text":""},{"location":"#a-scope-of-cases","title":"a- Scope of cases","text":"By default a case function is transformed into a lazy parameter using lazy_value. This is not a fixture, but simply a new parametrize mechanism that allows parameters to be provided by functions (See @parametrize).
However, as soon as a case function is either parametrized or requires a fixture, then it is automatically transformed into a fixture so that pytest can handle it properly. In that situation, the fixture needs to have a scope. By default this scope is \"function\". You can change it using the scope argument in @parametrize_with_cases.
"},{"location":"#b-parametrizing-fixtures","title":"b- Parametrizing fixtures","text":"In some scenarii you might wish to parametrize a fixture with the cases, rather than the test function. For example
-
to inject the same test cases in several test functions without duplicating the @parametrize_with_cases decorator on each of them,
-
to generate the test cases once for the whole session, using a scope='session' fixture or another scope,
-
to modify the test cases, log some message, or perform some other action before injecting them into the test functions, and/or after executing the test function (thanks to yield fixtures)
-
...
For this, simply use @fixture from pytest_cases instead of @pytest.fixture to define your fixture. That allows your fixtures to be easily parametrized with @parametrize_with_cases, @parametrize, and even @pytest.mark.parametrize.
from pytest_cases import fixture, parametrize_with_cases\n\n@fixture\n@parametrize_with_cases(\"a,b\")\ndef c(a, b):\n return a + b\n\ndef test_foo(c):\n assert isinstance(c, int)\n
"},{"location":"#c-caching-cases","title":"c- Caching cases","text":"After starting to reuse cases in several test functions, you might end-up thinking \"why do I have to spend the data parsing/generation time several times ? It is the same case.\".
pytest-cases follows the same philosophy than pytest: each test node should be independent. Therefore case functions are called for each test case. This ensures that mutable objects can not leak across tests, for example.
That being said, if you are certain that your tests do not modify your cases data, there are several ways to solve this issue:
- the easiest way is to use fixtures with a broad scope, as explained above. However in some parametrization scenarii,
pytest does not guarantee that the fixture will be setup only once for the whole session, even if it is a session-scoped fixture. Also the cases will be parsed every time you run pytest, which might be cumbersome
from pytest_cases import parametrize, parametrize_with_cases, fixture\n\n\n@parametrize(a=range(2))\ndef case_dummy(a):\n # this is read only once per a, while there are 4 test runs \n return a\n\n@fixture(scope='session')\n@parametrize_with_cases(\"a\", cases='.')\ndef cached_a(a):\n return a\n\n\n@parametrize(d=range(2))\ndef test_caching(cached_a, d):\n assert d < 2\n assert 0 <= cached_a <= 1\n
-
an alternative is to use functools.lru_cache to explicitly set a memory cache on a case function. For simple cases you could simply decorate your case function with @lru_cache(maxsize=1) since simple case functions do not have arguments. However for case generators this is a bit more tricky to size the cache - the easiest thing is probably to let it to its default size of 128 with the no-argument version @lru_cache, or to remove the max limit and let it auto-grow, with @lru_cache(max_size=None). See lru_cache documentation for details. Note that an older version of pytest-cases was offering some facilities to set the cache size, this has been removed from the library in version 2.0.0 as it seemed to provide little added value.
-
finally, you might wish to persist some cases on disk in order for example to avoid downloading them again from their original source, and/or to avoid costly processing on every pytest session. For this, the perfect match for you is to use joblib's excellent Memory cache.
If you add a cache mechanism, make sure that your test functions do not modify the returned objects !
"},{"location":"#d-accessing-the-current-case","title":"d- Accessing the current case","text":"In some scenarii you may wish to access the case functions that are currently used to provide the parameter values. This may be
- to make your test behave differently depending on the case function, case id or case tags
- to
pytest.skip some combinations of parameters/cases that do not make sense - ...
With pytest-cases starting in version 3.5, this is now possible thanks to the current_cases fixture. Simply use this fixture to get a dictionary containing the actual parameter id and case function for all parameters parametrized with cases in the current test node. Parametrized fixtures, if any, will appear in a sub-dictionary indexed by the fixture name.
from pytest_cases import parametrize, parametrize_with_cases, fixture\n\n@parametrize(nb=(1,))\ndef case_a():\n return nb\n\n@fixture\n@parametrize_with_cases(\"foo\", cases=case_a)\ndef my_fixture(foo):\n return foo\n\n@parametrize_with_cases(\"data\", cases=case_a)\ndef test_get_current_case(data, my_fixture, current_cases):\n\n # access the case details for a test parameter\n data_id, data_fun, data_params = current_cases[\"data\"]\n\n # access the case details for a fixture parameter\n my_fixture_id, my_fixture_fun, my_fixture_params = current_cases[\"my_fixture\"][\"foo\"]\n\n # let's print all case information for this test node\n print(current_cases)\n
yields
{'data': Case(id='a', func=<function case_a at 0x000001C0CAE9E700>, params={'nb': 1}), \n 'my_fixture': {\n 'foo': Case(id='a', func=<function case_a at 0x000001C0CAE9E700>, params={'nb': 1})\n }}\n
As you can see above, details are provided as namedtuples. When a case itself is parametrized, its current parameter value(s) appear too (in the above example, case_a is parametrized with nb). This can be used to skip a test conditionally, for example:
if data_fun is case_a and data_params['nb'] == 1:\n pytest.skip(\"This specific case is skipped\")\n
It can also be used to insert a debug breakpoint for a specific case.
To get more information on the case function, you can use get_case_marks(func), get_case_tags(func). You can also use matches_tag_query(...) to check if a case function matches some expectations either concerning its id or its tags. See API reference.
Note: you can get the same information from a pytest hook, using the get_current_cases function. See API reference for details.
"},{"location":"#e-test-ids","title":"e- Test ids","text":"Starting from version 3.0.0, test ids induced by @parametrize_with_cases are similar to the ids induced by @pytest.mark.parametrize, even if a case function is itself parametrized or requires a fixture. In some situations you may wish to get a better control on the test ids.
For this you can pass a callable to @parametrize_with_cases(ids=<callable>). In this callable, you may wish to use get_case_id, get_case_marks, get_case_tags, or matches_tag_query to return a custom id.
from pytest_cases import parametrize, parametrize_with_cases, case, get_case_id\n\ndef case_hello():\n return \"hello !\"\n\n@case(id=\"hello_world\")\ndef case_basic2():\n return \"hello, world !\"\n\n@case(id=\"hello_name\")\n@parametrize(\"name\", [\"you\", \"earthling\"])\ndef case_basic3(name):\n return \"hello, %s !\" % name\n\ndef myidgen(case_fun):\n \"\"\"Custom test case id\"\"\"\n return \"#%s#\" % get_case_id(case_fun)\n\n@parametrize_with_cases(\"msg\", cases=\".\", ids=myidgen)\ndef test_foo(msg):\n print(msg)\n
pytest -s -v yields:
============================= test session starts =============================\nplatform win32 -- Python 3.7.3, pytest-5.3.5, py-1.9.0, pluggy-0.13.1\ncachedir: .pytest_cache\n(...)\n\ntest_doc_ids_debug.py::test_foo[#hello#] \ntest_doc_ids_debug.py::test_foo[#hello_world#] \ntest_doc_ids_debug.py::test_foo[#hello_name#-you] \ntest_doc_ids_debug.py::test_foo[#hello_name#-earthling] \n\n============================== 4 passed in 0.07s ==============================\n
"},{"location":"#f-debugging","title":"f- Debugging","text":"When all of your case functions are simple, @parametrize_with_cases generates a @parametrize decorator with argvalues being a list of lazy_value(<case_func>) for all of them. This in turn falls back to a good old @pytest.mark.parametrize, so the behaviour is close to what you are used to see when using pytest.
However when at least one case function is complex, typically when it requires a fixture, then @parametrize_with_cases wraps it into a fixture and passes a fixture_ref(<fixture_wrapping_case>) to @parametrize. This creates a so-called \"fixture union\": tests are not any more a cross-product of parameters, but a tree. This new feature brought by pytest-cases is not present in pytest by default, and adds a layer of complexity. But good news: starting in pytest-cases 3.0.0, this complexity is entirely hidden. In other words, generated test ids do not differ between this mode, and the simple mode. For debugging purposes however, you might wish to make this visible by setting idstyle:
from pytest_cases import parametrize, parametrize_with_cases, case, fixture\n\ndef case_hello():\n return \"hello !\"\n\n@fixture\n@parametrize(\"_name\", [\"you\", \"earthling\"])\ndef name(_name):\n return _name\n\n@case(id=\"hello_fixture\")\ndef case_basic3(name):\n return \"hello, %s !\" % name\n\n@parametrize_with_cases(\"msg\", cases=\".\", idstyle=\"nostyle\")\ndef test_default_idstyle(msg):\n print(msg)\n\n@parametrize_with_cases(\"msg\", cases=\".\", idstyle=\"compact\")\ndef test_compact_idstyle(msg):\n print(msg)\n\n@parametrize_with_cases(\"msg\", cases=\".\", idstyle=\"explicit\")\ndef test_explicit_idstyle(msg):\n print(msg)\n
pytest -s -v yields
test_doc_debug.py::test_default_idstyle[hello] \ntest_doc_debug.py::test_default_idstyle[hello_fixture-you] \ntest_doc_debug.py::test_default_idstyle[hello_fixture-earthling] \ntest_doc_debug.py::test_compact_idstyle[\\hello] \ntest_doc_debug.py::test_compact_idstyle[\\hello_fixture-you] \ntest_doc_debug.py::test_compact_idstyle[\\hello_fixture-earthling] \ntest_doc_debug.py::test_explicit_idstyle[msg\\hello] \ntest_doc_debug.py::test_explicit_idstyle[msg\\hello_fixture-you] \ntest_doc_debug.py::test_explicit_idstyle[msg\\hello_fixture-earthling] \n
See also @parametrize documentation for details.
"},{"location":"#main-features-benefits","title":"Main features / benefits","text":" -
Separation of concerns: test code on one hand, test cases data on the other hand. This is particularly relevant for data science projects where a lot of test datasets are used on the same block of test code.
-
Everything in the test case or in the fixture, not outside. A side-effect of @pytest.mark.parametrize is that users tend to create or parse their datasets outside of the test function. pytest_cases suggests a model where the potentially time and memory consuming step of case data generation/retrieval is performed inside the test node or the required fixture, thus keeping every test case run more independent. It is also easy to put debug breakpoints on specific test cases.
-
User experience fully aligned with pytest. Cases collection and filtering, cases parametrization, cases output unpacking as test arguments, cases using fixtures... all of this will look very familiar to pytest users.
"},{"location":"#see-also","title":"See Also","text":""},{"location":"#pytest","title":"pytest","text":" - pytest documentation on parametrize
- pytest documentation on fixtures
- pytest-steps
- pytest-harvest
- pytest-patterns for examples showing how to combine the various plugins to create data science benchmarks.
"},{"location":"#others","title":"Others","text":" - makefun is used to dynamically generate functions or modify user-provided function signatures.
- decopatch is used to create decorators.
Do you like this library ? You might also like my other python libraries
"},{"location":"#want-to-contribute","title":"Want to contribute ?","text":"Details on the github page: https://github.com/smarie/python-pytest-cases
"},{"location":"api_reference/","title":"API reference","text":"In general, using help(symbol) is the recommended way to get the latest documentation. In addition, this page provides an overview of the various elements in this package.
"},{"location":"api_reference/#1-fixtures","title":"1 - Fixtures","text":""},{"location":"api_reference/#current_cases","title":"current_cases","text":"A fixture containing get_current_cases(request).
This is a dictionary containing all case parameters for the currently active pytest item. For each test function argument parametrized using a @parametrize_with_case(<argname>, ...) this dictionary contains an entry {<argname>: (case_id, case_function, case_params)}. If several argnames are parametrized this way, a dedicated entry will be present for each argname. The tuple is a namedtuple containing
- `id` a string containing the actual case id constructed by `@parametrize_with_cases`.\n - `function` the original case function.\n - `params` a dictionary, containing the parameters of the case, if itself is parametrized. Note that if the\ncase is parametrized with `@parametrize_with_cases`, the associated parameter value in the dictionary will also be\n`(case_id, case_function, case_params)`.\n
If a fixture parametrized with cases is active, the dictionary will contain an entry {<fixturename>: <dct>} where <dct> is a dictionary {<argname>: (case_id, case_function, case_params)}.
To get more information on a case function, you can use get_case_marks(f), get_case_tags(f). You can also use matches_tag_query to check if a case function matches some expectations either concerning its id or its tags. See filters and tags documentation.
"},{"location":"api_reference/#2-case-functions","title":"2 - Case functions","text":"As explained in the documentation, case functions have no requirement anymore, and starting from version 2.0.0 of pytest_cases they can be parametrized with the usual @pytest.mark.parametrize or its improvement @parametrize. Therefore the only remaining decorator is the optional @case decorator:
"},{"location":"api_reference/#case","title":"@case","text":"@case(id=None, # type: str # noqa\n tags=None, # type: Union[Any, Iterable[Any]]\n marks=(), # type: Union[MarkDecorator, Iterable[MarkDecorator]]\n )\n
Optional decorator for case functions so as to customize some information.
@case(id='hey')\ndef case_hi():\n return 1\n
Parameters:
-
id: the custom pytest id that should be used when this case is active. Replaces the deprecated @case_name decorator from v1. If no id is provided, the id is generated from case functions by removing their prefix, see @parametrize_with_cases(prefix='case_').
-
tags: custom tags to be used for filtering in @parametrize_with_cases(has_tags). Replaces the deprecated @case_tags and @target decorators.
-
marks: optional pytest marks to add on the case. Note that decorating the function directly with the mark also works, and if marks are provided in both places they are merged.
"},{"location":"api_reference/#copy_case_info","title":"copy_case_info","text":"def copy_case_info(from_fun, # type: Callable\n to_fun # type: Callable\n ):\n
Copies all information from case function from_fun to to_fun.
"},{"location":"api_reference/#set_case_id","title":"set_case_id","text":"def set_case_id(id, # type: str\n case_func # type: Callable\n ):\n
Sets an explicit id on case function case_func.
"},{"location":"api_reference/#get_case_id","title":"get_case_id","text":"def get_case_id(case_func, # type: Callable\n prefix_for_default_ids='case_' # type: str\n ):\n
Return the case id associated with this case function.
If a custom id is not present, a case id is automatically created from the function name based on removing the provided prefix if present at the beginning of the function name. If the resulting case id is empty, \"\" will be returned.
Parameters:
"},{"location":"api_reference/#get_case_marks","title":"get_case_marks","text":"def get_case_marks(case_func, # type: Callable\n concatenate_with_fun_marks=False, # type: bool\n as_decorators=False # type: bool\n ):\n
Return the marks that are on the case function.
There are currently two ways to place a mark on a case function: either with @pytest.mark.<name> or in @case(marks=...). This function returns a list of marks containing either both (if concatenate_with_fun_marks is True) or only the ones set with @case (concatenate_with_fun_marks is False, default).
Parameters:
-
case_func: the case function
-
concatenate_with_fun_marks: if False (default) only the marks declared in @case will be returned. Otherwise a concatenation of marks in @case and on the function (for example directly with @pytest.mark.<mk>) will be returned.
-
as_decorators: when True, the marks (MarkInfo) will be transformed into MarkDecorators before being returned. Otherwise (default) the marks are returned as is.
"},{"location":"api_reference/#get_case_tags","title":"get_case_tags","text":"def get_case_tags(case_func # type: Callable\n ):\n
Return the tags on this case function or an empty tuple.
Parameters:
case_func: the case function
"},{"location":"api_reference/#matches_tag_query","title":"matches_tag_query","text":"def matches_tag_query(case_fun, # type: Callable\n has_tag=None, # type: Union[str, Iterable[str]]\n filter=None, # type: Union[Callable[[Callable], bool], Iterable[Callable[[Callable], bool]]] # noqa\n ):\n
This function is the one used by @parametrize_with_cases to filter the case functions collected. It can be used manually for tests/debug.
Returns True if the case function is selected by the query:
-
if has_tag contains one or several tags, they should ALL be present in the tags set on case_fun (get_case_tags)
-
if filter contains one or several filter callables, they are all called in sequence and the case_fun is only selected if ALL of them return a True truth value
Parameters:
-
case_fun: the case function
-
has_tag: one or several tags that should ALL be present in the tags set on case_fun for it to be selected.
-
filter: one or several filter callables that will be called in sequence. If all of them return a True truth value, case_fun is selected.
"},{"location":"api_reference/#is_case_class","title":"is_case_class","text":"def is_case_class(cls, # type: Any\n case_marker_in_name='Case', # type: str\n check_name=True # type: bool\n ):\n
This function is the one used by @parametrize_with_cases to collect cases within classes. It can be used manually for tests/debug.
Returns True if the given object is a class and, if check_name=True (default), if its name contains case_marker_in_name.
Parameters:
-
cls: the object to check
-
case_marker_in_name: the string that should be present in a class name so that it is selected. Default is 'Case'.
-
check_name: a boolean (default True) to enforce that the name contains the word case_marker_in_name. If False, any class will lead to a True result whatever its name.
"},{"location":"api_reference/#is_case_function","title":"is_case_function","text":"def is_case_function(f, # type: Any\n prefix='case_', # type: str\n check_prefix=True # type: bool\n ):\n
This function is the one used by @parametrize_with_cases to collect cases. It can be used manually for tests/debug.
Returns True if the provided object is a function or callable and, if check_prefix=True (default), if it starts with prefix.
Parameters:
-
f: the object to check
-
prefix: the string that should be present at the beginning of a function name so that it is selected. Default is 'case_'.
-
check_prefix: if this boolean is True (default), the prefix will be checked. If False, any function will lead to a True result whatever its name.
"},{"location":"api_reference/#the-filters-submodule","title":"The filters submodule","text":"This submodule contains symbols to help you create filters for @parametrize_with_cases(filter=...).
All helper filters in this submodule return an instance of CaseFilter, so that you can combine them easily with \"and\" (&) \"or\" (|) and \"invert\" (~) in order to create new custom filters.
"},{"location":"api_reference/#has_tag","title":"has_tag","text":"def has_tag(tag_name: str)\n
Selects cases that have the tag tag_name. See @case(tags=...) to add tags to a case.
"},{"location":"api_reference/#has_tags","title":"has_tags","text":"def has_tags(*tag_names: str)\n
Selects cases that have all tags tag_names. See @case(tags=...) to add tags to a case.
"},{"location":"api_reference/#id_has_prefix","title":"id_has_prefix","text":"def id_has_prefix(prefix: str)\n
Selects cases that have a case id prefix prefix. Note that this is not the prefix of the whole case function name, but the case id, possibly overridden with @case(id=)
"},{"location":"api_reference/#id_has_suffix","title":"id_has_suffix","text":"def id_has_suffix(suffix: str)\n
Selects cases that have a case id suffix suffix. Note that this is not the suffix of the whole case function name, but the case id, possibly overridden with @case(id=)
"},{"location":"api_reference/#id_match_regex","title":"id_match_regex","text":"def id_match_regex(regex: str)\n
Selects cases that have a case id matching regex pattern regex. Note that this is not a match of the whole case function name, but the case id, possibly overridden with @case(id=)
"},{"location":"api_reference/#casefilter","title":"CaseFilter","text":"CaseFilter(filter_function: Callable)\n
CaseFilter is the class used by all filters above, and implementing logical operations \"and\" (&) \"or\" (|) and \"not\" (~). You can use it to define a composable filter from any callable receiving a single case argument and returning a boolean indicating if the case is selected.
"},{"location":"api_reference/#3-cases-collection","title":"3 - Cases collection","text":""},{"location":"api_reference/#parametrize_with_cases","title":"@parametrize_with_cases","text":"CaseType = Union[Callable, Type, ModuleRef]\n\n@parametrize_with_cases(argnames: str,\n cases: Union[CaseType, List[CaseType]] = AUTO,\n prefix: str = 'case_',\n glob: str = None,\n has_tag: Union[str, Iterable[str]] = None,\n filter: Callable = None,\n ids: Union[Callable, Iterable[str]] = None,\n idstyle: Union[str, Callable] = None,\n scope: str = \"function\",\n import_fixtures: bool = False\n )\n
A decorator for test functions or fixtures, to parametrize them based on test cases. It works similarly to @pytest.mark.parametrize: argnames represent a coma-separated string of arguments to inject in the decorated test function or fixture. The argument values (argvalues in @pytest.mark.parametrize) are collected from the various case functions found according to cases, and injected as lazy values so that the case functions are called just before the test or fixture is executed.
By default (cases=AUTO) the list of test cases is automatically drawn from the python module file named test_<name>_cases.py or if not found, cases_<name>.py, where test_<name> is the current module name.
Finally, the cases argument also accepts an explicit case function, cases-containing class, module or module name; or a list containing any mix of these elements. Note that both absolute and relative module names are supported.
Note that @parametrize_with_cases collection and parameter creation steps are strictly equivalent to get_all_cases + get_parametrize_args. This can be handy for debugging purposes.
# Collect all cases\ncases_funs = get_all_cases(f, cases=cases, prefix=prefix,\n glob=glob, has_tag=has_tag, filter=filter)\n\n# Transform the various functions found\nargvalues = get_parametrize_args(host_class_or_module_of_f, cases_funs)\n
Parameters
-
argnames: same than in @pytest.mark.parametrize
-
cases: a case function, a class containing cases, a module object or a module name string (relative module names accepted). Or a list of such items. You may use THIS_MODULE or '.' to include current module. AUTO (default) means that the module named test_<name>_cases.py or if not found, cases_<name>.py, will be loaded, where test_<name>.py is the module file of the decorated function. When a module is listed, all of its functions matching the prefix, filter and has_tag are selected, including those functions nested in classes following naming pattern *Case*. Nested subclasses are taken into account, as long as they follow the *Case* naming pattern. When classes are explicitly provided in the list, they can have any name and do not need to follow this *Case* pattern.
-
prefix: the prefix for case functions. Default is 'case_' but you might wish to use different prefixes to denote different kind of cases, for example 'data_', 'algo_', 'user_', etc.
-
glob: a matching pattern for case ids, for example *_success or *_failure. The only special character that can be used for now in this pattern is *, it can not be escaped, and it can be used several times in the same expression. The pattern should match the entire case id for the case to be selected. Note that this is applied on the case id, and therefore if it is customized through @case(id=...) it will be taken into account.
-
has_tag: a single tag or a tuple, set, list of tags that should be matched by the ones set with the @case decorator on the case function(s) to be selected.
-
filter: a callable receiving the case function and returning True or a truth value in case the function needs to be selected.
-
ids: optional custom ids, similar to the one in pytest.mark.parametrize. Users may either provide an iterable of string ids, or a callable. If a callable is provided it will receive the case functions. Users may wish to use get_case_id or other helpers in the API to inspect the case functions.
-
idstyle: This is mostly for debug. Style of ids to be used in the \"union\" fixtures generated by @parametrize if some cases are transformed into fixtures behind the scenes. idstyle possible values are 'compact', 'explicit' or None/'nostyle' (default), or a callable. idstyle has no effect if no cases are transformed into fixtures. As opposed to ids, a callable provided here will receive a ParamAlternative object indicating which generated fixture should be used. See @parametrize for details.
-
scope: The scope of the union fixture to create if fixture_refs are found in the argvalues
-
import_fixtures: experimental feature. Turn this to True in order to automatically import all fixtures defined in the cases module into the current module.
"},{"location":"api_reference/#get_current_cases","title":"get_current_cases","text":"def get_current_cases(request_or_item):\n
Returns a dictionary containing all case parameters for the currently active pytest item. You can either pass the pytest item (available in some hooks) or the request (available in hooks, and also directly as a fixture).
For each test function argument parametrized using a @parametrize_with_case(<argname>, ...) this dictionary contains an entry {<argname>: (case_id, case_function, case_params)}. If several argnames are parametrized this way, a dedicated entry will be present for each argname. The tuple is a namedtuple containing
- `id` a string containing the actual case id constructed by `@parametrize_with_cases`.\n - `function` the original case function.\n - `params` a dictionary, containing the parameters of the case, if itself is parametrized. Note that if the\ncase is parametrized with `@parametrize_with_cases`, the associated parameter value in the dictionary will also be\n`(case_id, case_function, case_params)`.\n
If a fixture parametrized with cases is active, the dictionary will contain an entry {<fixturename>: <dct>} where <dct> is a dictionary {<argname>: (case_id, case_function, case_params)}.
To get more information on a case function, you can use get_case_id(f), get_case_marks(f), get_case_tags(f). You can also use matches_tag_query to check if a case function matches some expectations either concerning its id or its tags. See filters and tags documentation.
Note that you can get the same contents directly by using the current_cases fixture.
"},{"location":"api_reference/#get_all_cases","title":"get_all_cases","text":"CaseType = Union[Callable, Type, ModuleRef]\n\ndef get_all_cases(parametrization_target: Callable,\n cases: Union[CaseType, List[CaseType]] = None,\n prefix: str = 'case_',\n glob: str = None,\n has_tag: Union[str, Iterable[str]] = None,\n filter: Callable[[Callable], bool] = None\n ) -> List[Callable]:\n
Collect all cases as used with @parametrize_with_cases. See @parametrize_with_cases for more details on the parameters. This can be used to lists all desired cases for a given parametrization_target (a test function or a fixture) which may be convenient for debugging purposes. # Get the cases for f that are defined in the current file\ncases = get_all_cases(f, cases=\".\")\n\n# Get the cases from cases_xyz.py or test_xyz_cases.py\nimport test.test_xyz\nxyz_cases = get_all_cases(test.test_xyz)\n\n# Can be used to filter explicit cases, in which case no parametrization_target is needed\nfiltered_cases = get_all_cases(cases=[case_1, case_2, case_3], has_tag=[\"banana\"])\n
- If using a
cases argument that requires module information, such as \".\" AUTO or a relative module like \".xyz\", the value of parametrization_target will be used to to determine the context. If None or simply left empty, it will use the module from which get_all_cases was called. You can pass an explicit module object or a function, in which case the module in which it's defined will be used.
"},{"location":"api_reference/#get_parametrize_args","title":"get_parametrize_args","text":"def get_parametrize_args(host_class_or_module: Union[Type, ModuleType],\n cases_funs: List[Callable],\n debug: bool = False\n ) -> List[Union[lazy_value, fixture_ref]]:\n
Transforms a list of cases (obtained from get_all_cases) into a list of argvalues for @parametrize. Each case function case_fun is transformed into one or several lazy_value(s) or a fixture_ref:
-
If case_fun requires at least on fixture, a fixture will be created if not yet present, and a fixture_ref will be returned.
-
If case_fun is a parametrized case, one lazy_value with a partialized version will be created for each parameter combination.
-
Otherwise, case_fun represents a single case: in that case a single lazy_value is returned.
"},{"location":"api_reference/#4-pytest-goodies","title":"4 - Pytest goodies","text":""},{"location":"api_reference/#fixture","title":"@fixture","text":"@fixture(scope: str = \"function\", \n autouse: bool = False, \n name: str = None, \n unpack_into: Iterable[str] = None,\n hook: Callable = None,\n **kwargs)\n
Identical to @pytest.fixture decorator, except that
-
when used in a fixture union (either explicit fixture_union or indirect through @parametrize+fixture_ref or @parametrize_with_cases), it will not be setup/teardown unnecessarily in tests that do not require it.
-
it supports multi-parametrization with @pytest.mark.parametrize as requested in pytest#3960. As a consequence it does not support the params and ids arguments anymore.
-
it supports a new argument unpack_into where you can provide names for fixtures where to unpack this fixture into.
As a consequence it does not support the params and ids arguments anymore.
Parameters:
- scope: the scope for which this fixture is shared, one of \"function\" (default), \"class\", \"module\" or \"session\".
- autouse: if True, the fixture func is activated for all tests that can see it. If False (the default) then an explicitreference is needed to activate the fixture.
- name: the name of the fixture. This defaults to the name of the decorated function. Note: If a fixture is used in the same module in which it is defined, the function name of the fixture will be shadowed by the function arg that requests the fixture; one wayto resolve this is to name the decorated function
fixture_<fixturename> and then use @pytest.fixture(name='<fixturename>'). - unpack_into: an optional iterable of names, or string containing coma-separated names, for additional fixtures to create to represent parts of this fixture. See
unpack_fixture for details. - hook: an optional hook to apply to each fixture function that is created during this call. The hook function will be called every time a fixture is about to be created. It will receive a single argument (the function implementing the fixture) and should return the function to use. For example you can use
saved_fixture from pytest-harvest as a hook in order to save all such created fixtures in the fixture store. - kwargs: other keyword arguments for
@pytest.fixture
"},{"location":"api_reference/#unpack_fixture","title":"unpack_fixture","text":"def unpack_fixture(argnames: str,\n fixture: Union[str, Callable],\n in_cls: bool = False,\n hook: Callable = None\n ) -> Tuple[<Fixture>, ...]\n
Creates several fixtures with names argnames from the source fixture. Created fixtures will correspond to elements unpacked from fixture in order. For example if fixture is a tuple of length 2, argnames=\"a,b\" will create two fixtures containing the first and second element respectively.
The created fixtures are automatically registered into the callers' module, but you may wish to assign them to variables for convenience. In that case make sure that you use the same names, e.g. a, b = unpack_fixture('a,b', 'c').
import pytest\nfrom pytest_cases import unpack_fixture, fixture\n\n@fixture\n@pytest.mark.parametrize(\"o\", ['hello', 'world'])\ndef c(o):\n return o, o[0]\n\na, b = unpack_fixture(\"a,b\", c)\n\ndef test_function(a, b):\n assert a[0] == b\n
You can also use this function inside a class with in_cls=True. In that case you MUST assign the output of the function to variables, as the created fixtures won't be registered with the encompassing module.
import pytest\nfrom pytest_cases import unpack_fixture, fixture\n\n@fixture\n@pytest.mark.parametrize(\"o\", ['hello', 'world'])\ndef c(o):\n return o, o[0]\n\nclass TestClass:\n a, b = unpack_fixture(\"a,b\", c, in_cls=True)\n\n def test_function(self, a, b):\n assert a[0] == b\n
Parameters
- argnames: same as
@pytest.mark.parametrize argnames. - fixture: a fixture name string or a fixture symbol. If a fixture symbol is provided, the created fixtures will have the same scope. If a name is provided, they will have scope='function'. Note that in practice the performance loss resulting from using
function rather than a higher scope is negligible since the created fixtures' body is a one-liner. - in_cls: a boolean (default
False). You may wish to turn this to True to use this function inside a class. If you do so, you MUST assign the output to variables in the class. - hook: an optional hook to apply to each fixture function that is created during this call. The hook function will be called every time a fixture is about to be created. It will receive a single argument (the function implementing the fixture) and should return the function to use. For example you can use
saved_fixture from pytest-harvest as a hook in order to save all such created fixtures in the fixture store.
Outputs: the created fixtures.
"},{"location":"api_reference/#fixture_union","title":"fixture_union","text":"def fixture_union(name: str,\n fixtures: Iterable[Union[str, Callable]],\n scope: str = \"function\",\n idstyle: Optional[str] = 'compact',\n ids: Union[Callable, Iterable[str]] = None,\n unpack_into: Iterable[str] = None,\n autouse: bool = False,\n hook: Callable = None,\n **kwargs) -> <Fixture>\n
Creates a fixture that will take all values of the provided fixtures in order. That fixture is automatically registered into the callers' module, but you may wish to assign it to a variable for convenience. In that case make sure that you use the same name, e.g. a = fixture_union('a', ['b', 'c'])
The style of test ids corresponding to the union alternatives can be changed with idstyle. Three values are allowed:
'explicit' favors readability with names as <union>/<alternative>,'compact' (default) adds a small mark so that at least one sees which parameters are union alternatives and which others are normal parameters: /<alternative>None or 'nostyle' provides minimalistic ids : <alternative>
See UnionIdMakers class for details.
You can also pass a callable idstyle that will receive instances of UnionFixtureAlternative. For example str leads to very explicit ids: <union>/<idx>/<alternative>. See UnionFixtureAlternative class for details.
Parameters:
name: the name of the fixture to createfixtures: an array-like containing fixture names and/or fixture symbolsscope: the scope of the union. Since the union depends on the sub-fixtures, it should be smaller than the smallest scope of fixtures referenced.idstyle: The style of test ids corresponding to the union alternatives. One of 'explicit', 'compact','nostyle'/None, or a callable (e.g. str) that will receive instances of UnionFixtureAlternative.unpack_into: an optional iterable of names, or string containing coma-separated names, for additional fixtures to create to represent parts of this fixture. See unpack_fixture for details.ids: as in pytest. The default value returns the correct fixtureautouse: as in pytesthook: an optional hook to apply to each fixture function that is created during this call. The hook function will be called every time a fixture is about to be created. It will receive a single argument (the function implementing the fixture) and should return the function to use. For example you can use saved_fixture from pytest-harvest as a hook in order to save all such created fixtures in the fixture store.kwargs: other pytest fixture options. They might not be supported correctly.
Outputs: the new fixture. Note: you do not need to capture that output in a symbol, since the fixture is automatically registered in your module. However if you decide to do so make sure that you use the same name.
"},{"location":"api_reference/#param_fixtures","title":"param_fixtures","text":"def param_fixtures(argnames: str,\n argvalues: Iterable[Any],\n autouse: bool = False,\n ids: Union[Callable, Iterable[str]] = None, \n scope: str = \"function\",\n hook: Callable = None,\n debug: bool = False,\n **kwargs) -> Tuple[<Fixture>]\n
Creates one or several \"parameters\" fixtures - depending on the number or coma-separated names in argnames. The created fixtures are automatically registered into the callers' module, but you may wish to assign them to variables for convenience. In that case make sure that you use the same names, e.g. p, q = param_fixtures('p,q', [(0, 1), (2, 3)]).
Note that the (argnames, argvalues, ids) signature is similar to @pytest.mark.parametrize for consistency, see pytest doc on parametrize.
import pytest\nfrom pytest_cases import param_fixtures, param_fixture\n\n# create a 2-tuple parameter fixture\narg1, arg2 = param_fixtures(\"arg1, arg2\", [(1, 2), (3, 4)])\n\n@pytest.fixture\ndef fixture_uses_param2(arg2):\n ...\n\ndef test_uses_param2(arg1, arg2, fixture_uses_param2):\n ...\n
Parameters:
argnames: same as @pytest.mark.parametrize argnames.argvalues: same as @pytest.mark.parametrize argvalues.autouse: see fixture autouseids: same as @pytest.mark.parametrize idsscope: see fixture scope hook: an optional hook to apply to each fixture function that is created during this call. The hook function will be called every time a fixture is about to be created. It will receive a single argument (the function implementing the fixture) and should return the function to use. For example you can use saved_fixture from pytest-harvest as a hook in order to save all such created fixtures in the fixture store.kwargs: any other argument for the created 'fixtures'
"},{"location":"api_reference/#param_fixture","title":"param_fixture","text":"param_fixture(argname, argvalues, \n autouse=False, ids=None, hook=None, scope=\"function\", **kwargs)\n -> <Fixture>\n
Identical to param_fixtures but for a single parameter name, so that you can assign its output to a single variable.
"},{"location":"api_reference/#parametrize","title":"@parametrize","text":"def parametrize(argnames: str=None,\n argvalues: Iterable[Any]=None,\n indirect: bool = False,\n ids: Union[Callable, Iterable[str]] = None,\n idstyle: Union[str, Callable] = None,\n idgen: Union[str, Callable] = _IDGEN,\n auto_refs: bool = True,\n scope: str = None,\n hook: Callable = None,\n scope: str = \"function\",\n debug: bool = False,\n **args)\n
Equivalent to @pytest.mark.parametrize but also supports
New alternate style for argnames/argvalues. One can also use **args to pass additional {argnames: argvalues} in the same parametrization call. This can be handy in combination with idgen to master the whole id template associated with several parameters. Note that you can pass coma-separated argnames too, by de-referencing a dict: e.g. **{'a,b': [(0, True), (1, False)], 'c': [-1, 2]}.
New alternate style for ids. One can use idgen instead of ids. idgen can be a callable receiving all parameters at once (**args) and returning an id ; or it can be a string template using the new-style string formatting where the argnames can be used as variables (e.g. idgen=lambda **args: \"a={a}\".format(**args) or idgen=\"my_id where a={a}\"). The special idgen=AUTO symbol can be used to generate a default string template equivalent to lambda **args: \"-\".join(\"%s=%s\" % (n, v) for n, v in args.items()). This is enabled by default if you use the alternate style for argnames/argvalues (e.g. if len(args) > 0), and if there are no fixture_refs in your argvalues.
New possibilities in argvalues:
-
one can include references to fixtures with fixture_ref(<fixture>) where can be the fixture name or fixture function. When such a fixture reference is detected in the argvalues, a new function-scope \"union\" fixture will be created with a unique name, and the test function will be wrapped so as to be injected with the correct parameters from this fixture. Special test ids will be created to illustrate the switching between the various normal parameters and fixtures. You can see debug print messages about all fixtures created using debug=True. New: from version 3.2 on, if auto_refs=True (default), @parametrize will automatically detect fixture symbols in the list of argvalues, and will create fixture_refs automatically around them so that you don't need to.
-
one can include lazy argvalues with lazy_value(<valuegetter>, [id=..., marks=...]). A lazy_value is the same thing than a function-scoped fixture, except that the value getter function is not a fixture and therefore can neither be parametrized nor depend on fixtures. It should have no mandatory argument.
Both fixture_ref and lazy_value can be used to represent a single argvalue, or a whole tuple of argvalues when there are several argnames. Several of them can be used in a tuple.
Finally, pytest.param is supported even when there are fixture_ref and lazy_value.
Here as for all functions above, an optional hook can be passed, to apply on each fixture function that is created during this call. The hook function will be called every time a fixture is about to be created. It will receive a single argument (the function implementing the fixture) and should return the function to use. For example you can use saved_fixture from pytest-harvest as a hook in order to save all such created fixtures in the fixture store.
Parameters
-
argnames: same than in @pytest.mark.parametrize
-
argvalues: same as in pytest.mark.parametrize except thatfixture_refandlazy_value` are supported
-
indirect: same as in pytest.mark.parametrize. Note that it is not recommended and is not guaranteed to work in complex parametrization scenarii.
-
ids: same as in pytest.mark.parametrize. Note that an alternative way to create ids exists with idgen. Only one non-None ids or idgen should be provided.
-
idgen: an id formatter. Either a string representing a template, or a callable receiving all argvalues at once (as opposed to the behaviour in pytest ids). This alternative way to generate ids can only be used when ids is not provided (None). You can use the special pytest_cases.AUTO formatter to generate an automatic id with template <name>=<value>-<name2>=<value2>-.... AUTO is enabled by default if you use the alternate style for argnames/argvalues (e.g. if len(args) > 0), and if there are no fixture_refs in your argvalues.
-
idstyle: This is mostly for debug. Style of ids to be used in the \"union\" fixtures generated by @parametrize if at least one fixture_ref is found in the argvalues. idstyle possible values are 'compact', 'explicit' or None/'nostyle' (default), or a callable. idstyle has no effect if no fixture_ref are present in the argvalues. As opposed to ids, a callable provided here will receive a ParamAlternative object indicating which generated fixture should be used.
-
auto_refs: a boolean. If this is True (default), argvalues containing fixture symbols will automatically be wrapped into a fixture_ref, for convenience.
-
scope: The scope of the union fixture to create if fixture_refs are found in the argvalues. Otherwise same as in pytest.mark.parametrize.
-
hook: an optional hook to apply to each fixture function that is created during this call. The hook function will be called every time a fixture is about to be created. It will receive a single argument (the function implementing the fixture) and should return the function to use. For example you can use saved_fixture from pytest-harvest as a hook in order to save all such created fixtures in the fixture store.
-
debug: print debug messages on stdout to analyze fixture creation (use pytest -s to see them)
"},{"location":"api_reference/#lazy_value","title":"lazy_value","text":"
def lazy_value(valuegetter: Callable[[], Any],\n id: str = None,\n marks: Union[Any, Sequence[Any]] = ()\n ) -> LazyValue\n
A reference to a value getter (an argvalue-providing callable), to be used in @parametrize.
A lazy_value is the same thing than a function-scoped fixture, except that the value getter function is not a fixture and therefore can neither be parametrized nor depend on fixtures. It should have no mandatory argument. The underlying function will be called exactly once per test node.
By default the associated id is the name of the valuegetter callable, but a specific id can be provided otherwise. Note that this id does not take precedence over custom ids or idgen passed to @parametrize.
Note that a lazy_value can be included in a pytest.param without problem. In that case the id defined by pytest.param will take precedence over the one defined in lazy_value if any. The marks, however, will all be kept wherever they are defined.
Parameters
valuegetter: a callable without mandatory argumentsid: an optional id. Otherwise valuegetter.__name__ will be used by defaultmarks: optional marks. valuegetter marks will also be preserved.
"},{"location":"api_reference/#is_lazy","title":"is_lazy","text":"def is_lazy(argval) -> bool\n
Return True if argval is the outcome of processing a lazy_value through @parametrize. This encompasses parameters that are items of lazy tuples that are created when parametrizing several argnames with the same lazy_value().
"},{"location":"api_reference/#fixture_ref","title":"fixture_ref","text":"def fixture_ref(fixture: Union[str, Fixture]\n )\n
A reference to a fixture to be used with @parametrize. Create it with fixture_ref(<fixture>) where can be the fixture name or actual fixture function."},{"location":"changelog/","title":"Changelog","text":""},{"location":"changelog/#382-bugfixes-and-project-improvements","title":"3.8.2 - bugfixes and project improvements","text":"
- Fixed issue with upcoming
pytest 8.1 release. PR #322 by @bluetech - Corrected API documentation (and comments) for the second file-name pattern for
AUTO-cases lookup (cases_<name>.py instead of case_<name>.py). PR #320 by @michele-riva. - Fixed
AssertionError on AUTO cases outside a 'normal' test module. Fixes #309. PR #320 by @michele-riva. - Improved error message in case of cases loading error in
@parametrize_with_cases when the cases argument is a string refering to a relative or absolute module name. Fixed import file mismatch with pytest 8 when executing our own tests. Fixes #323. - Fixed failing tests in our builds due to the
event_loop_policy fixture that appeared in pytest-asyncio 0.23. Fixes part of #321.
"},{"location":"changelog/#381-bugfixes","title":"3.8.1 - bugfixes","text":" - Fixed
ScopeMismatch with parametrized cases in non-trivial test trees. scope is now correctly handled for (i) fixture cases, and (ii) fixtures defined in conftest.py files at any depth. Fixes #311. PR #317 by @michele-riva.
"},{"location":"changelog/#380-async-generators-and-strict-markers","title":"3.8.0 - async, generators and strict-markers","text":" @fixture and @parametrize are now async and generator aware. Fixes #286. PR #301 by jgersti.- Fixed error with
pytest --strict-markers. Fixes #283. PR #300 by chrsmcgrr.
"},{"location":"changelog/#370-python-312","title":"3.7.0 - python 3.12","text":" - Added official support for Python 3.10, 3.11 and 3.12. Fixes #314
- Fixed
ModuleNotFoundError: distutils on Python 3.12 thanks to packaging. PR #312 by @jayqi. - Internal: switched to virtualenv backend.
"},{"location":"changelog/#3614-bugfixes","title":"3.6.14 - bugfixes","text":" - Fixed
AttributeError issue in is_case_function when an inspected symbol is a parametrized type hint without __name__. Fixes #287 - Fixed issue with
get_all_cases: default value for cases was wrong. Fixes #290
"},{"location":"changelog/#3613-bugfix","title":"3.6.13 - bugfix","text":" - Fixed issue where a lazy value (for example a case function) was not resolved before being injected in a parametrized function, and was therefore appearing as a
_LazyValueCaseParamValue. Fixed #274
"},{"location":"changelog/#3612-type-hint-fix-enhanced-compatibility-with-pytest-plugins","title":"3.6.12 - type hint fix + enhanced compatibility with pytest plugins","text":" - Improved compatibility with other
pytest plugins, in particular pytest-repeat, by supporting removal from fixture closure tree. Fixed #269. - Fixed type hint errors detected by
pyright. Fixed #270
"},{"location":"changelog/#3611-bugfix-for-pytest-xdist-and-get_all_cases-api-improvement","title":"3.6.11 - bugfix for pytest-xdist and get_all_cases API improvement","text":" get_all_cases can now be called without parametrization_target (defaulting to the caller module), and with an explicit module object. Fixed #258. PR #260 by @eddiebergman.- Fixed
AttributeError: module 'pytest_cases' has no attribute 'CasesCollectionWarning' when running pytest-xdist and at least one cases class is ignored because of __init__ or __new__. Fixed #249.
"},{"location":"changelog/#3610-bugfix-for-pytest-71","title":"3.6.10 - bugfix for pytest 7.1","text":" - Fixed
ImportError when using pytest 7.1. Fixed #264 and pytest-dev#9762.
"},{"location":"changelog/#369-bugfix-with-pytest-7","title":"3.6.9 - Bugfix with pytest 7","text":" - Fixed
FrozenInstanceError when using pytest 7.0.0. Fixed #251. PR#253 by jammer87
"},{"location":"changelog/#368-bugfix-support-for-multiprocessing-pool","title":"3.6.8 - Bugfix: support for multiprocessing Pool","text":" - Suppressed warnings in our own tests, to improve packaging maintenance. Fixed #248
- Fixed bug where setting
ids in @parametrize without setting explicitly idgen=None would raise a ValueError. Fixed #238. - Fixed bug where case-holding class marks were not propagated to static methods and class methods. Fixed #246
- Fixed support for multiprocessing
Pool. Fixes #242
"},{"location":"changelog/#367-minor-improvements-and-preparing-for-pytest-7","title":"3.6.7 - Minor improvements and preparing for pytest 7","text":" - Improved error message when a case function nested in a class has no
self argument and is not static. Fixes #243 - Added support for the new Scopes enum in pytest 7. Fixed #241
- Fixed
__version__ in development mode.
"},{"location":"changelog/#366-layout-change","title":"3.6.6 - Layout change","text":" - Restructured project so that tests are truly independent, to ease rpm/apt/etc. packaging. Fixed #220.
"},{"location":"changelog/#365-bugfix","title":"3.6.5 - Bugfix","text":" - Fixed an issue where using keyword
argnames in @pytest.mark.parametrize would cause IndexError: tuple index out of range in the tests collection phase. Fixed #234.
"},{"location":"changelog/#364-bugfix","title":"3.6.4 - Bugfix","text":" - A case id can now be a reserved keyword without triggering any
SyntaxError, even if the case is transformed into a fixture. Fixes #230
"},{"location":"changelog/#363-bugfix","title":"3.6.3 - Bugfix","text":" - Fixed an issue where a lazy value would not be resolved. This happens when the \"auto-simplify fixture\" happens in
@parametrize. Fixes #225
"},{"location":"changelog/#362-qualimetry","title":"3.6.2 - Qualimetry","text":" - Fixed most
flake8 errors and updated documentation to use genbadge. Fixes #223
"},{"location":"changelog/#361-bugfix-parametrizing-a-class-with-parametrize","title":"3.6.1 - bugfix - parametrizing a class with @parametrize","text":" - Fixed
ValueError when @parametrize is used to parametrize a class. Also, added a more explicit TypeError when @parametrize is used to parametrize a class and at least a fixture reference is present. Fixed #215.
"},{"location":"changelog/#360-unpack_fixtures-in-classes-current_cases-improvements","title":"3.6.0 - unpack_fixtures in classes + current_cases improvements","text":" -
The current_cases fixture now contains case parameters if any. Fixes #214
-
The current_cases fixture entries are now instances of namedtuple.
-
New in_cls argument in unpack_fixtures so that it can be used inside classes. Fixes #201
-
Fixed minor issue where empty entries could be present in currentcases. Fixes #213
"},{"location":"changelog/#352-bugfix-with-the-currentcases-fixture","title":"3.5.2 - bugfix with the currentcases fixture","text":" - Fixed issues where the
currentcases fixture would not return the correct case function. Fixed #212
"},{"location":"changelog/#351-python-310-compatibility-improved-error-message","title":"3.5.1 - python 3.10 compatibility + improved error message","text":" - Fixed error message related to misuse of
fixture_ref. Fixes #209 - Fixed import error with python 3.10. Fixes #207
"},{"location":"changelog/#350-new-current_cases-fixture-and-get_current_cases-function-fixes","title":"3.5.0 - New current_cases fixture and get_current_cases function + Fixes","text":" -
New: Users can now easily access the current cases for each parametrized argument thanks to the new current_cases fixture. A new helper function get_current_cases is also provided, for direct access from a hook. get_current_case_id becomes deprecated in favour of these two. Fixes #195
-
Bugfix: Fixed issue where the cache of a lazy_value used for a tuple of parameters (several argnames) was not considering the pytest context and thus was wrongly used across pytest nodes. Fixes #202
-
Improved error message when a fixture parametrized with several argnames as once is not able to unpack the parameter values received (non subscriptable object).
-
parametrize_plus and fixture_plus are now deprecated in favour of parametrize and fixture, as most users seem to have adopted these names without issues.
-
(internal) Replaced the \"used\" parameter with a dedicated singleton USED
"},{"location":"changelog/#346-increased-compatibility-with-other-plugins","title":"3.4.6 - Increased compatibility with other plugins","text":" LazyValue, LazyTuple and LazyTupleItem are now hashable. This increases compatibility with plugins hashing the parameter values, such as pytest-steps. See pytest-steps#41 . Fixes #199
"},{"location":"changelog/#345-bugfix","title":"3.4.5 - Bugfix","text":" - Fixed bug when a test module containing
@parametrize_with_cases was executed outside of pytest, typically through its __main__. Fixes #198
"},{"location":"changelog/#344-bugfix","title":"3.4.4 - Bugfix","text":" - Fixed issue when
@parametrize_with_cases was used on a fixture in a conftest.py. Fixes #196
"},{"location":"changelog/#343-technical-release-zenodo","title":"3.4.3 - Technical release - Zenodo","text":"Technical release to check that Zenodo metadata is now preserved. Same as 3.4.2, 3.4.1 and 3.4.0.
"},{"location":"changelog/#340-goodbye-v1-api-support-for-unbound-cases-bugfix-with-marks-fixtures-in-case-files","title":"3.4.0 - Goodbye v1 API + Support for unbound cases + Bugfix with marks + Fixtures in case files","text":" -
Legacy v1 API was dropped. Fixes #192
-
Unbound case functions in a class (e.g. Foo.bar) can now be directly passed to parametrize_with_cases without instantiating the class, e.g. parametrize_with_cases(cases=Foo.bar). Fixes #159
-
Fixed bug with concatenation of marks on cases. Fixes #191
-
Fixed an issue where a case transformed into a fixture, with the same name as the fixture it requires, would lead to a pytest fixture recursion.
-
Fixtures in case files can now be automatically imported using the experimental @parametrize_with_cases(import_fixtures=True). Fixes #193
"},{"location":"changelog/#330-filter-helpers-current-id-getter-support-for-pytest-asyncio-and-other-plugins","title":"3.3.0 - Filter helpers, Current id getter, Support for pytest-asyncio and other plugins","text":" -
Migrated to Github Actions + nox for CI/CD.
-
New helper function get_current_case_id to get the current case id for a given pytest request or item. Fixes #189
-
Extended the support of fixture closure modifications to remove and insert(0, f). This in particular solves an issue with pytest-asyncio. Fixes #176
-
New filters module providing helper functions has_tag, id_has_prefix, id_has_suffix, id_match_regex to easily create custom filters for use in @parametrize_with_cases(filter=...). PR #184 by @saroad2, (thanks !).
"},{"location":"changelog/#321-fixture-bugfix","title":"3.2.1 - @fixture Bugfix","text":" - Fixed
fixture 'self' not found issue when @fixture was used to decorate a class method not explicitly depending on request. Fixed #182
"},{"location":"changelog/#320-automatic-fixture_ref-test-ordering-bugfix","title":"3.2.0 - Automatic fixture_ref + test ordering bugfix","text":" -
New: from version 3.2 on, if auto_refs=True (default), @parametrize will automatically detect fixture symbols in the list of argvalues, and will create fixture_refs automatically around them so that you don't need to. Fixes #177
-
Fixed ordering issue happening on linux targets when several @parametrize are used to decorate the same function. Fixes #180
"},{"location":"changelog/#312-bugfixes-with-nesting-and-pytest-asyncio","title":"3.1.2 - Bugfixes with nesting and pytest-asyncio","text":" -
Now appending fixtures to the closure once it has been built is supported. This fixes an issue with pytest-asyncio. Fixes #176
-
Fixed issue when parametrize_with_cases was used on case functions themselves (nesting/recursion). This was due to a lack of support of the place_as magic pytest attribute. Fixes #179
-
Added a warning concerning usage of indirect in parametrize when fixture references are present. See #150
"},{"location":"changelog/#311-bugfix-with-ids","title":"3.1.1 - Bugfix with ids","text":" - Fixed issue with Empty id marker leaking to test ids. Fixed #171
"},{"location":"changelog/#310-improved-cases-collection","title":"3.1.0 - Improved cases collection","text":" -
@parametrize_with_cases now by default (cases=AUTO) looks for both file naming patterns test_<name>_cases.py and cases_<name>.py. Removed the AUTO2 constant. Fixed #140
-
Nested classes containing case functions are now officially supported (they were, but undocumented). Fixed #160
-
Case functions that are staticmethod and classmethod are now supported as well. Fixes #168
"},{"location":"changelog/#300-harmonization-of-ids-and-public-api-for-cases-info","title":"3.0.0 - harmonization of ids and public API for cases info","text":" -
Major refactoring of the way ids and marks are generated and customized in fixture_union, @parametrize and @parametrize_with_cases. Now idstyle has a consistent behaviour across the board, ids and idstyle can work together correctly, @parametrize_with_cases and @parametrize have much better default values for ids, and many others. See documentation for details. Fixed #154
-
New public API to manipulate information about a case function: copy_case_info, set_case_id, get_case_id, get_case_marks, get_case_tags, matches_tag_query, is_case_class, is_case_function. See API reference.
-
Fixed default behaviour of idgen in @parametrize: it only defaults to AUTO when no fixture_ref are used in the argvalues.
"},{"location":"changelog/#272-bugfix-with-doctest","title":"2.7.2 - Bugfix with doctest","text":" - Fixed
AttributeError: 'DoctestItem' object has no attribute '_request' when executing doctests. Fixes #156
"},{"location":"changelog/#271-pytestmarkusefixtures-can-be-used-on-case-functions","title":"2.7.1 - @pytest.mark.usefixtures can be used on case functions","text":" @pytest.mark.usefixtures can be now be used on case functions. Fixes #152.
"},{"location":"changelog/#270-parametrize_with_cases-now-supports-id-customization","title":"2.7.0 - @parametrize_with_cases now supports id customization","text":" @parametrize_with_cases now explicitly supports all id customization methods (ids, idgen and idstyle) supported by @parametrize (ids, idgen and idstyle). Updated documentation accordingly. Fixed #151
"},{"location":"changelog/#260-better-cache-for-lazy-values-and-support-for-infinite-id-generators","title":"2.6.0 - better cache for lazy values and support for infinite id generators","text":" -
lazy_value parameters are now cached by pytest node id only. So plugins can access the value without triggering an extra function call, but a new call is triggered for each pytest node, so as to prevent mutable object leakage across tests. Fixed #149 while ensuring no regression for #143.
-
The ids argument of parametrize now accepts a (possibly infinite) generator of ids, e.g. (f\"foo{i}\" for i in itertools.count()), just as pytest does. This was not always the case, inparticular when parametrizing a @fixture. The ids arguments of fixture_union, param_fixture[s], etc. now also support this pattern. Fixed #148
"},{"location":"changelog/#250-case-ids-glob-match-improvements","title":"2.5.0 - case ids glob match improvements","text":" - Improved description for the
glob argument in @parametrize_with_cases. Also made the implementation escape all regex special characters so that they can't be used. Finally a pattern should now match the entire case id (previously, a partial match would work if it was at the beginning of the string). One step towards #147
"},{"location":"changelog/#240-various-fixes-for-test-ids-and-lazy-values","title":"2.4.0 - various fixes for test ids and lazy values","text":" -
is_lazy is now part of public API, and _LazyValue now has a cache mechanism like _LazyTuple. Fixes #143
-
@parametrize: custom ids are now correctly taken into account when a single lazy_valueis used for a tuple of parameters. This issue could be seen also with @parametrize_with_cases: idgen does not seem to be taken into account when cases are unpacked into a tuple. Fixes #144.
-
Empty case ids are now replaced with '<empty_case_id>' to avoid ambiguous interpretation of test ids. Fixes #142.
"},{"location":"changelog/#230-better-lazyvalue-internal-api","title":"2.3.0 - better LazyValue internal API","text":" - new
clone(self, remove_int_base=False) API on LazyValue and LazyTupleItem instances. With this new API, on old pytest < 5.3, other plugins such as pytest-harvest can easily clone the contents from lazy values without having them inherit from int - which was a dirty hack used by pytest-cases to trick pytest to generate acceptable test ids in these old pytest versions. Also improved the LazyValue, LazyTuple and LazyTupleItem object model with equality and repr. Fixes pytest-harvest#43
"},{"location":"changelog/#225-marks-are-now-correctly-propagated-from-case-class","title":"2.2.5 - Marks are now correctly propagated from Case class","text":" - Marks set on a case class are now propagated to cases in the class. So you can use for example
pytest-pilot more easily ! Fixes #139
"},{"location":"changelog/#224-fixes-issue","title":"2.2.4 - Fixes issue","text":" - Fixed \"Created fixture names are not unique, please report\" error when duplicate fixture reference is provided in a pytest.param. Fixes #138.
"},{"location":"changelog/#223-fixed-issue-with-pytest-3x","title":"2.2.3 - Fixed issue with pytest 3.X","text":" - Fixed
TypeError: _idval() got an unexpected keyword argument 'item' with pytest versions between 3.0.0 and 3.7.4. Fixed #136
"},{"location":"changelog/#222-parametrize_with_cases-compatibility-improvements","title":"2.2.2 - @parametrize_with_cases compatibility improvements","text":" @parametrize_with_cases now supports that argnames is a list or tuple, just as @pytest.mark.parametrize does. PR #132, by @saroad2.
"},{"location":"changelog/#221-setuppy-fix-to-enforce-dependency-version","title":"2.2.1 - setup.py fix to enforce dependency version","text":" - Now enforcing usage of
makefun 1.9.3 or above to avoid issue AttributeError: 'functools.partial' object has no attribute '__module__' mentioned in #128
"},{"location":"changelog/#220-doc-improvements-bugfix-for-cases-requiring-fixtures","title":"2.2.0 - Doc improvements + bugfix for cases requiring fixtures","text":" -
Improved documentation to explain why @fixture should be used instead of @pytest.fixture. Fixed #125
-
Fixed ValueError: fixture is being applied more than once to the same function when two functions parametrized with the same cases were sitting in the same file. Improved robustness when cases require fixtures, in particular when parametrized test/fixture sits in a class or when several of them sit in a class/module. Fixed #126
"},{"location":"changelog/#213-missing-deprecation-warning","title":"2.1.3 - Missing deprecation warning","text":" -
Added missing deprecation warning on @cases_generator. Fixes #124.
-
Removed target and tags arguments of @cases_generator (deprecated api anyway) that were added by mistake in version 2.0.0 but never used.
"},{"location":"changelog/#212-compatibility-fix","title":"2.1.2 - Compatibility fix","text":" - Added support for pytest items without funcargs. Fixes interoperability with other pytest plugins such as
pytest-black or pytest-flake8. Fixes #122
"},{"location":"changelog/#211-fixed-issue-with-pytest-6","title":"2.1.1 - Fixed issue with pytest 6","text":"pytest 6 is now supported. Fixes #121
"},{"location":"changelog/#210-internal-engine-improvements-bugfixes","title":"2.1.0 - Internal engine improvements + bugfixes","text":"Fixed issue with @parametrize_with_cases when two cases with the same id and both requiring a fixture were to be created. Fixed #117.
Fixture closure engine refactoring:
-
When no fixture unions are present, the fixture closure is now identical to the default one in pytest, to avoid issues originating from other plugins fiddling with the closure. Fixes #116
-
New SuperClosure class representing the \"list\" facade on top of the fixture tree (instead of FixtureClosureNode). In addition, this list facade now better handles editing the order of fixtures when possible. Fixes #111.
-
Session and Module-scoped fixtures that are not used in all union alternatives are not any more torn town/setup across union alternatives. Fixes #120
"},{"location":"changelog/#204-bugfix","title":"2.0.4 - Bugfix","text":" - Fixed
TypeError with iterable argvalue in standard parametrize. Fixed #115.
"},{"location":"changelog/#203-bugfixes","title":"2.0.3 - Bugfixes","text":""},{"location":"changelog/#202-better-string-representation-for-lazy-values","title":"2.0.2 - Better string representation for lazy values","text":"Lazy values (so, test cases) now have a much nicer string representation ; in particular in pytest-harvest results tables. Fixes #112
"},{"location":"changelog/#201-better-test-ids-and-theory-page","title":"2.0.1 - Better test ids and theory page","text":""},{"location":"changelog/#200-less-boilerplate-full-pytest-alignment","title":"2.0.0 - Less boilerplate & full pytest alignment","text":"I am very pleased to announce this new version of pytest-cases, providing a lot of major improvements. Creating powerful and complex test suites have never been so easy and intuitive !
Below is a complete list of changes, but the user guide has also been updated accordingly so feel free to have a look to get a complete example-based walkthrough.
A/ More powerful and flexible cases collection
New @parametrize_with_cases decorator to replace @cases_data (deprecated).
-
Aligned with pytest:
-
now argnames can contain several names, and the case functions are automatically unpacked into it. You don't need to perform a case.get() in the test anymore !
@parametrize_with_cases(\"a,b\")\ndef test_foo(a, b):\n # use a and b directly !\n ...\n
-
cases are unpacked at test setup time, so the clock does not run while the case is created - in case you use pytest-harvest to collect the timings.
-
@parametrize_with_cases can be used on test functions as well as fixture functions (it was already the case in v1)
-
Easier to configure:
-
the decorator now has a single cases argument to indicate the cases, wherever they come from (no module argument anymore)
-
default (cases=AUTO) automatically looks for cases in the associated case module named test_xxx_cases.py. Users can easily switch to alternate pattern cases_xxx.py with cases=AUTO2. Fixes #91.
-
cases can sit inside a class, like what you're used to do with pytest. This additional style makes it much more convenient to organize cases and associated them with tests, when cases sit in the same file than the tests. Fixes #93.
-
an explicit sequence can be provided, it can mix all kind of sources: functions, classes, modules, and module names as strings (even relative ones!).
@parametrize_with_cases(\"a\", cases=(CasesClass, '.my_extra_cases'))\ndef test_foo(a):\n ...\n
-
More powerful API for filtering:
-
a new prefix argument (default case_) can be used to define case functions for various type of parameters: welcome user_<id>, data_<id>, algo_<id>, model_<id> ! Fixes #108
-
a new glob argument receiving a glob-like string can be used to further filter cases based on their names. For example you can distinguish *_success from *_failure case ids, so as to dispatch them to the appropriate positive or negative test. Fixes #108
-
finally you can still use has_tag and/or provide a filter callable, but now the callable will receive the case function, and this case function has a f._pytestcase attribute containing the id, tags and marks - it is therefore much easier to implement custom filtering.
B/ Easier-to-define case functions
-
Case functions can start with different prefixes to denote different kind of data: e.g. data_<id>, user_<id>, model_<id>, etc.
-
Case functions can now be parametrized with @parametrize or @pytest.mark.parametrize, just as in pytest ! This includes the ability to put pytest marks on the whole case, or on some specific parameter values using pytest.param. @cases_generator is therefore now deprecated but its alternate style for ids and arguments definition was preserved in @parametrize, see below.
-
Now case functions can require fixtures ! In that case they will be transformed into fixtures and injected as fixture_ref in the parametrization. Fixes #56.
-
New single optional @case(id=None, tags=(), marks=()) decorator to replace @case_name and @case_tags (deprecated): a single simple way to customize all aspects of a case function. Also, @test_target completely disappears from the picture as it was just a tag like others - this could be misleading.
C/ Misc / pytest goodies
-
New aliases for readability: @fixture for @fixture_plus, and@parametrize for @parametrize_plus (both aliases will coexist with the old names). Fixes #107.
-
@parametrize was improved in order to support the alternate parametrization mode that was previously offered by @cases_generator, see api reference. That way, users will be able to choose the style of their choice. Fixes #57 and #106.
-
@parametrize now raises an explicit error message when the user makes a mistake with the argnames. Fixes #105.
-
More readable error messages in @parametrize when lazy_value does not return the same number of argvalues than expected from the argnames.
-
Any error message associated to a lazy_value function call is not caught and hidden anymore but is emitted to the user, for easier debugging.
-
Fixed issue with lazy_value when a single mark is passed in the constructor.
-
lazy_value used as a tuple for several arguments now have a correct id generated even in old pytest version 2.
-
New pytest goodie assert_exception that can be used as a context manager. Fixes #104.
"},{"location":"changelog/#1170-lazy_value-improvements-annoying-warnings-suppression","title":"1.17.0 - lazy_value improvements + annoying warnings suppression","text":" -
lazy_value are now resolved at pytest setup stage, not pytest call stage. This is important for execution time recorded in the reports (see also pytest-harvest plugin). Fixes #102
-
A function used as a lazy_value can now be marked with pytest marks. Fixes #99
-
A lazy_value now has a nicer id when it is a partial. Fixes #97
-
Removed annoying PytestUnknownMarkWarning warning message when a mark was used on a case. Fixes #100
"},{"location":"changelog/#1160-new-lazy_value-for-parameters","title":"1.16.0 - New lazy_value for parameters","text":" - New marker
lazy_value for parametrize_plus. Fixes #92
"},{"location":"changelog/#1150-better-parametrize_plus-and-smaller-dependencies","title":"1.15.0 - better parametrize_plus and smaller dependencies","text":" -
Better support for pytest.param in parametrize_plus and also in fixture_union and fixture_param[s]. Improved corresponding ids. Fixed #79 and #86
-
New @ignore_unused decorator to protect a fixture function from the \"NOT_USED\" case happening when the fixture is used in a fixture union.
-
Removed six, wrapt and enum34 dependencies
-
(Internal) submodules reorganization for readability
- (Internal) suppressed a lot of code quality warnings
"},{"location":"changelog/#1140-bugfixes-and-hook-feature","title":"1.14.0 - bugfixes and hook feature","text":" -
Fixed ids precedence order when using pytest.mark.parametrize in a fixture_plus. Fixed #87
-
Fixed issue with fixture_union when using the same fixture twice in it. Fixes #85
-
Added the possibility to pass a hook function in all API where fixtures are created behind the scenes, so as to ease debugging and/or save fixtures (with stored_fixture from pytest harvest). Fixes #83
-
Fixture closures now support reordering when no unions are present. This suppressed the annoying warning \"WARNING the new order is not taken into account !!\" when it was not relevant. Fixes #81
"},{"location":"changelog/#1131-packaging-improvements","title":"1.13.1 - packaging improvements","text":" - packaging improvements: set the \"universal wheel\" flag to 1, and cleaned up the
setup.py. In particular removed dependency to six for setup and added py.typed file. Fixes #78
"},{"location":"changelog/#1130-cases_generator-default-names","title":"1.13.0 - @cases_generator default names","text":"@cases_generator now has a default value for the names template, based on the parameters. Fixes #77.
"},{"location":"changelog/#1124-bugfix","title":"1.12.4 - Bugfix","text":"Fixed ValueError when a product of unions was used on a test node, for example when two parametrize_plus using fixture_refs were used on the same fixture or test function. Fixed #76
"},{"location":"changelog/#1123-improved-error-messages","title":"1.12.3 - Improved error messages","text":"Improved error message when something that is not a fixture is used in unpack_fixture or fixture_union. Fixed #75
"},{"location":"changelog/#1122-warning-fix","title":"1.12.2 - Warning fix","text":"Fixed deprecation warning #74
"},{"location":"changelog/#1121-bugfixes","title":"1.12.1 - Bugfixes","text":" -
Now using module name and not file path to detect symbols in cases files that are imported from elsewhere and not created locally. Indeed that was causing problems on some ^platforms where a .pyc cache file is created. Fixes #72
-
Fixed PluginValidationError when pytest_fixture_plus or pytest_parametrize_plus were used in a conftest.py file. Fixes #71. According to discussion in pytest#6475, pytest_fixture_plus and pytest_parametrize_plus are now renamed to fixture_plus and parametrize_plus in order for pytest (pluggy) not to think they are hooks. Old aliases will stay around for a few versions, with a deprecation warning.
"},{"location":"changelog/#1120-better-test-ids-for-parametrized-tests-with-fixture-refs-bugfix","title":"1.12.0 - better test ids for parametrized tests with fixture refs + bugfix","text":" -
Improved test ids for the cases where fixture_ref is used in the parameters list in @pytest_parametrize_plus. Fixed #69. Thanks last-partizan for the suggestion !
-
Fixed TypeError: got an unexpected keyword argument 'indirect' with pytest 5+. Fixed #70.
"},{"location":"changelog/#1119-bugfix","title":"1.11.9 - bugfix","text":"FixtureClosureNode is now able to properly handle ignore_args, and now supports that plugins append fixtures to the closure, such as pytest-asyncio. Added corresponding tests. Fixes #68
"},{"location":"changelog/#1118-bugfix","title":"1.11.8 - bugfix","text":"Fixed KeyError issue happening when a fixture is not found. Now users will see the \"normal\" error message from pytest (\"fixture <name> not found\"). Fixed #67.
"},{"location":"changelog/#1117-bugfix","title":"1.11.7 - bugfix","text":"Fixed ValueError issue happening with indirectly parametrized fixtures. Fixed #64.
"},{"location":"changelog/#1116-pyprojecttoml","title":"1.11.6 - pyproject.toml","text":"raddessi added a pyproject.toml - thanks! Fixed #65.
"},{"location":"changelog/#1115-bugfix","title":"1.11.5 - bugfix","text":"pytest_parametrize_plus was not working correctly with test classes, leading to fixture 'self' not found. Fixed #63.
"},{"location":"changelog/#1114-python-2-bugfix","title":"1.11.4 - python 2 bugfix","text":"Fixed issue happening with @pytest.mark.parametrize with python 2. Fixed #62.
"},{"location":"changelog/#1113-minor-improvements","title":"1.11.3 - minor improvements","text":"Better error message when users use THIS_MODULE in cases= instead of module=.
Added __version__ package-level attribute.
"},{"location":"changelog/#1112-increased-tolerance-to-other-plugins-bugfix","title":"1.11.2 - Increased tolerance to other plugins + bugfix","text":"Now when other plugins try to manipulate the fixture closure, warning messages are emitted but no error is raised. Fixed #55.
Also fixed issue #58 happening with doctest.
"},{"location":"changelog/#1111-added-six-dependency-explicitly","title":"1.11.1 - Added six dependency explicitly","text":"It was missing from setup.py.
"},{"location":"changelog/#1110-fixture_ref-can-now-be-used-inside-tuples-leading-to-cross-products","title":"1.11.0 - fixture_ref can now be used inside tuples, leading to cross-products","text":"Fixes #47.
"},{"location":"changelog/#1102-more-intuitive-error-messages","title":"1.10.2 - More intuitive error messages","text":"Now raising an explicit InvalidParamsList when pytest parametrize argvalues are incorrect. See #54
"},{"location":"changelog/#1101-bugfix","title":"1.10.1 - Bugfix","text":"Fixed #52.
"},{"location":"changelog/#1100-new-feature-fixtures-unpacking","title":"1.10.0 - New feature: fixtures unpacking","text":"You can now unpack a fixture iterable into several individual fixtures using unpack_fixture or using @pytest_fixture_plus(unpack_into=<names>). This is also available in union_fixture(unpack_into=<names>). Fixed #50 and #51.
"},{"location":"changelog/#193-bugfix","title":"1.9.3 - Bugfix","text":"Fixed issues when parametrize argnames contains a list. This fixed #49
"},{"location":"changelog/#192-bugfix-with-pytest-37","title":"1.9.2 - Bugfix with pytest 3.7","text":"Fixed #48.
"},{"location":"changelog/#191-bugfix-with-pytest-37","title":"1.9.1 - Bugfix with pytest 3.7","text":"Fixed #48.
"},{"location":"changelog/#190-new-with-reorder-commandline-option","title":"1.9.0 - New --with-reorder commandline option","text":"New commandline option '--with-reorder' to change the reordering strategy currently in application. Fixes #45.
The --with-reorder \"skip\" mode was not working correctly in presence of marks, fixed it. Fixed #46.
"},{"location":"changelog/#181-bugfixes","title":"1.8.1 - BugFixes","text":"Ids should not be used when setting a NOT_USED parametrization. Fixes #43
Fixed issue with ordering and setup/teardown for higher-level scope fixtures (session and module scopes) when using union fixtures. Fixes #44
"},{"location":"changelog/#180-better-ids-for-fixture-unions","title":"1.8.0 - Better ids for fixture unions","text":"New:
fixture_union now accept a non-None value for ids. It also has a new idstyle argument allowing users to change the style of ids used. Finally pytest_parametrize_plus relies on this ids argument to set a more readable list of ids for the created union. Fixes #41.
Misc:
- Added non-regression test for fixture order. It passes already for all recent pytest versions (after 3.3). Fixes #42
"},{"location":"changelog/#170-new-pytest_parametrize_plus-allowing-fixture-references-to-be-used-in-parameter-values","title":"1.7.0 - New @pytest_parametrize_plus allowing fixture references to be used in parameter values","text":"New decorator @pytest_parametrize_plus able to handle the case where a fixture_ref(<fixture_name>) is present in the parameter values list. This decorator can be applied both on test functions and fixtures (if they are decorated with @pytest_fixture_plus). Fixes #40
Major refactoring of the \"union fixtures\" mechanism.
- The
NOT_USED status is now correctly propagated between dependent fixtures. This should fix a few cases where user fixtures were setup/teardown while not used in the current test node. - Empty fixture unions are not permitted anymore.
- The way unions are handled in test parametrization was redesigned. The new design is based on a two-steps approach: first build the fixture closure for each node as a tree (and not a list as in
pytest), and then apply parametrization intelligently based on this tree structure. This fixes several unintuitive behaviours that were happening with unions.
Note: interestingly this also fixes pytest#5054.
"},{"location":"changelog/#163-minor-exception-enhancement","title":"1.6.3 - Minor exception enhancement","text":"Improved the error message when the name template is wrong in @cases_generator. Fixes #39.
"},{"location":"changelog/#162-bug-fixes","title":"1.6.2 - bug fixes","text":"fixture_union:
param_fixtures:
-
param_fixtures now delegates to param_fixture when a single parameter name is provided. This is more consistent. Fixed #36.
-
param_fixture[s] now support all arguments from fixture (scope and autouse in particular).
"},{"location":"changelog/#161-pytest_fixture_plus-improvement-to-handle-not_used-cases","title":"1.6.1 - @pytest_fixture_plus improvement to handle NOT_USED cases","text":"Fixed issue where fixtures get called with NOT_USED as a parameter when using a fixture_union. This issue is actually only fixed in @pytest_fixture_plus, if you use @pytest.fixture you have to handle it manually. Fixes #37.
"},{"location":"changelog/#160-fixture_union-and-param_fixtures-bugfix","title":"1.6.0 - fixture_union and param_fixture[s] bugfix","text":"New fixture_union method to create a fixture that is the union/combination of other fixtures. This is an attempt to solve this pytest proposal.
Also, param_fixture and param_fixtures can now be used without necessarily storing the return value into a variable: they will automatically register the created fixtures in the calling module.
Finally, fixed a bug with param_fixtures when called to create a fixture for a single parameter.
"},{"location":"changelog/#151-param_fixtures-bugfix","title":"1.5.1 - param_fixtures bugfix","text":"Fixed param_fixtures issue: all parameter values were identical to the last parameter of the tuple. Fixes #32.
"},{"location":"changelog/#150-new-helpers-param_fixture-and-param_fixtures","title":"1.5.0 - new helpers param_fixture and param_fixtures","text":"Following Sup3rGeo's proposal, introduced two helper methods to create simple \"parameter fixtures\". Fixes #31.
"},{"location":"changelog/#142-parametrized-pytest_fixture_plus-minor-bug-fix","title":"1.4.2 - parametrized @pytest_fixture_plus minor bug fix","text":"@pytest_fixture_plus now correctly honors parameter id and marks overridden at single parameter level using pytest.param. Fixed #30.
"},{"location":"changelog/#141-parametrized-pytest_fixture_plus-minor-bug-fix","title":"1.4.1 - parametrized @pytest_fixture_plus minor bug fix","text":"Fixed @pytest_fixture_plus in case it is used with parametrize and one parameter is itself customized using pytest.param. Fixed #29.
"},{"location":"changelog/#140-pytest_fixture_plus-major-improvement","title":"1.4.0 - @pytest_fixture_plus major improvement","text":" -
Major improvement of @pytest_fixture_plus: instead of generating fixtures, it now correctly parametrizes the fixture. Skip/fail Marks are correctly copied too. Fixes #28.
-
pytest_fixture_plus does not accept the params and ids arguments any more, it only relies on parametrization marks.
"},{"location":"changelog/#133-parametrized-pytest_fixture_plus-bugfix","title":"1.3.3 - parametrized @pytest_fixture_plus Bugfix","text":"Fixed minor bug with parametrized @pytest_fixture_plus: spaces are now correctly removed when multiple parameter names are provided in the same parametrize call. Fixes #27.
"},{"location":"changelog/#132-parametrized-pytest_fixture_plus-bugfix","title":"1.3.2 - parametrized @pytest_fixture_plus Bugfix","text":"Fixed bug with @pytest_fixture_plus when used in parametrized mode. Fixes #26. Thanks Sup3rGeo!
"},{"location":"changelog/#131-minor-dependency-change","title":"1.3.1 - Minor dependency change","text":"Now using decopatch to create the decorators.
"},{"location":"changelog/#130-more-flexible-case-generators-names-minor-dependency-change","title":"1.3.0 - More flexible case generators names + Minor dependency change","text":"Cases generators can now support explicit name lists, and name generator callables, in addition to the name template strings. Fixed #24.
Dependency to decorator has been dropped and replaced with makefun. Fixed #25.
"},{"location":"changelog/#122-fixed-bug-with-marks-on-cases-with-pytest-33","title":"1.2.2 - fixed bug with marks on cases with pytest 3.3","text":"Marks on cases are now also working with pytest 3.3. Fixed #23.
Ids for marked tests are now better managed. A new function get_pytest_parametrize_args is now used to transform the list of cases obtained by get_all_cases(module), into the list of marked cases and ids required by @pytest.mark.parametrize. The doc has been updated to explain this for advanced users wishing to perform this step manually.
"},{"location":"changelog/#121-fixed-id-of-test-cases-with-marks","title":"1.2.1 - fixed id of test cases with marks","text":"Id of test cases with marks was appearing as ParameterSet. Fixed it.
"},{"location":"changelog/#120-pytestmark-can-be-used-on-cases-pytest_fixture_plus-parametrization-order-bugfix","title":"1.2.0 - @pytest.mark can be used on cases + @pytest_fixture_plus parametrization order bugfix","text":"Pytest marks such as @pytest.mark.skipif can now be used on case functions. As a consequence, get_all_cases is now the recommended function to use instead of extract_cases_from_module to perform manual collection. Indeed get_all_cases correctly prepares the resulting parameters list so that pytest sees the marks. Fixed #21.
Fixed parametrization order when @pytest_fixture_plus is used with several @pytest.mark.parametrize. Fixed #22.
"},{"location":"changelog/#111-improved-generated-fixture-names-for-pytest_fixture_plus","title":"1.1.1 - Improved generated fixture names for @pytest_fixture_plus","text":"When @pytest_fixture_plus is used on a function marked as parametrized, some fixtures are generated (one for each parameter). Generated fixture names now follow the pattern <fixturename>__<paramname>. Fixed #20.
"},{"location":"changelog/#110-new-pytest_fixture_plus","title":"1.1.0 - New @pytest_fixture_plus","text":"New decorator @pytest_fixture_plus allows to use several @pytest.mark.parametrize on a fixture. Therefore one can use multiple @cases_data decorators, too. Fixes #19. Note: this is a temporary feature, that will be removed if/when pytest supports it.
"},{"location":"changelog/#100-cases_fixture-pytest-2x-support","title":"1.0.0 - @cases_fixture + pytest 2.x support","text":"Pytest 2.x is now supported. Fixes #14.
New feature: @cases_fixture ! Now you can put your cases data retrieval in a fixture so that its duration does not enter into the test duration. This is particularly interesting if you use pytest-harvest to create benchmarks: you probably do not want the case data retrieval/parsing to be counted in the test duration, especially if you use caching on the case function to accelerate subsequent retrievals. Fixes #15.
"},{"location":"changelog/#0101-minor-encoding-issue-in-setuppy","title":"0.10.1 - minor encoding issue in setup.py","text":""},{"location":"changelog/#0100-support-for-python-2","title":"0.10.0 - support for python 2","text":"Python 2 is now supported. Fixed #3.
- Note:
CaseData, Given, ExpectedNormal, ExpectedError, and MultipleStepsCaseData type hints is not created in python 2 and python<3.5
"},{"location":"changelog/#091-pytest-steps-is-now-an-independent-project","title":"0.9.1 - pytest-steps is now an independent project","text":" - Light refactoring: some internal function names are now private, and there are now two submodules.
- pytest-steps is now an independent project. Examples in the documentation have been updated
- New documentation page: API reference
"},{"location":"changelog/#080-filtering-can-now-be-done-using-a-callable","title":"0.8.0 - Filtering can now be done using a callable.","text":" @cases_data: the filter argument now contains a filtering function. WARNING: the previous behaviour is still available but has been renamed has_tag. Fixes #8.
"},{"location":"changelog/#070-hardcoded-cases-selection-and-multi-module-selection","title":"0.7.0 - Hardcoded cases selection, and multi-module selection","text":" @cases_data has a new parameters cases that can be used to hardcode a case or a list of cases. Its module parameter can also now take a list of modules
"},{"location":"changelog/#060-case-parameters-and-better-test-suites","title":"0.6.0 - Case parameters and better test suites","text":" get_for is deprecated: it was too specific to a given case data format.MultipleStepsCaseData was fixed to also support multiple inputs.- Case functions can now have parameters (even case generators). This is particularly useful for test suites. Fixes #9.
"},{"location":"changelog/#050-support-for-test-suites","title":"0.5.0 - support for test suites","text":" - test functions can now be decorated with
@test_steps to easily define a test suite with several steps. This fixes #7.
"},{"location":"changelog/#040-support-for-data-caching-with-lru_cache","title":"0.4.0 - support for data caching with lru_cache","text":" - cases can now be decorated with
@lru_cache. @cases_generator also provides a lru_cache parameter to enable caching. Fixes #6.
"},{"location":"changelog/#030-case-generators","title":"0.3.0 - case generators","text":" - New decorator
@cases_generator to define case generators. Fixes #1. - Also, removed unused functions
is_expected_error_instance and assert_exception_equal
"},{"location":"changelog/#020-this_module-constant-taggingfiltering-doc","title":"0.2.0 - THIS_MODULE constant + Tagging/Filtering + doc","text":" -
New constant THIS_MODULE so that cases and test functions can coexist in the same file. This fixes #5.
-
Added @test_target and @case_tags decorators for case functions, and added filter parameter in @cases_data. This allows users to :
- tag a case function with any item (and in particular with the reference to the function it relates to),
- and to filter the case functions used by a test function according to a particular tag.
This fixes #4.
"},{"location":"changelog/#010-first-public-version","title":"0.1.0 - First public version","text":" - Initial fork from private repo
"},{"location":"examples/","title":"Examples","text":"Our examples are hosted in a separate repository so that they can be updated independently.
- data science benchmark demonstrates how
pytest can be used as a benchmarking engine thanks to pytest-cases and pytest-harvest, to compare the performances of several regression algorithms on several datasets and produce various reports (plots, csv table...).
"},{"location":"long_description/","title":"pytest-cases","text":"Separate test code from test cases in pytest.
The documentation for users is available here: https://smarie.github.io/python-pytest-cases/
A readme for developers is available here: https://github.com/smarie/python-pytest-cases
"},{"location":"pytest_goodies/","title":"pytest Goodies","text":"Many pytest features were missing to make pytest_cases work with such a \"no-boilerplate\" experience. Many of these can be of interest to the general pytest audience, so they are exposed in the public API.
"},{"location":"pytest_goodies/#fixture","title":"@fixture","text":"@fixture is similar to pytest.fixture but without its param and ids arguments. Instead, it is able to pick the parametrization from @pytest.mark.parametrize marks applied on fixtures. This makes it very intuitive for users to parametrize both their tests and fixtures. As a bonus, its name argument works even in old versions of pytest (which is not the case for fixture).
Finally it now supports unpacking, see unpacking feature.
@fixture deprecation if/when @pytest.fixture supports @pytest.mark.parametrize
The ability for pytest fixtures to support the @pytest.mark.parametrize annotation is a feature that clearly belongs to pytest scope, and has been requested already. It is therefore expected that @fixture will be deprecated in favor of @pytest_fixture if/when the pytest team decides to add the proposed feature. As always, deprecation will happen slowly across versions (at least two minor, or one major version update) so as for users to have the time to update their code bases.
"},{"location":"pytest_goodies/#unpack_fixture-unpack_into","title":"unpack_fixture / unpack_into","text":"In some cases fixtures return a tuple or a list of items. It is not easy to refer to a single of these items in a test or another fixture. With unpack_fixture you can easily do it:
import pytest\nfrom pytest_cases import unpack_fixture, fixture\n\n@fixture\n@pytest.mark.parametrize(\"o\", ['hello', 'world'])\ndef c(o):\n return o, o[0]\n\na, b = unpack_fixture(\"a,b\", c)\n\ndef test_function(a, b):\n assert a[0] == b\n
Note that you can also use the unpack_into= argument of @fixture to do the same thing:
import pytest\nfrom pytest_cases import fixture\n\n@fixture(unpack_into=\"a,b\")\n@pytest.mark.parametrize(\"o\", ['hello', 'world'])\ndef c(o):\n return o, o[0]\n\ndef test_function(a, b):\n assert a[0] == b\n
And it is also available in fixture_union:
import pytest\nfrom pytest_cases import fixture, fixture_union\n\n@fixture\n@pytest.mark.parametrize(\"o\", ['hello', 'world'])\ndef c(o):\n return o, o[0]\n\n@fixture\n@pytest.mark.parametrize(\"o\", ['yeepee', 'yay'])\ndef d(o):\n return o, o[0]\n\nfixture_union(\"c_or_d\", [c, d], unpack_into=\"a, b\")\n\ndef test_function(a, b):\n assert a[0] == b\n
"},{"location":"pytest_goodies/#param_fixtures","title":"param_fixture[s]","text":"If you wish to share some parameters across several fixtures and tests, it might be convenient to have a fixture representing this parameter. This is relatively easy for single parameters, but a bit harder for parameter tuples.
The two utilities functions param_fixture (for a single parameter name) and param_fixtures (for a tuple of parameter names) handle the difficulty for you:
import pytest\nfrom pytest_cases import param_fixtures, param_fixture\n\n# create a single parameter fixture\nmy_parameter = param_fixture(\"my_parameter\", [1, 2, 3, 4])\n\n@pytest.fixture\ndef fixture_uses_param(my_parameter):\n ...\n\ndef test_uses_param(my_parameter, fixture_uses_param):\n ...\n\n# -----\n# create a 2-tuple parameter fixture\narg1, arg2 = param_fixtures(\"arg1, arg2\", [(1, 2), (3, 4)])\n\n@pytest.fixture\ndef fixture_uses_param2(arg2):\n ...\n\ndef test_uses_param2(arg1, arg2, fixture_uses_param2):\n ...\n
You can mark any of the argvalues with pytest.mark to pass a custom id or a custom \"skip\" or \"fail\" mark, just as you do in pytest. See pytest documentation.
"},{"location":"pytest_goodies/#fixture_union","title":"fixture_union","text":"As of pytest 5, it is not possible to create a \"union\" fixture, i.e. a parametrized fixture that would first take all the possible values of fixture A, then all possible values of fixture B, etc. Indeed all fixture dependencies (a.k.a. \"closure\") of each test node are grouped together, and if they have parameters a big \"cross-product\" of the parameters is done by pytest.
The topic has been largely discussed in pytest-dev#349 and a request for proposal has been finally made.
fixture_union is an implementation of this proposal. It is also used by @parametrize to support fixture_ref in parameter values, see below. The theory is presented in more details in this page, while below are more practical examples.
from pytest_cases import fixture, fixture_union\n\n@fixture\ndef first():\n return 'hello'\n\n@fixture(params=['a', 'b'])\ndef second(request):\n return request.param\n\n# c will first take all the values of 'first', then all of 'second'\nc = fixture_union('c', [first, second])\n\ndef test_basic_union(c):\n print(c)\n
yields
<...>::test_basic_union[\\first] hello PASSED\n<...>::test_basic_union[\\second-a] a PASSED\n<...>::test_basic_union[\\second-b] b PASSED\n
"},{"location":"pytest_goodies/#idstyle","title":"idstyle","text":"As you can see the ids of union fixtures are slightly different from standard ids, so that you can easily understand what is going on. You can change this feature with \u00ecdstyle, see API documentation for details.
"},{"location":"pytest_goodies/#marks-and-ids","title":"marks and ids","text":"You can mark any of the alternatives with pytest.mark to pass a custom id or a custom \"skip\" or \"fail\" mark, just as you do in pytest. See pytest documentation.
"},{"location":"pytest_goodies/#unpacking","title":"unpacking","text":"Fixture unions also support unpacking with the unpack_into argument, see unpacking feature.
"},{"location":"pytest_goodies/#to-conclude","title":"to conclude","text":"Fixture unions are a major change in the internal pytest engine, as fixture closures (the ordered set of all fixtures required by a test node to run - directly or indirectly) now become trees where branches correspond to alternative paths taken in the \"unions\", and leafs are the alternative fixture closures. This feature has been tested in very complex cases (several union fixtures, fixtures that are not selected by a given union but that is requested by the test function, etc.). But if you find some strange behaviour don't hesitate to report it in the issues page !
IMPORTANT if you do not use @fixture but only @pytest.fixture, then you will see that your fixtures are called even when they are not used, with a parameter NOT_USED. This symbol is automatically ignored if you use @fixture, otherwise you have to handle it. Alternatively you can use @ignore_unused on your fixture function.
fixture unions vs. cases
If you're familiar with pytest-cases already, you might note that @cases_data is not so different than a fixture union: we do a union of all case functions. If one day union fixtures are directly supported by pytest, we will probably refactor this lib to align all the concepts.
"},{"location":"pytest_goodies/#parametrize","title":"@parametrize","text":"@parametrize is a replacement for @pytest.mark.parametrize with many additional features to make the most of parametrization. See API reference for details about all the new features. In particular it allows you to include references to fixtures and to value-generating functions in the parameter values.
- Simply use
fixture_ref(<fixture>) in the parameter values, where <fixture> can be the fixture name or fixture function. New: from version 3.2 on, if auto_refs=True (default), @parametrize will automatically detect fixture symbols in the list of argvalues, and will create fixture_refs automatically around them so that you don't need to. - if you do not wish to create a fixture, you can also use
lazy_value(<function>) - Note that when parametrizing several argnames, both
fixture_ref and lazy_value can be used as the tuple, or in the tuple. Several fixture_ref and/or lazy_value can be used in the same tuple, too. - By default the id associated with a
fixture_ref or a lazy_value is the name of the fixture or function. Custom ids can be passed with the id=<id> parameter.
For example, with a single argument:
import pytest\nfrom pytest_cases import parametrize, fixture, fixture_ref, lazy_value\n\n@pytest.fixture\ndef world_str():\n return 'world'\n\ndef whatfun():\n return 'what'\n\n@fixture\n@parametrize('who', [world_str, 'you'])\ndef greetings(who):\n return 'hello ' + who\n\n@parametrize('main_msg', ['nothing', \n fixture_ref(world_str),\n lazy_value(whatfun),\n \"1\",\n fixture_ref(greetings)], \n auto_refs=False)\n@pytest.mark.parametrize('ending', ['?', '!'])\ndef test_prints(main_msg, ending):\n print(main_msg + ending)\n
yields the following
> pytest -s -v\ncollected 12 items\ntest_prints[nothing-?] PASSED [ 8%]nothing?\ntest_prints[nothing-!] PASSED [ 16%]nothing!\ntest_prints[world_str-?] PASSED [ 25%]world?\ntest_prints[world_str-!] PASSED [ 33%]world!\ntest_prints[whatfun-?] PASSED [ 41%]what?\ntest_prints[whatfun-!] PASSED [ 50%]what!\ntest_prints[1-?] PASSED [ 58%]1?\ntest_prints[1-!] PASSED [ 66%]1!\ntest_prints[greetings-world_str-?] PASSED [ 75%]hello world?\ntest_prints[greetings-world_str-!] PASSED [ 83%]hello world!\ntest_prints[greetings-you-?] PASSED [ 91%]hello you?\ntest_prints[greetings-you-!] PASSED [100%]hello you!\n
"},{"location":"pytest_goodies/#ids-and-marks","title":"ids and marks","text":"You can also mark any of the argvalues with pytest.param to pass a custom id or a custom \"skip\" or \"fail\" mark, just as you do in pytest. See pytest documentation.
You can also pass a custom callable or generator in ids as in @pytest.mark.parametrize.
"},{"location":"pytest_goodies/#idstyle-customization","title":"idstyle customization","text":"As you can see in the example above, the default ids are similar to what you would intuitively expect, even when you use fixture_ref.
This is because by default idstyle=None, to preserve test ids very close to standard pytest by default. But still, a fixture_union is generated behind the scenes when there is a fixture reference. So this is actually non-standard. You may therefore prefer to see explicit ids showing the various fixture alternatives, as in fixture_union. For this simply set the idstyle to 'compact', 'explicit' or to a callable such as str.
For example, changing the previous example to add idstyle=\"explicit\":
(...same as above...)\n\n@parametrize('main_msg', ['nothing',\n fixture_ref(world_str),\n lazy_value(whatfun),\n \"1\",\n fixture_ref(greetings)], idstyle=\"explicit\")\n@pytest.mark.parametrize('ending', ['?', '!'])\ndef test_prints(main_msg, ending):\n print(main_msg + ending)\n
yields to
> pytest -s -v\ncollected 12 items\ntest_prints[main_msg\\nothing-?] PASSED [ 8%]nothing?\ntest_prints[main_msg\\nothing-!] PASSED [ 16%]nothing!\ntest_prints[main_msg\\world_str-?] PASSED [ 25%]world?\ntest_prints[main_msg\\world_str-!] PASSED [ 33%]world!\ntest_prints[main_msg\\P2:4-whatfun-?] PASSED [ 41%]what?\ntest_prints[main_msg\\P2:4-whatfun-!] PASSED [ 50%]what!\ntest_prints[main_msg\\P2:4-1-?] PASSED [ 58%]1?\ntest_prints[main_msg\\P2:4-1-!] PASSED [ 66%]1!\ntest_prints[main_msg\\greetings-world_str-?] PASSED [ 75%]hello world?\ntest_prints[main_msg\\greetings-world_str-!] PASSED [ 83%]hello world!\ntest_prints[main_msg\\greetings-you-?] PASSED [ 91%]hello you?\ntest_prints[main_msg\\greetings-you-!] PASSED [100%]hello you!\n
You can see that with this explicit style, the various \"alternatives\" in the fixture union generated behind the scenes for the main_msg parameter appear explicitly. In particular you see that there is an alternative main_msg\\P2:4 covering several parameters in a row.
Note that this idstyle is not taken into account if you only use lazy_values but no fixture_ref, as lazy_values do not require to create a fixture union behind the scenes.
"},{"location":"pytest_goodies/#parametrization-order","title":"parametrization order","text":"Another consequence of using fixture_ref is that the priority order of the parameters, relative to other standard pytest.mark.parametrize parameters that you would place on the same function, will get impacted. You may solve this by replacing your other @pytest.mark.parametrize calls with param_fixtures so that all the parameters are fixtures (see param_fixture).
"},{"location":"pytest_goodies/#passing-a-hook","title":"passing a hook","text":"As per version 1.14, all the above functions now support passing a hook argument. This argument should be a callable. It will be called every time a fixture is about to be created by pytest_cases on your behalf. The fixture function is passed as the argument of the hook, and the hook should return it as the result.
You can use this fixture to better understand which fixtures are created behind the scenes, and also to decorate the fixture functions before they are created. For example you can use hook=saved_fixture (from pytest-harvest) in order to save the created fixtures in the fixture store.
"},{"location":"pytest_goodies/#assert_exception","title":"assert_exception","text":"assert_exception context manager is an alternative to pytest.raises to check exceptions in your tests. You can either check type, instance equality, repr string pattern, or use custom validation functions. See API reference.
"},{"location":"pytest_goodies/#-with-reorder","title":"--with-reorder","text":"pytest postprocesses the order of the collected items in order to optimize setup/teardown of session, module and class fixtures. This optimization algorithm happens at the pytest_collection_modifyitems stage, and is still under improvement, as can be seen in pytest#3551, pytest#3393, #2846...
Besides other plugins such as pytest-reorder can modify the order as well.
This new commandline is a goodie to change the reordering:
-
--with-reorder normal is the default behaviour: it lets pytest and all the plugins execute their reordering in each of their pytest_collection_modifyitems hooks, and simply does not interact
-
--with-reorder skip allows you to restore the original order that was active before pytest_collection_modifyitems was initially called, thus not taking into account any reordering done by pytest or by any of its plugins.
"},{"location":"unions_theory/","title":"Theory behind fixture_union","text":""},{"location":"unions_theory/#1-how-pytest-works-today","title":"1. How pytest works today","text":"As of pytest 5, there are three kind of concepts at play to generate the list of test nodes and their received parameters (\"call spec\" in pytest internals).
-
test functions are the functions defined with def test_<name>(<args>).
-
they can be parametrized using @pytest.mark.parametrize (or our enhanced version @parametrize). That means that some of the <args> will take several values, and for each combination a distinct test node will be created
-
they can require fixtures, that is, functions decorated with @pytest.fixture (or our enhanced version @fixture). That means that some of the <args> will take the value of the corresponding fixture(s).
-
fixtures can be parametrized too (with @fixture it is easier :) ), and can require other fixtures.
-
finally fixtures can enable an \"auto-use\" mode, so that they are called even when not explicitly required by anything.
Therefore, a test plan can be represented as an acyclic directed graph of fixtures, where nodes are fixtures and edges represent dependencies. On top of this layout, we can overlay the information of which fixture nodes are parametrized, which ones are required by which test function, and which test function is parametrized. The resulting figure is presented below:
The following code can be used to easily check the number of tests run. Note that we use @fixture and @parametrize from pytest-cases to ease code readability here but you would get a similar behaviour with @pytest.fixture and @pytest.mark.parametrize (the test ids would not show the parameter names by default though, which is helpful for our demonstration here).
from pytest_cases import fixture, parametrize\n\n@fixture(autouse=True)\n@parametrize(ie=[-1, 1])\ndef e(ie):\n return \"e%s\" % ie\n\n@fixture\ndef d():\n return \"d\"\n\n@fixture\ndef c():\n return \"c\"\n\n@fixture\n@parametrize(ia=[0, 1])\ndef a(c, d, ia):\n return \"a%s\" % ia + c + d\n\n@parametrize(i2=['x', 'z'])\ndef test_2(a, i2):\n assert (a + i2) in (\"a0cdx\", \"a0cdz\", \"a1cdx\", \"a1cdz\")\n\n@fixture\n@parametrize(ib=['x', 'z'])\ndef b(a, c, ib):\n return \"b%s\" % ib + c + a\n\ndef test_1(a, b):\n assert a in (\"a0cd\", \"a1cd\")\n assert a == b[-4:]\n assert b[:-4] in (\"bxc\", \"bzc\")\n
calling pytest yields:
============================= test session starts =============================\ncollecting ... collected 16 items\n\ntest_doc_fixture_graph.py::test_2[ie=-1-ia=0-i2=x] \ntest_doc_fixture_graph.py::test_2[ie=-1-ia=0-i2=z] \ntest_doc_fixture_graph.py::test_2[ie=-1-ia=1-i2=x] \ntest_doc_fixture_graph.py::test_2[ie=-1-ia=1-i2=z] \ntest_doc_fixture_graph.py::test_2[ie=1-ia=0-i2=x] \ntest_doc_fixture_graph.py::test_2[ie=1-ia=0-i2=z] \ntest_doc_fixture_graph.py::test_2[ie=1-ia=1-i2=x] \ntest_doc_fixture_graph.py::test_2[ie=1-ia=1-i2=z] \ntest_doc_fixture_graph.py::test_1[ie=-1-ia=0-ib=x] \ntest_doc_fixture_graph.py::test_1[ie=-1-ia=0-ib=z] \ntest_doc_fixture_graph.py::test_1[ie=-1-ia=1-ib=x] \ntest_doc_fixture_graph.py::test_1[ie=-1-ia=1-ib=z] \ntest_doc_fixture_graph.py::test_1[ie=1-ia=0-ib=x] \ntest_doc_fixture_graph.py::test_1[ie=1-ia=0-ib=z] \ntest_doc_fixture_graph.py::test_1[ie=1-ia=1-ib=x] \ntest_doc_fixture_graph.py::test_1[ie=1-ia=1-ib=z] \n\n============================= 16 passed in 0.14s ==============================\n
So each test is called 8 times. How are these calls computed ?
- first for each test,
pytest computes the set of all fixtures that are directly or indirectly required to run it. This is known as the \"fixture closure\". So for test_1 this closure is {a, b, c, d, e} while for test 2 it is {a, c, d, e}. We can show this on the following picture:
- then a cartesian product is made across the parameters of all parametrization marks found on any item in the closure (including parameters of the test itself), So for
test_1 the cartesian product is <ie> x <ia> x <ib> while for test_2 it is <ie> x <ia> x <i2>. This is why both tests result in having 8 variants being called (see details in the test ids above).
"},{"location":"unions_theory/#2-extension-to-fixture-unions","title":"2. Extension to fixture unions.","text":"A fixture union is by definition a fixture that is parametrized to alternately depend on other fixtures. We will represent this in the figures with a special dashed orange arrow, to remind that a special parameter is associated with the selection of which arrow is activated.
Let's consider the following modification of the above example, where we introduce two \"unions\": one as an explicit fixture u, and the other implicitly created by using fixture_refs in the parametrization of b.
We can create such a configuration with a slight modification to the above example:
from pytest_cases import fixture, parametrize, fixture_ref, fixture_union\n\n(... same as above ...)\n\n@fixture\n@parametrize(ib=['x', 'z'])\n@parametrize(ub=(fixture_ref(a), fixture_ref(c)), idstyle=\"explicit\")\ndef b(ub, ib):\n return \"b%s\" % ib + ub\n\nu = fixture_union(\"u\", (a, b), idstyle=\"explicit\")\n\ndef test_1(u):\n pass\n
Note the idstyle=\"explicit\" keyword arguments, that will help us get more details in the test ids.
Calling pytest yields:
============================= test session starts =============================\ncollecting ... collected 24 items\n\ntest_doc_fixture_graph_union.py::test_2[ie=-1-ia=0-i2=x] PASSED [ 4%]\ntest_doc_fixture_graph_union.py::test_2[ie=-1-ia=0-i2=z] PASSED [ 8%]\ntest_doc_fixture_graph_union.py::test_2[ie=-1-ia=1-i2=x] PASSED [ 12%]\ntest_doc_fixture_graph_union.py::test_2[ie=-1-ia=1-i2=z] PASSED [ 16%]\ntest_doc_fixture_graph_union.py::test_2[ie=1-ia=0-i2=x] PASSED [ 20%]\ntest_doc_fixture_graph_union.py::test_2[ie=1-ia=0-i2=z] PASSED [ 25%]\ntest_doc_fixture_graph_union.py::test_2[ie=1-ia=1-i2=x] PASSED [ 29%]\ntest_doc_fixture_graph_union.py::test_2[ie=1-ia=1-i2=z] PASSED [ 33%]\ntest_doc_fixture_graph_union.py::test_1[ie=-1-u\\a-ia=0] PASSED [ 37%]\ntest_doc_fixture_graph_union.py::test_1[ie=-1-u\\a-ia=1] PASSED [ 41%]\ntest_doc_fixture_graph_union.py::test_1[ie=-1-u\\b-ib=x-ub\\a-ia=0] PASSED [ 45%]\ntest_doc_fixture_graph_union.py::test_1[ie=-1-u\\b-ib=x-ub\\a-ia=1] PASSED [ 50%]\ntest_doc_fixture_graph_union.py::test_1[ie=-1-u\\b-ib=x-ub\\c] PASSED [ 54%]\ntest_doc_fixture_graph_union.py::test_1[ie=-1-u\\b-ib=z-ub\\a-ia=0] PASSED [ 58%]\ntest_doc_fixture_graph_union.py::test_1[ie=-1-u\\b-ib=z-ub\\a-ia=1] PASSED [ 62%]\ntest_doc_fixture_graph_union.py::test_1[ie=-1-u\\b-ib=z-ub\\c] PASSED [ 66%]\ntest_doc_fixture_graph_union.py::test_1[ie=1-u\\a-ia=0] PASSED [ 70%]\ntest_doc_fixture_graph_union.py::test_1[ie=1-u\\a-ia=1] PASSED [ 75%]\ntest_doc_fixture_graph_union.py::test_1[ie=1-u\\b-ib=x-ub\\a-ia=0] PASSED [ 79%]\ntest_doc_fixture_graph_union.py::test_1[ie=1-u\\b-ib=x-ub\\a-ia=1] PASSED [ 83%]\ntest_doc_fixture_graph_union.py::test_1[ie=1-u\\b-ib=x-ub\\c] PASSED [ 87%]\ntest_doc_fixture_graph_union.py::test_1[ie=1-u\\b-ib=z-ub\\a-ia=0] PASSED [ 91%]\ntest_doc_fixture_graph_union.py::test_1[ie=1-u\\b-ib=z-ub\\a-ia=1] PASSED [ 95%]\ntest_doc_fixture_graph_union.py::test_1[ie=1-u\\b-ib=z-ub\\c] PASSED [100%]\n\n======================== 24 passed, 1 warning in 0.30s ========================\n
Now 24 tests were created ! test_2 still has 8 runs, which is normal as it does not depend on any union fixture. Let's try to understand what happened to parametrization of test_1. It is actually fairly simple:
-
first a global fixture closure is created as usual, consisting in {u, a, b, c, d, e}
-
then for each union fixture in test_1's closure, starting from the bottom of the graph, we generate several closures by activating each of the arrows in turn. We progress upwards through the graph of remaining dependencies for each alternative:
So the result consists in 3 alternate fixture closures for test_1:
-
finally, as usual, for each closure a cartesian product is made across the parameters of all parametrization marks found on any item in the closure (including parameters of the test itself), So
- for
test_1 alternative u\\a, the cartesian product is <ie> x <ia> (4 tests) - for
test_1 alternative u\\b-ub\\a, the cartesian product is <ie> x <ia> x <ib> (8 tests) - for
test_1 alternative u\\b-ub\\c, the cartesian product is <ie> x <ib> (4 tests) - for
test_2 it is <ie> x <ia> x <i2>. (8 tests).
The total is indeed 4 + 8 + 4 + 8 = 24 tests. Once again the test ids may be used to check that everything is correct, see above.
"}]}
\ No newline at end of file
+{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"pytest-cases","text":"Separate test code from test cases in pytest.
Slides from the pytest-cases presentation at EuroPython 2021 are now available here.
New current_cases fixture to easily know the current case for each parameter ! See below for details.
Did you ever think that most of your test functions were actually the same test code, but with different data inputs and expected results/exceptions ?
-
pytest-cases leverages pytest and its great @pytest.mark.parametrize decorator, so that you can separate your test cases from your test functions.
-
In addition, pytest-cases provides several useful goodies to empower pytest. In particular it improves the fixture mechanism to support \"fixture unions\". This is a major change in the internal pytest engine, unlocking many possibilities such as using fixture references as parameter values in a test function. See here.
pytest-cases is fully compliant with pytest-harvest so you can easily monitor the execution times and created artifacts. With it, it becomes very easy to create a complete data science benchmark, for example comparing various models on various datasets as illustrated below (from the example section):
"},{"location":"#citing","title":"Citing","text":"If pytest-cases helps you with your research work, don't hesitate to spread the word ! For this simply use this Zenodo link to get the proper citation entry (at the bottom right of the page, many formats available including BibTeX).
"},{"location":"#installing","title":"Installing","text":"> pip install pytest-cases\n
Note: Installing pytest-cases has effects on the order of pytest tests execution, even if you do not use its features. One positive side effect is that it fixed pytest#5054. But if you see less desirable ordering please report it.
"},{"location":"#why-pytest-cases","title":"Why pytest-cases ?","text":"pytest philosophy
Let's consider the following foo function under test, located in example.py:
def foo(a, b):\n return a + 1, b + 1\n
If we were using plain pytest to test it with various inputs, we would create a test_foo.py file and use @pytest.mark.parametrize:
import pytest\nfrom example import foo\n\n@pytest.mark.parametrize(\"a,b\", [(1, 2), (-1, -2)])\ndef test_foo(a, b):\n # check that foo runs correctly and that the result is a tuple. \n assert isinstance(foo(a, b), tuple)\n
This is the fastest and most compact thing to do when you have a few number of test cases, that do not require code to generate each test case.
pytest current limitations
Now imagine that instead of (1, 2) and (-1, -2) each of our test cases
- requires a few lines of code to be generated. For example artificial data creation using
numpy and/or pandas:
import numpy as np\nimport pandas as pd\n\n# case 1: non-sorted uniformly sampled timeseries with 2 holes\ncase1 = pd.DataFrame({\"datetime\": pd.date_range(start='20/1/1', periods=20, \n freq='-1d', tz='UTC'),\n \"data1\": np.arange(0, 20),\n \"data2\": np.arange(1, 21),\n \"data3\": np.arange(1, 21)})\ncase1.drop([3, 12], inplace=True)\n
-
requires documentation to explain the other developers the intent of that precise test case
-
requires external resources (data files on the filesystem, databases...), with a variable number of cases depending on what is available on the resource - but of course not all the cases would come from the same resource, that would be too easy :).
-
requires a readable id, such as 'uniformly_sampled_nonsorted_with_holes' for the above example. Of course we could use pytest.param or ids=<list> but that is \"a pain to maintain\" according to pytest doc (I agree!). Such a design does not feel right as the id is detached from the case.
With standard pytest there is no particular pattern to simplify your life here. Investigating a little bit, people usually end up trying to mix parameters and fixtures and asking this kind of question: so1, so2. But by design it is not possible to solve this problem using fixtures, because pytest does not handle \"unions\" of fixtures.
So all in all, the final answer is \"you have to do this yourself\", and have pytest use your handcrafted list of parameters as the list of argvalues in @pytest.mark.parametrize. Typically we would end up creating a get_all_foo_test_cases function, independently from pytest:
@pytest.mark.parametrize(\"a,b\", get_all_foo_test_cases())\ndef test_foo(a, b):\n ...\n
There is also an example in pytest doc with a metafunc hook.
The issue with such workarounds is that you can do anything. And anything is a bit too much: this does not provide any convention / \"good practice\" on how to organize test cases, which is an open door to developing ad-hoc unreadable or unmaintainable solutions.
pytest_cases was created to provide an answer to this precise situation. It proposes a simple framework to separate test cases from test functions. The test cases are typically located in a separate \"companion\" file:
test_foo.py is your usual test file containing the test functions (named test_<id>),test_foo_cases.py contains the test cases, that are also functions (named case_<id> or even <prefix>_<id> if you prefer). Note: an alternate file naming style cases_foo.py is also available if you prefer it.
Test cases can also be provided explicitly, for example in a class container:
And many more as we'll see below.
"},{"location":"#basic-usage","title":"Basic usage","text":""},{"location":"#a-case-functions","title":"a- Case functions","text":"Let's create a test_foo_cases.py file. This file will contain test cases generator functions, that we will call case functions for brevity. In these functions, you will typically either parse some test data files, generate some simulated test data, expected results, etc.
def case_two_positive_ints():\n \"\"\" Inputs are two positive integers \"\"\"\n return 1, 2\n\ndef case_two_negative_ints():\n \"\"\" Inputs are two negative integers \"\"\"\n return -1, -2\n
Case functions do not have any particular requirement, apart from the default name convention case_<id> - but even that can be customized: you can use distinct prefixes to denote distinct kind of parameters, such as data_<id>, user_<id>, model_<id>...
Case functions can return anything that is considered useful to run the associated test. We will see below that you can use all classic pytest mechanism on case functions (id customization, skip/fail marks, parametrization, fixtures injection).
"},{"location":"#b-test-functions","title":"b- Test functions","text":"As usual we write our pytest test functions starting with test_, in a test_foo.py file. The only difference is that we now decorate it with @parametrize_with_cases instead of @pytest.mark.parametrize as we were doing previously:
from example import foo\nfrom pytest_cases import parametrize_with_cases\n\n@parametrize_with_cases(\"a,b\")\ndef test_foo(a, b):\n # check that foo runs correctly and that the result is a tuple. \n assert isinstance(foo(a, b), tuple)\n
As simple as that ! The syntax is basically the same than in pytest.mark.parametrize.
Executing pytest will now run our test function once for every case function:
>>> pytest -s -v\n============================= test session starts =============================\n(...)\n<your_project>/tests/test_foo.py::test_foo[two_positive_ints] PASSED [ 50%]\n<your_project>/tests/test_foo.py::test_foo[two_negative_ints] PASSED [ 100%]\n\n========================== 2 passed in 0.24 seconds ==========================\n
"},{"location":"#tools-for-daily-use","title":"Tools for daily use","text":""},{"location":"#a-cases-collection","title":"a- Cases collection","text":""},{"location":"#alternate-sources","title":"Alternate source(s)","text":"It is not mandatory that case functions should be in a different file than the test functions: both can be in the same file. For this you can use cases='.' or cases=THIS_MODULE to refer to the module in which the test function is located:
from pytest_cases import parametrize_with_cases\n\ndef case_one_positive_int():\n return 1\n\ndef case_one_negative_int():\n return -1\n\n@parametrize_with_cases(\"i\", cases='.')\ndef test_with_this_module(i):\n assert i == int(i)\n
However WARNING: only the case functions defined BEFORE the test function in the module file will be taken into account!
@parametrize_with_cases(cases=...) also accepts explicit list of case functions, classes containing case functions, and modules. See API Reference for details. A typical way to organize cases is to use classes for example:
from pytest_cases import parametrize_with_cases\n\nclass Foo:\n def case_a_positive_int(self):\n return 1\n\n def case_another_positive_int(self):\n return 2\n\n@parametrize_with_cases(\"a\", cases=Foo)\ndef test_foo(a):\n assert a > 0\n
Note that as for pytest, self is recreated for every test and therefore should not be used to store any useful information.
"},{"location":"#alternate-prefix","title":"Alternate prefix","text":"case_ might not be your preferred prefix, especially if you wish to store in the same module or class various kind of case data. @parametrize_with_cases offers a prefix=... argument to select an alternate prefix for your case functions. That way, you can store in the same module or class case functions as diverse as datasets (e.g. data_), user descriptions (e.g. user_), algorithms or machine learning models (e.g. model_ or algo_), etc.
from pytest_cases import parametrize_with_cases, parametrize\n\ndef data_a():\n return 'a'\n\n@parametrize(\"hello\", [True, False])\ndef data_b(hello):\n return \"hello\" if hello else \"world\"\n\ndef case_c():\n return dict(name=\"hi i'm not used\")\n\ndef user_bob():\n return \"bob\"\n\n@parametrize_with_cases(\"data\", cases='.', prefix=\"data_\")\n@parametrize_with_cases(\"user\", cases='.', prefix=\"user_\")\ndef test_with_data(data, user):\n assert data in ('a', \"hello\", \"world\")\n assert user == 'bob'\n
yields
test_doc_filters_n_tags.py::test_with_data[bob-a] PASSED [ 33%]\ntest_doc_filters_n_tags.py::test_with_data[bob-b-True] PASSED [ 66%]\ntest_doc_filters_n_tags.py::test_with_data[bob-b-False] PASSED [ 100%]\n
"},{"location":"#filters-and-tags","title":"Filters and tags","text":"The easiest way to select only a subset of case functions in a module or a class, is to specify a custom prefix instead of the default one ('case_'), as shown above.
However sometimes more advanced filtering is required. In that case, you can also rely on three additional mechanisms provided in @parametrize_with_cases:
- the
glob argument can contain a glob-like pattern for case ids. This can become handy to separate for example good or bad cases, the latter returning an expected error type and/or message for use with pytest.raises or with our alternative assert_exception.
from math import sqrt\nimport pytest\nfrom pytest_cases import parametrize_with_cases\n\n\ndef case_int_success():\n return 1\n\ndef case_negative_int_failure():\n # note that we decide to return the expected type of failure to check it\n return -1, ValueError, \"math domain error\"\n\n\n@parametrize_with_cases(\"data\", cases='.', glob=\"*success\")\ndef test_good_datasets(data):\n assert sqrt(data) > 0\n\n@parametrize_with_cases(\"data, err_type, err_msg\", cases='.', glob=\"*failure\")\ndef test_bad_datasets(data, err_type, err_msg):\n with pytest.raises(err_type, match=err_msg):\n sqrt(data)\n
- the
has_tag argument allows you to filter cases based on tags set on case functions using the @case decorator. See API reference of @case and @parametrize_with_cases.
from pytest_cases import parametrize_with_cases, case\n\nclass FooCases:\n def case_two_positive_ints(self):\n return 1, 2\n\n @case(tags='foo')\n def case_one_positive_int(self):\n return 1\n\n@parametrize_with_cases(\"a\", cases=FooCases, has_tag='foo')\ndef test_foo(a):\n assert a > 0\n
- Finally if none of the above matches your expectations, you can provide a callable to
filter. This callable will receive each collected case function and should return True (or a truth-value convertible object) in case of success. Note that your function can leverage the get_case_id, get_case_marks, get_case_tags etc. helper functions to read the tags, marks and id found on it. See API doc.
from pytest_cases import get_case_id\n\n@parametrize_with_cases(\"data\", cases='.', \n filter=lambda cf: \"success\" in get_case_id(cf))\ndef test_good_datasets2(data):\n ...\n
- An array of default filters is available in the
filters module: has_tag, id_has_prefix, etc. You can use logical operations on them such as \"and\" (&) \"or\" (|) and \"not\" (~) in order to create your own custom filters. See API reference for details.
from pytest_cases import filters as ft\n\n\n@parametrize_with_cases(\"data\", cases='.',\n filter=ft.has_tag(\"success\") & ft.id_has_prefix(\"case_b\")\ndef test_good_datasets3(data):\n ...\n
"},{"location":"#b-case-functions","title":"b- Case functions","text":""},{"location":"#custom-case-name","title":"Custom case name","text":"The id used by pytest for a given case is automatically taken from the case function name by removing the case_ (or other custom) prefix. It can instead be customized explicitly by decorating your case function with the @case(id=<id>) decorator. See API reference.
from pytest_cases import case\n\n@case(id=\"2 positive integers\")\ndef case_two_positive_ints():\n return 1, 2\n
"},{"location":"#pytest-marks-skip-xfail","title":"Pytest marks (skip, xfail...)","text":"pytest marks such as @pytest.mark.skipif can be applied on case functions the same way as with test functions.
import sys\nimport pytest\n\n@pytest.mark.skipif(sys.version_info < (3, 0), reason=\"Not useful on python 2\")\ndef case_two_positive_ints():\n return 1, 2\n
"},{"location":"#case-generators","title":"Case generators","text":"In many real-world usage we want to generate one test case per <something>. The most intuitive way would be to use a for loop to create the case functions, and to use the @case decorator to set their names ; however this would not be very readable.
Instead, case functions can be parametrized the same way as with test functions: simply add the parameter names as arguments in their signature and decorate with @pytest.mark.parametrize. Even better, you can use the enhanced @parametrize from pytest-cases so as to benefit from its additional usability features (see API reference):
from pytest_cases import parametrize, parametrize_with_cases\n\nclass CasesFoo:\n def case_hello(self):\n return \"hello world\"\n\n @parametrize(who=('you', 'there'))\n def case_simple_generator(self, who):\n return \"hello %s\" % who\n\n\n@parametrize_with_cases(\"msg\", cases=CasesFoo)\ndef test_foo(msg):\n assert isinstance(msg, str) and msg.startswith(\"hello\")\n
Yields
test_generators.py::test_foo[hello] PASSED [ 33%]\ntest_generators.py::test_foo[simple_generator-who=you] PASSED [ 66%]\ntest_generators.py::test_foo[simple_generator-who=there] PASSED [100%]\n
"},{"location":"#cases-requiring-fixtures","title":"Cases requiring fixtures","text":"Cases can use fixtures the same way as test functions do: simply add the fixture names as arguments in their signature and make sure the fixture exists or is imported either in the module where @parametrize_with_cases is used, or in a conftest.py file in one of the parent packages.
See pytest documentation on sharing fixturesand this blog.
You can use the experimental @parametrize_with_cases(import_fixtures=True) argument to perform the import automatically for you, see API reference.
Use @fixture instead of @pytest.fixture
If a fixture is used by some of your cases only, then you should use the @fixture decorator from pytest-cases instead of the standard @pytest.fixture. Otherwise you fixture will be setup/teardown for all cases even those not requiring it. See @fixture doc.
from pytest_cases import parametrize_with_cases, fixture, parametrize\n\n@fixture(scope='session')\ndef db():\n return {0: 'louise', 1: 'bob'}\n\ndef user_bob(db):\n return db[1]\n\n@parametrize(id=range(2))\ndef user_from_db(db, id):\n return db[id]\n\n@parametrize_with_cases(\"a\", cases='.', prefix='user_')\ndef test_users(a, db, request):\n print(\"this is test %r\" % request.node.nodeid)\n assert a in db.values()\n
yields
test_fixtures.py::test_users[a_is_bob] \ntest_fixtures.py::test_users[a_is_from_db-id=0] \ntest_fixtures.py::test_users[a_is_from_db-id=1] \n
"},{"location":"#advanced-topics","title":"Advanced topics","text":""},{"location":"#a-scope-of-cases","title":"a- Scope of cases","text":"By default a case function is transformed into a lazy parameter using lazy_value. This is not a fixture, but simply a new parametrize mechanism that allows parameters to be provided by functions (See @parametrize).
However, as soon as a case function is either parametrized or requires a fixture, then it is automatically transformed into a fixture so that pytest can handle it properly. In that situation, the fixture needs to have a scope. By default this scope is \"function\". You can change it using the scope argument in @parametrize_with_cases.
"},{"location":"#b-parametrizing-fixtures","title":"b- Parametrizing fixtures","text":"In some scenarii you might wish to parametrize a fixture with the cases, rather than the test function. For example
-
to inject the same test cases in several test functions without duplicating the @parametrize_with_cases decorator on each of them,
-
to generate the test cases once for the whole session, using a scope='session' fixture or another scope,
-
to modify the test cases, log some message, or perform some other action before injecting them into the test functions, and/or after executing the test function (thanks to yield fixtures)
-
...
For this, simply use @fixture from pytest_cases instead of @pytest.fixture to define your fixture. That allows your fixtures to be easily parametrized with @parametrize_with_cases, @parametrize, and even @pytest.mark.parametrize.
from pytest_cases import fixture, parametrize_with_cases\n\n@fixture\n@parametrize_with_cases(\"a,b\")\ndef c(a, b):\n return a + b\n\ndef test_foo(c):\n assert isinstance(c, int)\n
"},{"location":"#c-caching-cases","title":"c- Caching cases","text":"After starting to reuse cases in several test functions, you might end-up thinking \"why do I have to spend the data parsing/generation time several times ? It is the same case.\".
pytest-cases follows the same philosophy than pytest: each test node should be independent. Therefore case functions are called for each test case. This ensures that mutable objects can not leak across tests, for example.
That being said, if you are certain that your tests do not modify your cases data, there are several ways to solve this issue:
- the easiest way is to use fixtures with a broad scope, as explained above. However in some parametrization scenarii,
pytest does not guarantee that the fixture will be setup only once for the whole session, even if it is a session-scoped fixture. Also the cases will be parsed every time you run pytest, which might be cumbersome
from pytest_cases import parametrize, parametrize_with_cases, fixture\n\n\n@parametrize(a=range(2))\ndef case_dummy(a):\n # this is read only once per a, while there are 4 test runs \n return a\n\n@fixture(scope='session')\n@parametrize_with_cases(\"a\", cases='.')\ndef cached_a(a):\n return a\n\n\n@parametrize(d=range(2))\ndef test_caching(cached_a, d):\n assert d < 2\n assert 0 <= cached_a <= 1\n
-
an alternative is to use functools.lru_cache to explicitly set a memory cache on a case function. For simple cases you could simply decorate your case function with @lru_cache(maxsize=1) since simple case functions do not have arguments. However for case generators this is a bit more tricky to size the cache - the easiest thing is probably to let it to its default size of 128 with the no-argument version @lru_cache, or to remove the max limit and let it auto-grow, with @lru_cache(max_size=None). See lru_cache documentation for details. Note that an older version of pytest-cases was offering some facilities to set the cache size, this has been removed from the library in version 2.0.0 as it seemed to provide little added value.
-
finally, you might wish to persist some cases on disk in order for example to avoid downloading them again from their original source, and/or to avoid costly processing on every pytest session. For this, the perfect match for you is to use joblib's excellent Memory cache.
If you add a cache mechanism, make sure that your test functions do not modify the returned objects !
"},{"location":"#d-accessing-the-current-case","title":"d- Accessing the current case","text":"In some scenarii you may wish to access the case functions that are currently used to provide the parameter values. This may be
- to make your test behave differently depending on the case function, case id or case tags
- to
pytest.skip some combinations of parameters/cases that do not make sense - ...
With pytest-cases starting in version 3.5, this is now possible thanks to the current_cases fixture. Simply use this fixture to get a dictionary containing the actual parameter id and case function for all parameters parametrized with cases in the current test node. Parametrized fixtures, if any, will appear in a sub-dictionary indexed by the fixture name.
from pytest_cases import parametrize, parametrize_with_cases, fixture\n\n@parametrize(nb=(1,))\ndef case_a():\n return nb\n\n@fixture\n@parametrize_with_cases(\"foo\", cases=case_a)\ndef my_fixture(foo):\n return foo\n\n@parametrize_with_cases(\"data\", cases=case_a)\ndef test_get_current_case(data, my_fixture, current_cases):\n\n # access the case details for a test parameter\n data_id, data_fun, data_params = current_cases[\"data\"]\n\n # access the case details for a fixture parameter\n my_fixture_id, my_fixture_fun, my_fixture_params = current_cases[\"my_fixture\"][\"foo\"]\n\n # let's print all case information for this test node\n print(current_cases)\n
yields
{'data': Case(id='a', func=<function case_a at 0x000001C0CAE9E700>, params={'nb': 1}), \n 'my_fixture': {\n 'foo': Case(id='a', func=<function case_a at 0x000001C0CAE9E700>, params={'nb': 1})\n }}\n
As you can see above, details are provided as namedtuples. When a case itself is parametrized, its current parameter value(s) appear too (in the above example, case_a is parametrized with nb). This can be used to skip a test conditionally, for example:
if data_fun is case_a and data_params['nb'] == 1:\n pytest.skip(\"This specific case is skipped\")\n
It can also be used to insert a debug breakpoint for a specific case.
To get more information on the case function, you can use get_case_marks(func), get_case_tags(func). You can also use matches_tag_query(...) to check if a case function matches some expectations either concerning its id or its tags. See API reference.
Note: you can get the same information from a pytest hook, using the get_current_cases function. See API reference for details.
"},{"location":"#e-test-ids","title":"e- Test ids","text":"Starting from version 3.0.0, test ids induced by @parametrize_with_cases are similar to the ids induced by @pytest.mark.parametrize, even if a case function is itself parametrized or requires a fixture. In some situations you may wish to get a better control on the test ids.
For this you can pass a callable to @parametrize_with_cases(ids=<callable>). In this callable, you may wish to use get_case_id, get_case_marks, get_case_tags, or matches_tag_query to return a custom id.
from pytest_cases import parametrize, parametrize_with_cases, case, get_case_id\n\ndef case_hello():\n return \"hello !\"\n\n@case(id=\"hello_world\")\ndef case_basic2():\n return \"hello, world !\"\n\n@case(id=\"hello_name\")\n@parametrize(\"name\", [\"you\", \"earthling\"])\ndef case_basic3(name):\n return \"hello, %s !\" % name\n\ndef myidgen(case_fun):\n \"\"\"Custom test case id\"\"\"\n return \"#%s#\" % get_case_id(case_fun)\n\n@parametrize_with_cases(\"msg\", cases=\".\", ids=myidgen)\ndef test_foo(msg):\n print(msg)\n
pytest -s -v yields:
============================= test session starts =============================\nplatform win32 -- Python 3.7.3, pytest-5.3.5, py-1.9.0, pluggy-0.13.1\ncachedir: .pytest_cache\n(...)\n\ntest_doc_ids_debug.py::test_foo[#hello#] \ntest_doc_ids_debug.py::test_foo[#hello_world#] \ntest_doc_ids_debug.py::test_foo[#hello_name#-you] \ntest_doc_ids_debug.py::test_foo[#hello_name#-earthling] \n\n============================== 4 passed in 0.07s ==============================\n
"},{"location":"#f-debugging","title":"f- Debugging","text":"When all of your case functions are simple, @parametrize_with_cases generates a @parametrize decorator with argvalues being a list of lazy_value(<case_func>) for all of them. This in turn falls back to a good old @pytest.mark.parametrize, so the behaviour is close to what you are used to see when using pytest.
However when at least one case function is complex, typically when it requires a fixture, then @parametrize_with_cases wraps it into a fixture and passes a fixture_ref(<fixture_wrapping_case>) to @parametrize. This creates a so-called \"fixture union\": tests are not any more a cross-product of parameters, but a tree. This new feature brought by pytest-cases is not present in pytest by default, and adds a layer of complexity. But good news: starting in pytest-cases 3.0.0, this complexity is entirely hidden. In other words, generated test ids do not differ between this mode, and the simple mode. For debugging purposes however, you might wish to make this visible by setting idstyle:
from pytest_cases import parametrize, parametrize_with_cases, case, fixture\n\ndef case_hello():\n return \"hello !\"\n\n@fixture\n@parametrize(\"_name\", [\"you\", \"earthling\"])\ndef name(_name):\n return _name\n\n@case(id=\"hello_fixture\")\ndef case_basic3(name):\n return \"hello, %s !\" % name\n\n@parametrize_with_cases(\"msg\", cases=\".\", idstyle=\"nostyle\")\ndef test_default_idstyle(msg):\n print(msg)\n\n@parametrize_with_cases(\"msg\", cases=\".\", idstyle=\"compact\")\ndef test_compact_idstyle(msg):\n print(msg)\n\n@parametrize_with_cases(\"msg\", cases=\".\", idstyle=\"explicit\")\ndef test_explicit_idstyle(msg):\n print(msg)\n
pytest -s -v yields
test_doc_debug.py::test_default_idstyle[hello] \ntest_doc_debug.py::test_default_idstyle[hello_fixture-you] \ntest_doc_debug.py::test_default_idstyle[hello_fixture-earthling] \ntest_doc_debug.py::test_compact_idstyle[\\hello] \ntest_doc_debug.py::test_compact_idstyle[\\hello_fixture-you] \ntest_doc_debug.py::test_compact_idstyle[\\hello_fixture-earthling] \ntest_doc_debug.py::test_explicit_idstyle[msg\\hello] \ntest_doc_debug.py::test_explicit_idstyle[msg\\hello_fixture-you] \ntest_doc_debug.py::test_explicit_idstyle[msg\\hello_fixture-earthling] \n
See also @parametrize documentation for details.
"},{"location":"#main-features-benefits","title":"Main features / benefits","text":" -
Separation of concerns: test code on one hand, test cases data on the other hand. This is particularly relevant for data science projects where a lot of test datasets are used on the same block of test code.
-
Everything in the test case or in the fixture, not outside. A side-effect of @pytest.mark.parametrize is that users tend to create or parse their datasets outside of the test function. pytest_cases suggests a model where the potentially time and memory consuming step of case data generation/retrieval is performed inside the test node or the required fixture, thus keeping every test case run more independent. It is also easy to put debug breakpoints on specific test cases.
-
User experience fully aligned with pytest. Cases collection and filtering, cases parametrization, cases output unpacking as test arguments, cases using fixtures... all of this will look very familiar to pytest users.
"},{"location":"#see-also","title":"See Also","text":""},{"location":"#pytest","title":"pytest","text":" - pytest documentation on parametrize
- pytest documentation on fixtures
- pytest-steps
- pytest-harvest
- pytest-patterns for examples showing how to combine the various plugins to create data science benchmarks.
"},{"location":"#others","title":"Others","text":" - makefun is used to dynamically generate functions or modify user-provided function signatures.
- decopatch is used to create decorators.
Do you like this library ? You might also like my other python libraries
"},{"location":"#want-to-contribute","title":"Want to contribute ?","text":"Details on the github page: https://github.com/smarie/python-pytest-cases
"},{"location":"api_reference/","title":"API reference","text":"In general, using help(symbol) is the recommended way to get the latest documentation. In addition, this page provides an overview of the various elements in this package.
"},{"location":"api_reference/#1-fixtures","title":"1 - Fixtures","text":""},{"location":"api_reference/#current_cases","title":"current_cases","text":"A fixture containing get_current_cases(request).
This is a dictionary containing all case parameters for the currently active pytest item. For each test function argument parametrized using a @parametrize_with_case(<argname>, ...) this dictionary contains an entry {<argname>: (case_id, case_function, case_params)}. If several argnames are parametrized this way, a dedicated entry will be present for each argname. The tuple is a namedtuple containing
- `id` a string containing the actual case id constructed by `@parametrize_with_cases`.\n - `function` the original case function.\n - `params` a dictionary, containing the parameters of the case, if itself is parametrized. Note that if the\ncase is parametrized with `@parametrize_with_cases`, the associated parameter value in the dictionary will also be\n`(case_id, case_function, case_params)`.\n
If a fixture parametrized with cases is active, the dictionary will contain an entry {<fixturename>: <dct>} where <dct> is a dictionary {<argname>: (case_id, case_function, case_params)}.
To get more information on a case function, you can use get_case_marks(f), get_case_tags(f). You can also use matches_tag_query to check if a case function matches some expectations either concerning its id or its tags. See filters and tags documentation.
"},{"location":"api_reference/#2-case-functions","title":"2 - Case functions","text":"As explained in the documentation, case functions have no requirement anymore, and starting from version 2.0.0 of pytest_cases they can be parametrized with the usual @pytest.mark.parametrize or its improvement @parametrize. Therefore the only remaining decorator is the optional @case decorator:
"},{"location":"api_reference/#case","title":"@case","text":"@case(id=None, # type: str # noqa\n tags=None, # type: Union[Any, Iterable[Any]]\n marks=(), # type: Union[MarkDecorator, Iterable[MarkDecorator]]\n )\n
Optional decorator for case functions so as to customize some information.
@case(id='hey')\ndef case_hi():\n return 1\n
Parameters:
-
id: the custom pytest id that should be used when this case is active. Replaces the deprecated @case_name decorator from v1. If no id is provided, the id is generated from case functions by removing their prefix, see @parametrize_with_cases(prefix='case_').
-
tags: custom tags to be used for filtering in @parametrize_with_cases(has_tags). Replaces the deprecated @case_tags and @target decorators.
-
marks: optional pytest marks to add on the case. Note that decorating the function directly with the mark also works, and if marks are provided in both places they are merged.
"},{"location":"api_reference/#copy_case_info","title":"copy_case_info","text":"def copy_case_info(from_fun, # type: Callable\n to_fun # type: Callable\n ):\n
Copies all information from case function from_fun to to_fun.
"},{"location":"api_reference/#set_case_id","title":"set_case_id","text":"def set_case_id(id, # type: str\n case_func # type: Callable\n ):\n
Sets an explicit id on case function case_func.
"},{"location":"api_reference/#get_case_id","title":"get_case_id","text":"def get_case_id(case_func, # type: Callable\n prefix_for_default_ids='case_' # type: str\n ):\n
Return the case id associated with this case function.
If a custom id is not present, a case id is automatically created from the function name based on removing the provided prefix if present at the beginning of the function name. If the resulting case id is empty, \"\" will be returned.
Parameters:
"},{"location":"api_reference/#get_case_marks","title":"get_case_marks","text":"def get_case_marks(case_func, # type: Callable\n concatenate_with_fun_marks=False, # type: bool\n as_decorators=False # type: bool\n ):\n
Return the marks that are on the case function.
There are currently two ways to place a mark on a case function: either with @pytest.mark.<name> or in @case(marks=...). This function returns a list of marks containing either both (if concatenate_with_fun_marks is True) or only the ones set with @case (concatenate_with_fun_marks is False, default).
Parameters:
-
case_func: the case function
-
concatenate_with_fun_marks: if False (default) only the marks declared in @case will be returned. Otherwise a concatenation of marks in @case and on the function (for example directly with @pytest.mark.<mk>) will be returned.
-
as_decorators: when True, the marks (MarkInfo) will be transformed into MarkDecorators before being returned. Otherwise (default) the marks are returned as is.
"},{"location":"api_reference/#get_case_tags","title":"get_case_tags","text":"def get_case_tags(case_func # type: Callable\n ):\n
Return the tags on this case function or an empty tuple.
Parameters:
case_func: the case function
"},{"location":"api_reference/#matches_tag_query","title":"matches_tag_query","text":"def matches_tag_query(case_fun, # type: Callable\n has_tag=None, # type: Union[str, Iterable[str]]\n filter=None, # type: Union[Callable[[Callable], bool], Iterable[Callable[[Callable], bool]]] # noqa\n ):\n
This function is the one used by @parametrize_with_cases to filter the case functions collected. It can be used manually for tests/debug.
Returns True if the case function is selected by the query:
-
if has_tag contains one or several tags, they should ALL be present in the tags set on case_fun (get_case_tags)
-
if filter contains one or several filter callables, they are all called in sequence and the case_fun is only selected if ALL of them return a True truth value
Parameters:
-
case_fun: the case function
-
has_tag: one or several tags that should ALL be present in the tags set on case_fun for it to be selected.
-
filter: one or several filter callables that will be called in sequence. If all of them return a True truth value, case_fun is selected.
"},{"location":"api_reference/#is_case_class","title":"is_case_class","text":"def is_case_class(cls, # type: Any\n case_marker_in_name='Case', # type: str\n check_name=True # type: bool\n ):\n
This function is the one used by @parametrize_with_cases to collect cases within classes. It can be used manually for tests/debug.
Returns True if the given object is a class and, if check_name=True (default), if its name contains case_marker_in_name.
Parameters:
-
cls: the object to check
-
case_marker_in_name: the string that should be present in a class name so that it is selected. Default is 'Case'.
-
check_name: a boolean (default True) to enforce that the name contains the word case_marker_in_name. If False, any class will lead to a True result whatever its name.
"},{"location":"api_reference/#is_case_function","title":"is_case_function","text":"def is_case_function(f, # type: Any\n prefix='case_', # type: str\n check_prefix=True # type: bool\n ):\n
This function is the one used by @parametrize_with_cases to collect cases. It can be used manually for tests/debug.
Returns True if the provided object is a function or callable and, if check_prefix=True (default), if it starts with prefix.
Parameters:
-
f: the object to check
-
prefix: the string that should be present at the beginning of a function name so that it is selected. Default is 'case_'.
-
check_prefix: if this boolean is True (default), the prefix will be checked. If False, any function will lead to a True result whatever its name.
"},{"location":"api_reference/#the-filters-submodule","title":"The filters submodule","text":"This submodule contains symbols to help you create filters for @parametrize_with_cases(filter=...).
All helper filters in this submodule return an instance of CaseFilter, so that you can combine them easily with \"and\" (&) \"or\" (|) and \"invert\" (~) in order to create new custom filters.
"},{"location":"api_reference/#has_tag","title":"has_tag","text":"def has_tag(tag_name: str)\n
Selects cases that have the tag tag_name. See @case(tags=...) to add tags to a case.
"},{"location":"api_reference/#has_tags","title":"has_tags","text":"def has_tags(*tag_names: str)\n
Selects cases that have all tags tag_names. See @case(tags=...) to add tags to a case.
"},{"location":"api_reference/#id_has_prefix","title":"id_has_prefix","text":"def id_has_prefix(prefix: str)\n
Selects cases that have a case id prefix prefix. Note that this is not the prefix of the whole case function name, but the case id, possibly overridden with @case(id=)
"},{"location":"api_reference/#id_has_suffix","title":"id_has_suffix","text":"def id_has_suffix(suffix: str)\n
Selects cases that have a case id suffix suffix. Note that this is not the suffix of the whole case function name, but the case id, possibly overridden with @case(id=)
"},{"location":"api_reference/#id_match_regex","title":"id_match_regex","text":"def id_match_regex(regex: str)\n
Selects cases that have a case id matching regex pattern regex. Note that this is not a match of the whole case function name, but the case id, possibly overridden with @case(id=)
"},{"location":"api_reference/#casefilter","title":"CaseFilter","text":"CaseFilter(filter_function: Callable)\n
CaseFilter is the class used by all filters above, and implementing logical operations \"and\" (&) \"or\" (|) and \"not\" (~). You can use it to define a composable filter from any callable receiving a single case argument and returning a boolean indicating if the case is selected.
"},{"location":"api_reference/#3-cases-collection","title":"3 - Cases collection","text":""},{"location":"api_reference/#parametrize_with_cases","title":"@parametrize_with_cases","text":"CaseType = Union[Callable, Type, ModuleRef]\n\n@parametrize_with_cases(argnames: str,\n cases: Union[CaseType, List[CaseType]] = AUTO,\n prefix: str = 'case_',\n glob: str = None,\n has_tag: Union[str, Iterable[str]] = None,\n filter: Callable = None,\n ids: Union[Callable, Iterable[str]] = None,\n idstyle: Union[str, Callable] = None,\n scope: str = \"function\",\n import_fixtures: bool = False\n )\n
A decorator for test functions or fixtures, to parametrize them based on test cases. It works similarly to @pytest.mark.parametrize: argnames represent a coma-separated string of arguments to inject in the decorated test function or fixture. The argument values (argvalues in @pytest.mark.parametrize) are collected from the various case functions found according to cases, and injected as lazy values so that the case functions are called just before the test or fixture is executed.
By default (cases=AUTO) the list of test cases is automatically drawn from the python module file named test_<name>_cases.py or if not found, cases_<name>.py, where test_<name> is the current module name.
Finally, the cases argument also accepts an explicit case function, cases-containing class, module or module name; or a list containing any mix of these elements. Note that both absolute and relative module names are supported.
Note that @parametrize_with_cases collection and parameter creation steps are strictly equivalent to get_all_cases + get_parametrize_args. This can be handy for debugging purposes.
# Collect all cases\ncases_funs = get_all_cases(f, cases=cases, prefix=prefix,\n glob=glob, has_tag=has_tag, filter=filter)\n\n# Transform the various functions found\nargvalues = get_parametrize_args(host_class_or_module_of_f, cases_funs)\n
Parameters
-
argnames: same than in @pytest.mark.parametrize
-
cases: a case function, a class containing cases, a module object or a module name string (relative module names accepted). Or a list of such items. You may use THIS_MODULE or '.' to include current module. AUTO (default) means that the module named test_<name>_cases.py or if not found, cases_<name>.py, will be loaded, where test_<name>.py is the module file of the decorated function. When a module is listed, all of its functions matching the prefix, filter and has_tag are selected, including those functions nested in classes following naming pattern *Case*. Nested subclasses are taken into account, as long as they follow the *Case* naming pattern. When classes are explicitly provided in the list, they can have any name and do not need to follow this *Case* pattern.
-
prefix: the prefix for case functions. Default is 'case_' but you might wish to use different prefixes to denote different kind of cases, for example 'data_', 'algo_', 'user_', etc.
-
glob: a matching pattern for case ids, for example *_success or *_failure. The only special character that can be used for now in this pattern is *, it can not be escaped, and it can be used several times in the same expression. The pattern should match the entire case id for the case to be selected. Note that this is applied on the case id, and therefore if it is customized through @case(id=...) it will be taken into account.
-
has_tag: a single tag or a tuple, set, list of tags that should be matched by the ones set with the @case decorator on the case function(s) to be selected.
-
filter: a callable receiving the case function and returning True or a truth value in case the function needs to be selected.
-
ids: optional custom ids, similar to the one in pytest.mark.parametrize. Users may either provide an iterable of string ids, or a callable. If a callable is provided it will receive the case functions. Users may wish to use get_case_id or other helpers in the API to inspect the case functions.
-
idstyle: This is mostly for debug. Style of ids to be used in the \"union\" fixtures generated by @parametrize if some cases are transformed into fixtures behind the scenes. idstyle possible values are 'compact', 'explicit' or None/'nostyle' (default), or a callable. idstyle has no effect if no cases are transformed into fixtures. As opposed to ids, a callable provided here will receive a ParamAlternative object indicating which generated fixture should be used. See @parametrize for details.
-
scope: The scope of the union fixture to create if fixture_refs are found in the argvalues
-
import_fixtures: experimental feature. Turn this to True in order to automatically import all fixtures defined in the cases module into the current module.
"},{"location":"api_reference/#get_current_cases","title":"get_current_cases","text":"def get_current_cases(request_or_item):\n
Returns a dictionary containing all case parameters for the currently active pytest item. You can either pass the pytest item (available in some hooks) or the request (available in hooks, and also directly as a fixture).
For each test function argument parametrized using a @parametrize_with_case(<argname>, ...) this dictionary contains an entry {<argname>: (case_id, case_function, case_params)}. If several argnames are parametrized this way, a dedicated entry will be present for each argname. The tuple is a namedtuple containing
- `id` a string containing the actual case id constructed by `@parametrize_with_cases`.\n - `function` the original case function.\n - `params` a dictionary, containing the parameters of the case, if itself is parametrized. Note that if the\ncase is parametrized with `@parametrize_with_cases`, the associated parameter value in the dictionary will also be\n`(case_id, case_function, case_params)`.\n
If a fixture parametrized with cases is active, the dictionary will contain an entry {<fixturename>: <dct>} where <dct> is a dictionary {<argname>: (case_id, case_function, case_params)}.
To get more information on a case function, you can use get_case_id(f), get_case_marks(f), get_case_tags(f). You can also use matches_tag_query to check if a case function matches some expectations either concerning its id or its tags. See filters and tags documentation.
Note that you can get the same contents directly by using the current_cases fixture.
"},{"location":"api_reference/#get_all_cases","title":"get_all_cases","text":"CaseType = Union[Callable, Type, ModuleRef]\n\ndef get_all_cases(parametrization_target: Callable,\n cases: Union[CaseType, List[CaseType]] = None,\n prefix: str = 'case_',\n glob: str = None,\n has_tag: Union[str, Iterable[str]] = None,\n filter: Callable[[Callable], bool] = None\n ) -> List[Callable]:\n
Collect all cases as used with @parametrize_with_cases. See @parametrize_with_cases for more details on the parameters. This can be used to lists all desired cases for a given parametrization_target (a test function or a fixture) which may be convenient for debugging purposes. # Get the cases for f that are defined in the current file\ncases = get_all_cases(f, cases=\".\")\n\n# Get the cases from cases_xyz.py or test_xyz_cases.py\nimport test.test_xyz\nxyz_cases = get_all_cases(test.test_xyz)\n\n# Can be used to filter explicit cases, in which case no parametrization_target is needed\nfiltered_cases = get_all_cases(cases=[case_1, case_2, case_3], has_tag=[\"banana\"])\n
- If using a
cases argument that requires module information, such as \".\" AUTO or a relative module like \".xyz\", the value of parametrization_target will be used to to determine the context. If None or simply left empty, it will use the module from which get_all_cases was called. You can pass an explicit module object or a function, in which case the module in which it's defined will be used.
"},{"location":"api_reference/#get_parametrize_args","title":"get_parametrize_args","text":"def get_parametrize_args(host_class_or_module: Union[Type, ModuleType],\n cases_funs: List[Callable],\n debug: bool = False\n ) -> List[Union[lazy_value, fixture_ref]]:\n
Transforms a list of cases (obtained from get_all_cases) into a list of argvalues for @parametrize. Each case function case_fun is transformed into one or several lazy_value(s) or a fixture_ref:
-
If case_fun requires at least on fixture, a fixture will be created if not yet present, and a fixture_ref will be returned.
-
If case_fun is a parametrized case, one lazy_value with a partialized version will be created for each parameter combination.
-
Otherwise, case_fun represents a single case: in that case a single lazy_value is returned.
"},{"location":"api_reference/#4-pytest-goodies","title":"4 - Pytest goodies","text":""},{"location":"api_reference/#fixture","title":"@fixture","text":"@fixture(scope: str = \"function\", \n autouse: bool = False, \n name: str = None, \n unpack_into: Iterable[str] = None,\n hook: Callable = None,\n **kwargs)\n
Identical to @pytest.fixture decorator, except that
-
when used in a fixture union (either explicit fixture_union or indirect through @parametrize+fixture_ref or @parametrize_with_cases), it will not be setup/teardown unnecessarily in tests that do not require it.
-
it supports multi-parametrization with @pytest.mark.parametrize as requested in pytest#3960. As a consequence it does not support the params and ids arguments anymore.
-
it supports a new argument unpack_into where you can provide names for fixtures where to unpack this fixture into.
As a consequence it does not support the params and ids arguments anymore.
Parameters:
- scope: the scope for which this fixture is shared, one of \"function\" (default), \"class\", \"module\" or \"session\".
- autouse: if True, the fixture func is activated for all tests that can see it. If False (the default) then an explicitreference is needed to activate the fixture.
- name: the name of the fixture. This defaults to the name of the decorated function. Note: If a fixture is used in the same module in which it is defined, the function name of the fixture will be shadowed by the function arg that requests the fixture; one wayto resolve this is to name the decorated function
fixture_<fixturename> and then use @pytest.fixture(name='<fixturename>'). - unpack_into: an optional iterable of names, or string containing coma-separated names, for additional fixtures to create to represent parts of this fixture. See
unpack_fixture for details. - hook: an optional hook to apply to each fixture function that is created during this call. The hook function will be called every time a fixture is about to be created. It will receive a single argument (the function implementing the fixture) and should return the function to use. For example you can use
saved_fixture from pytest-harvest as a hook in order to save all such created fixtures in the fixture store. - kwargs: other keyword arguments for
@pytest.fixture
"},{"location":"api_reference/#unpack_fixture","title":"unpack_fixture","text":"def unpack_fixture(argnames: str,\n fixture: Union[str, Callable],\n in_cls: bool = False,\n hook: Callable = None\n ) -> Tuple[<Fixture>, ...]\n
Creates several fixtures with names argnames from the source fixture. Created fixtures will correspond to elements unpacked from fixture in order. For example if fixture is a tuple of length 2, argnames=\"a,b\" will create two fixtures containing the first and second element respectively.
The created fixtures are automatically registered into the callers' module, but you may wish to assign them to variables for convenience. In that case make sure that you use the same names, e.g. a, b = unpack_fixture('a,b', 'c').
import pytest\nfrom pytest_cases import unpack_fixture, fixture\n\n@fixture\n@pytest.mark.parametrize(\"o\", ['hello', 'world'])\ndef c(o):\n return o, o[0]\n\na, b = unpack_fixture(\"a,b\", c)\n\ndef test_function(a, b):\n assert a[0] == b\n
You can also use this function inside a class with in_cls=True. In that case you MUST assign the output of the function to variables, as the created fixtures won't be registered with the encompassing module.
import pytest\nfrom pytest_cases import unpack_fixture, fixture\n\n@fixture\n@pytest.mark.parametrize(\"o\", ['hello', 'world'])\ndef c(o):\n return o, o[0]\n\nclass TestClass:\n a, b = unpack_fixture(\"a,b\", c, in_cls=True)\n\n def test_function(self, a, b):\n assert a[0] == b\n
Parameters
- argnames: same as
@pytest.mark.parametrize argnames. - fixture: a fixture name string or a fixture symbol. If a fixture symbol is provided, the created fixtures will have the same scope. If a name is provided, they will have scope='function'. Note that in practice the performance loss resulting from using
function rather than a higher scope is negligible since the created fixtures' body is a one-liner. - in_cls: a boolean (default
False). You may wish to turn this to True to use this function inside a class. If you do so, you MUST assign the output to variables in the class. - hook: an optional hook to apply to each fixture function that is created during this call. The hook function will be called every time a fixture is about to be created. It will receive a single argument (the function implementing the fixture) and should return the function to use. For example you can use
saved_fixture from pytest-harvest as a hook in order to save all such created fixtures in the fixture store.
Outputs: the created fixtures.
"},{"location":"api_reference/#fixture_union","title":"fixture_union","text":"def fixture_union(name: str,\n fixtures: Iterable[Union[str, Callable]],\n scope: str = \"function\",\n idstyle: Optional[str] = 'compact',\n ids: Union[Callable, Iterable[str]] = None,\n unpack_into: Iterable[str] = None,\n autouse: bool = False,\n hook: Callable = None,\n **kwargs) -> <Fixture>\n
Creates a fixture that will take all values of the provided fixtures in order. That fixture is automatically registered into the callers' module, but you may wish to assign it to a variable for convenience. In that case make sure that you use the same name, e.g. a = fixture_union('a', ['b', 'c'])
The style of test ids corresponding to the union alternatives can be changed with idstyle. Three values are allowed:
'explicit' favors readability with names as <union>/<alternative>,'compact' (default) adds a small mark so that at least one sees which parameters are union alternatives and which others are normal parameters: /<alternative>None or 'nostyle' provides minimalistic ids : <alternative>
See UnionIdMakers class for details.
You can also pass a callable idstyle that will receive instances of UnionFixtureAlternative. For example str leads to very explicit ids: <union>/<idx>/<alternative>. See UnionFixtureAlternative class for details.
Parameters:
name: the name of the fixture to createfixtures: an array-like containing fixture names and/or fixture symbolsscope: the scope of the union. Since the union depends on the sub-fixtures, it should be smaller than the smallest scope of fixtures referenced.idstyle: The style of test ids corresponding to the union alternatives. One of 'explicit', 'compact','nostyle'/None, or a callable (e.g. str) that will receive instances of UnionFixtureAlternative.unpack_into: an optional iterable of names, or string containing coma-separated names, for additional fixtures to create to represent parts of this fixture. See unpack_fixture for details.ids: as in pytest. The default value returns the correct fixtureautouse: as in pytesthook: an optional hook to apply to each fixture function that is created during this call. The hook function will be called every time a fixture is about to be created. It will receive a single argument (the function implementing the fixture) and should return the function to use. For example you can use saved_fixture from pytest-harvest as a hook in order to save all such created fixtures in the fixture store.kwargs: other pytest fixture options. They might not be supported correctly.
Outputs: the new fixture. Note: you do not need to capture that output in a symbol, since the fixture is automatically registered in your module. However if you decide to do so make sure that you use the same name.
"},{"location":"api_reference/#param_fixtures","title":"param_fixtures","text":"def param_fixtures(argnames: str,\n argvalues: Iterable[Any],\n autouse: bool = False,\n ids: Union[Callable, Iterable[str]] = None, \n scope: str = \"function\",\n hook: Callable = None,\n debug: bool = False,\n **kwargs) -> Tuple[<Fixture>]\n
Creates one or several \"parameters\" fixtures - depending on the number or coma-separated names in argnames. The created fixtures are automatically registered into the callers' module, but you may wish to assign them to variables for convenience. In that case make sure that you use the same names, e.g. p, q = param_fixtures('p,q', [(0, 1), (2, 3)]).
Note that the (argnames, argvalues, ids) signature is similar to @pytest.mark.parametrize for consistency, see pytest doc on parametrize.
import pytest\nfrom pytest_cases import param_fixtures, param_fixture\n\n# create a 2-tuple parameter fixture\narg1, arg2 = param_fixtures(\"arg1, arg2\", [(1, 2), (3, 4)])\n\n@pytest.fixture\ndef fixture_uses_param2(arg2):\n ...\n\ndef test_uses_param2(arg1, arg2, fixture_uses_param2):\n ...\n
Parameters:
argnames: same as @pytest.mark.parametrize argnames.argvalues: same as @pytest.mark.parametrize argvalues.autouse: see fixture autouseids: same as @pytest.mark.parametrize idsscope: see fixture scope hook: an optional hook to apply to each fixture function that is created during this call. The hook function will be called every time a fixture is about to be created. It will receive a single argument (the function implementing the fixture) and should return the function to use. For example you can use saved_fixture from pytest-harvest as a hook in order to save all such created fixtures in the fixture store.kwargs: any other argument for the created 'fixtures'
"},{"location":"api_reference/#param_fixture","title":"param_fixture","text":"param_fixture(argname, argvalues, \n autouse=False, ids=None, hook=None, scope=\"function\", **kwargs)\n -> <Fixture>\n
Identical to param_fixtures but for a single parameter name, so that you can assign its output to a single variable.
"},{"location":"api_reference/#parametrize","title":"@parametrize","text":"def parametrize(argnames: str=None,\n argvalues: Iterable[Any]=None,\n indirect: bool = False,\n ids: Union[Callable, Iterable[str]] = None,\n idstyle: Union[str, Callable] = None,\n idgen: Union[str, Callable] = _IDGEN,\n auto_refs: bool = True,\n scope: str = None,\n hook: Callable = None,\n scope: str = \"function\",\n debug: bool = False,\n **args)\n
Equivalent to @pytest.mark.parametrize but also supports
New alternate style for argnames/argvalues. One can also use **args to pass additional {argnames: argvalues} in the same parametrization call. This can be handy in combination with idgen to master the whole id template associated with several parameters. Note that you can pass coma-separated argnames too, by de-referencing a dict: e.g. **{'a,b': [(0, True), (1, False)], 'c': [-1, 2]}.
New alternate style for ids. One can use idgen instead of ids. idgen can be a callable receiving all parameters at once (**args) and returning an id ; or it can be a string template using the new-style string formatting where the argnames can be used as variables (e.g. idgen=lambda **args: \"a={a}\".format(**args) or idgen=\"my_id where a={a}\"). The special idgen=AUTO symbol can be used to generate a default string template equivalent to lambda **args: \"-\".join(\"%s=%s\" % (n, v) for n, v in args.items()). This is enabled by default if you use the alternate style for argnames/argvalues (e.g. if len(args) > 0), and if there are no fixture_refs in your argvalues.
New possibilities in argvalues:
-
one can include references to fixtures with fixture_ref(<fixture>) where can be the fixture name or fixture function. When such a fixture reference is detected in the argvalues, a new function-scope \"union\" fixture will be created with a unique name, and the test function will be wrapped so as to be injected with the correct parameters from this fixture. Special test ids will be created to illustrate the switching between the various normal parameters and fixtures. You can see debug print messages about all fixtures created using debug=True. New: from version 3.2 on, if auto_refs=True (default), @parametrize will automatically detect fixture symbols in the list of argvalues, and will create fixture_refs automatically around them so that you don't need to.
-
one can include lazy argvalues with lazy_value(<valuegetter>, [id=..., marks=...]). A lazy_value is the same thing than a function-scoped fixture, except that the value getter function is not a fixture and therefore can neither be parametrized nor depend on fixtures. It should have no mandatory argument.
Both fixture_ref and lazy_value can be used to represent a single argvalue, or a whole tuple of argvalues when there are several argnames. Several of them can be used in a tuple.
Finally, pytest.param is supported even when there are fixture_ref and lazy_value.
Here as for all functions above, an optional hook can be passed, to apply on each fixture function that is created during this call. The hook function will be called every time a fixture is about to be created. It will receive a single argument (the function implementing the fixture) and should return the function to use. For example you can use saved_fixture from pytest-harvest as a hook in order to save all such created fixtures in the fixture store.
Parameters
-
argnames: same than in @pytest.mark.parametrize
-
argvalues: same as in pytest.mark.parametrize except thatfixture_refandlazy_value` are supported
-
indirect: same as in pytest.mark.parametrize. Note that it is not recommended and is not guaranteed to work in complex parametrization scenarii.
-
ids: same as in pytest.mark.parametrize. Note that an alternative way to create ids exists with idgen. Only one non-None ids or idgen should be provided.
-
idgen: an id formatter. Either a string representing a template, or a callable receiving all argvalues at once (as opposed to the behaviour in pytest ids). This alternative way to generate ids can only be used when ids is not provided (None). You can use the special pytest_cases.AUTO formatter to generate an automatic id with template <name>=<value>-<name2>=<value2>-.... AUTO is enabled by default if you use the alternate style for argnames/argvalues (e.g. if len(args) > 0), and if there are no fixture_refs in your argvalues.
-
idstyle: This is mostly for debug. Style of ids to be used in the \"union\" fixtures generated by @parametrize if at least one fixture_ref is found in the argvalues. idstyle possible values are 'compact', 'explicit' or None/'nostyle' (default), or a callable. idstyle has no effect if no fixture_ref are present in the argvalues. As opposed to ids, a callable provided here will receive a ParamAlternative object indicating which generated fixture should be used.
-
auto_refs: a boolean. If this is True (default), argvalues containing fixture symbols will automatically be wrapped into a fixture_ref, for convenience.
-
scope: The scope of the union fixture to create if fixture_refs are found in the argvalues. Otherwise same as in pytest.mark.parametrize.
-
hook: an optional hook to apply to each fixture function that is created during this call. The hook function will be called every time a fixture is about to be created. It will receive a single argument (the function implementing the fixture) and should return the function to use. For example you can use saved_fixture from pytest-harvest as a hook in order to save all such created fixtures in the fixture store.
-
debug: print debug messages on stdout to analyze fixture creation (use pytest -s to see them)
"},{"location":"api_reference/#lazy_value","title":"lazy_value","text":"
def lazy_value(valuegetter: Callable[[], Any],\n id: str = None,\n marks: Union[Any, Sequence[Any]] = ()\n ) -> LazyValue\n
A reference to a value getter (an argvalue-providing callable), to be used in @parametrize.
A lazy_value is the same thing than a function-scoped fixture, except that the value getter function is not a fixture and therefore can neither be parametrized nor depend on fixtures. It should have no mandatory argument. The underlying function will be called exactly once per test node.
By default the associated id is the name of the valuegetter callable, but a specific id can be provided otherwise. Note that this id does not take precedence over custom ids or idgen passed to @parametrize.
Note that a lazy_value can be included in a pytest.param without problem. In that case the id defined by pytest.param will take precedence over the one defined in lazy_value if any. The marks, however, will all be kept wherever they are defined.
Parameters
valuegetter: a callable without mandatory argumentsid: an optional id. Otherwise valuegetter.__name__ will be used by defaultmarks: optional marks. valuegetter marks will also be preserved.
"},{"location":"api_reference/#is_lazy","title":"is_lazy","text":"def is_lazy(argval) -> bool\n
Return True if argval is the outcome of processing a lazy_value through @parametrize. This encompasses parameters that are items of lazy tuples that are created when parametrizing several argnames with the same lazy_value().
"},{"location":"api_reference/#fixture_ref","title":"fixture_ref","text":"def fixture_ref(fixture: Union[str, Fixture]\n )\n
A reference to a fixture to be used with @parametrize. Create it with fixture_ref(<fixture>) where can be the fixture name or actual fixture function."},{"location":"changelog/","title":"Changelog","text":""},{"location":"changelog/#383-in-progress-tbd","title":"3.8.3 (in progress) - TBD","text":"
"},{"location":"changelog/#382-bugfixes-and-project-improvements","title":"3.8.2 - bugfixes and project improvements","text":" - Fixed issue with upcoming
pytest 8.1 release. PR #322 by @bluetech - Corrected API documentation (and comments) for the second file-name pattern for
AUTO-cases lookup (cases_<name>.py instead of case_<name>.py). PR #320 by @michele-riva. - Fixed
AssertionError on AUTO cases outside a 'normal' test module. Fixes #309. PR #320 by @michele-riva. - Improved error message in case of cases loading error in
@parametrize_with_cases when the cases argument is a string refering to a relative or absolute module name. Fixed import file mismatch with pytest 8 when executing our own tests. Fixes #323. - Fixed failing tests in our builds due to the
event_loop_policy fixture that appeared in pytest-asyncio 0.23. Fixes part of #321.
"},{"location":"changelog/#381-bugfixes","title":"3.8.1 - bugfixes","text":" - Fixed
ScopeMismatch with parametrized cases in non-trivial test trees. scope is now correctly handled for (i) fixture cases, and (ii) fixtures defined in conftest.py files at any depth. Fixes #311. PR #317 by @michele-riva.
"},{"location":"changelog/#380-async-generators-and-strict-markers","title":"3.8.0 - async, generators and strict-markers","text":" @fixture and @parametrize are now async and generator aware. Fixes #286. PR #301 by jgersti.- Fixed error with
pytest --strict-markers. Fixes #283. PR #300 by chrsmcgrr.
"},{"location":"changelog/#370-python-312","title":"3.7.0 - python 3.12","text":" - Added official support for Python 3.10, 3.11 and 3.12. Fixes #314
- Fixed
ModuleNotFoundError: distutils on Python 3.12 thanks to packaging. PR #312 by @jayqi. - Internal: switched to virtualenv backend.
"},{"location":"changelog/#3614-bugfixes","title":"3.6.14 - bugfixes","text":" - Fixed
AttributeError issue in is_case_function when an inspected symbol is a parametrized type hint without __name__. Fixes #287 - Fixed issue with
get_all_cases: default value for cases was wrong. Fixes #290
"},{"location":"changelog/#3613-bugfix","title":"3.6.13 - bugfix","text":" - Fixed issue where a lazy value (for example a case function) was not resolved before being injected in a parametrized function, and was therefore appearing as a
_LazyValueCaseParamValue. Fixed #274
"},{"location":"changelog/#3612-type-hint-fix-enhanced-compatibility-with-pytest-plugins","title":"3.6.12 - type hint fix + enhanced compatibility with pytest plugins","text":" - Improved compatibility with other
pytest plugins, in particular pytest-repeat, by supporting removal from fixture closure tree. Fixed #269. - Fixed type hint errors detected by
pyright. Fixed #270
"},{"location":"changelog/#3611-bugfix-for-pytest-xdist-and-get_all_cases-api-improvement","title":"3.6.11 - bugfix for pytest-xdist and get_all_cases API improvement","text":" get_all_cases can now be called without parametrization_target (defaulting to the caller module), and with an explicit module object. Fixed #258. PR #260 by @eddiebergman.- Fixed
AttributeError: module 'pytest_cases' has no attribute 'CasesCollectionWarning' when running pytest-xdist and at least one cases class is ignored because of __init__ or __new__. Fixed #249.
"},{"location":"changelog/#3610-bugfix-for-pytest-71","title":"3.6.10 - bugfix for pytest 7.1","text":" - Fixed
ImportError when using pytest 7.1. Fixed #264 and pytest-dev#9762.
"},{"location":"changelog/#369-bugfix-with-pytest-7","title":"3.6.9 - Bugfix with pytest 7","text":" - Fixed
FrozenInstanceError when using pytest 7.0.0. Fixed #251. PR#253 by jammer87
"},{"location":"changelog/#368-bugfix-support-for-multiprocessing-pool","title":"3.6.8 - Bugfix: support for multiprocessing Pool","text":" - Suppressed warnings in our own tests, to improve packaging maintenance. Fixed #248
- Fixed bug where setting
ids in @parametrize without setting explicitly idgen=None would raise a ValueError. Fixed #238. - Fixed bug where case-holding class marks were not propagated to static methods and class methods. Fixed #246
- Fixed support for multiprocessing
Pool. Fixes #242
"},{"location":"changelog/#367-minor-improvements-and-preparing-for-pytest-7","title":"3.6.7 - Minor improvements and preparing for pytest 7","text":" - Improved error message when a case function nested in a class has no
self argument and is not static. Fixes #243 - Added support for the new Scopes enum in pytest 7. Fixed #241
- Fixed
__version__ in development mode.
"},{"location":"changelog/#366-layout-change","title":"3.6.6 - Layout change","text":" - Restructured project so that tests are truly independent, to ease rpm/apt/etc. packaging. Fixed #220.
"},{"location":"changelog/#365-bugfix","title":"3.6.5 - Bugfix","text":" - Fixed an issue where using keyword
argnames in @pytest.mark.parametrize would cause IndexError: tuple index out of range in the tests collection phase. Fixed #234.
"},{"location":"changelog/#364-bugfix","title":"3.6.4 - Bugfix","text":" - A case id can now be a reserved keyword without triggering any
SyntaxError, even if the case is transformed into a fixture. Fixes #230
"},{"location":"changelog/#363-bugfix","title":"3.6.3 - Bugfix","text":" - Fixed an issue where a lazy value would not be resolved. This happens when the \"auto-simplify fixture\" happens in
@parametrize. Fixes #225
"},{"location":"changelog/#362-qualimetry","title":"3.6.2 - Qualimetry","text":" - Fixed most
flake8 errors and updated documentation to use genbadge. Fixes #223
"},{"location":"changelog/#361-bugfix-parametrizing-a-class-with-parametrize","title":"3.6.1 - bugfix - parametrizing a class with @parametrize","text":" - Fixed
ValueError when @parametrize is used to parametrize a class. Also, added a more explicit TypeError when @parametrize is used to parametrize a class and at least a fixture reference is present. Fixed #215.
"},{"location":"changelog/#360-unpack_fixtures-in-classes-current_cases-improvements","title":"3.6.0 - unpack_fixtures in classes + current_cases improvements","text":" -
The current_cases fixture now contains case parameters if any. Fixes #214
-
The current_cases fixture entries are now instances of namedtuple.
-
New in_cls argument in unpack_fixtures so that it can be used inside classes. Fixes #201
-
Fixed minor issue where empty entries could be present in currentcases. Fixes #213
"},{"location":"changelog/#352-bugfix-with-the-currentcases-fixture","title":"3.5.2 - bugfix with the currentcases fixture","text":" - Fixed issues where the
currentcases fixture would not return the correct case function. Fixed #212
"},{"location":"changelog/#351-python-310-compatibility-improved-error-message","title":"3.5.1 - python 3.10 compatibility + improved error message","text":" - Fixed error message related to misuse of
fixture_ref. Fixes #209 - Fixed import error with python 3.10. Fixes #207
"},{"location":"changelog/#350-new-current_cases-fixture-and-get_current_cases-function-fixes","title":"3.5.0 - New current_cases fixture and get_current_cases function + Fixes","text":" -
New: Users can now easily access the current cases for each parametrized argument thanks to the new current_cases fixture. A new helper function get_current_cases is also provided, for direct access from a hook. get_current_case_id becomes deprecated in favour of these two. Fixes #195
-
Bugfix: Fixed issue where the cache of a lazy_value used for a tuple of parameters (several argnames) was not considering the pytest context and thus was wrongly used across pytest nodes. Fixes #202
-
Improved error message when a fixture parametrized with several argnames as once is not able to unpack the parameter values received (non subscriptable object).
-
parametrize_plus and fixture_plus are now deprecated in favour of parametrize and fixture, as most users seem to have adopted these names without issues.
-
(internal) Replaced the \"used\" parameter with a dedicated singleton USED
"},{"location":"changelog/#346-increased-compatibility-with-other-plugins","title":"3.4.6 - Increased compatibility with other plugins","text":" LazyValue, LazyTuple and LazyTupleItem are now hashable. This increases compatibility with plugins hashing the parameter values, such as pytest-steps. See pytest-steps#41 . Fixes #199
"},{"location":"changelog/#345-bugfix","title":"3.4.5 - Bugfix","text":" - Fixed bug when a test module containing
@parametrize_with_cases was executed outside of pytest, typically through its __main__. Fixes #198
"},{"location":"changelog/#344-bugfix","title":"3.4.4 - Bugfix","text":" - Fixed issue when
@parametrize_with_cases was used on a fixture in a conftest.py. Fixes #196
"},{"location":"changelog/#343-technical-release-zenodo","title":"3.4.3 - Technical release - Zenodo","text":"Technical release to check that Zenodo metadata is now preserved. Same as 3.4.2, 3.4.1 and 3.4.0.
"},{"location":"changelog/#340-goodbye-v1-api-support-for-unbound-cases-bugfix-with-marks-fixtures-in-case-files","title":"3.4.0 - Goodbye v1 API + Support for unbound cases + Bugfix with marks + Fixtures in case files","text":" -
Legacy v1 API was dropped. Fixes #192
-
Unbound case functions in a class (e.g. Foo.bar) can now be directly passed to parametrize_with_cases without instantiating the class, e.g. parametrize_with_cases(cases=Foo.bar). Fixes #159
-
Fixed bug with concatenation of marks on cases. Fixes #191
-
Fixed an issue where a case transformed into a fixture, with the same name as the fixture it requires, would lead to a pytest fixture recursion.
-
Fixtures in case files can now be automatically imported using the experimental @parametrize_with_cases(import_fixtures=True). Fixes #193
"},{"location":"changelog/#330-filter-helpers-current-id-getter-support-for-pytest-asyncio-and-other-plugins","title":"3.3.0 - Filter helpers, Current id getter, Support for pytest-asyncio and other plugins","text":" -
Migrated to Github Actions + nox for CI/CD.
-
New helper function get_current_case_id to get the current case id for a given pytest request or item. Fixes #189
-
Extended the support of fixture closure modifications to remove and insert(0, f). This in particular solves an issue with pytest-asyncio. Fixes #176
-
New filters module providing helper functions has_tag, id_has_prefix, id_has_suffix, id_match_regex to easily create custom filters for use in @parametrize_with_cases(filter=...). PR #184 by @saroad2, (thanks !).
"},{"location":"changelog/#321-fixture-bugfix","title":"3.2.1 - @fixture Bugfix","text":" - Fixed
fixture 'self' not found issue when @fixture was used to decorate a class method not explicitly depending on request. Fixed #182
"},{"location":"changelog/#320-automatic-fixture_ref-test-ordering-bugfix","title":"3.2.0 - Automatic fixture_ref + test ordering bugfix","text":" -
New: from version 3.2 on, if auto_refs=True (default), @parametrize will automatically detect fixture symbols in the list of argvalues, and will create fixture_refs automatically around them so that you don't need to. Fixes #177
-
Fixed ordering issue happening on linux targets when several @parametrize are used to decorate the same function. Fixes #180
"},{"location":"changelog/#312-bugfixes-with-nesting-and-pytest-asyncio","title":"3.1.2 - Bugfixes with nesting and pytest-asyncio","text":" -
Now appending fixtures to the closure once it has been built is supported. This fixes an issue with pytest-asyncio. Fixes #176
-
Fixed issue when parametrize_with_cases was used on case functions themselves (nesting/recursion). This was due to a lack of support of the place_as magic pytest attribute. Fixes #179
-
Added a warning concerning usage of indirect in parametrize when fixture references are present. See #150
"},{"location":"changelog/#311-bugfix-with-ids","title":"3.1.1 - Bugfix with ids","text":" - Fixed issue with Empty id marker leaking to test ids. Fixed #171
"},{"location":"changelog/#310-improved-cases-collection","title":"3.1.0 - Improved cases collection","text":" -
@parametrize_with_cases now by default (cases=AUTO) looks for both file naming patterns test_<name>_cases.py and cases_<name>.py. Removed the AUTO2 constant. Fixed #140
-
Nested classes containing case functions are now officially supported (they were, but undocumented). Fixed #160
-
Case functions that are staticmethod and classmethod are now supported as well. Fixes #168
"},{"location":"changelog/#300-harmonization-of-ids-and-public-api-for-cases-info","title":"3.0.0 - harmonization of ids and public API for cases info","text":" -
Major refactoring of the way ids and marks are generated and customized in fixture_union, @parametrize and @parametrize_with_cases. Now idstyle has a consistent behaviour across the board, ids and idstyle can work together correctly, @parametrize_with_cases and @parametrize have much better default values for ids, and many others. See documentation for details. Fixed #154
-
New public API to manipulate information about a case function: copy_case_info, set_case_id, get_case_id, get_case_marks, get_case_tags, matches_tag_query, is_case_class, is_case_function. See API reference.
-
Fixed default behaviour of idgen in @parametrize: it only defaults to AUTO when no fixture_ref are used in the argvalues.
"},{"location":"changelog/#272-bugfix-with-doctest","title":"2.7.2 - Bugfix with doctest","text":" - Fixed
AttributeError: 'DoctestItem' object has no attribute '_request' when executing doctests. Fixes #156
"},{"location":"changelog/#271-pytestmarkusefixtures-can-be-used-on-case-functions","title":"2.7.1 - @pytest.mark.usefixtures can be used on case functions","text":" @pytest.mark.usefixtures can be now be used on case functions. Fixes #152.
"},{"location":"changelog/#270-parametrize_with_cases-now-supports-id-customization","title":"2.7.0 - @parametrize_with_cases now supports id customization","text":" @parametrize_with_cases now explicitly supports all id customization methods (ids, idgen and idstyle) supported by @parametrize (ids, idgen and idstyle). Updated documentation accordingly. Fixed #151
"},{"location":"changelog/#260-better-cache-for-lazy-values-and-support-for-infinite-id-generators","title":"2.6.0 - better cache for lazy values and support for infinite id generators","text":" -
lazy_value parameters are now cached by pytest node id only. So plugins can access the value without triggering an extra function call, but a new call is triggered for each pytest node, so as to prevent mutable object leakage across tests. Fixed #149 while ensuring no regression for #143.
-
The ids argument of parametrize now accepts a (possibly infinite) generator of ids, e.g. (f\"foo{i}\" for i in itertools.count()), just as pytest does. This was not always the case, inparticular when parametrizing a @fixture. The ids arguments of fixture_union, param_fixture[s], etc. now also support this pattern. Fixed #148
"},{"location":"changelog/#250-case-ids-glob-match-improvements","title":"2.5.0 - case ids glob match improvements","text":" - Improved description for the
glob argument in @parametrize_with_cases. Also made the implementation escape all regex special characters so that they can't be used. Finally a pattern should now match the entire case id (previously, a partial match would work if it was at the beginning of the string). One step towards #147
"},{"location":"changelog/#240-various-fixes-for-test-ids-and-lazy-values","title":"2.4.0 - various fixes for test ids and lazy values","text":" -
is_lazy is now part of public API, and _LazyValue now has a cache mechanism like _LazyTuple. Fixes #143
-
@parametrize: custom ids are now correctly taken into account when a single lazy_valueis used for a tuple of parameters. This issue could be seen also with @parametrize_with_cases: idgen does not seem to be taken into account when cases are unpacked into a tuple. Fixes #144.
-
Empty case ids are now replaced with '<empty_case_id>' to avoid ambiguous interpretation of test ids. Fixes #142.
"},{"location":"changelog/#230-better-lazyvalue-internal-api","title":"2.3.0 - better LazyValue internal API","text":" - new
clone(self, remove_int_base=False) API on LazyValue and LazyTupleItem instances. With this new API, on old pytest < 5.3, other plugins such as pytest-harvest can easily clone the contents from lazy values without having them inherit from int - which was a dirty hack used by pytest-cases to trick pytest to generate acceptable test ids in these old pytest versions. Also improved the LazyValue, LazyTuple and LazyTupleItem object model with equality and repr. Fixes pytest-harvest#43
"},{"location":"changelog/#225-marks-are-now-correctly-propagated-from-case-class","title":"2.2.5 - Marks are now correctly propagated from Case class","text":" - Marks set on a case class are now propagated to cases in the class. So you can use for example
pytest-pilot more easily ! Fixes #139
"},{"location":"changelog/#224-fixes-issue","title":"2.2.4 - Fixes issue","text":" - Fixed \"Created fixture names are not unique, please report\" error when duplicate fixture reference is provided in a pytest.param. Fixes #138.
"},{"location":"changelog/#223-fixed-issue-with-pytest-3x","title":"2.2.3 - Fixed issue with pytest 3.X","text":" - Fixed
TypeError: _idval() got an unexpected keyword argument 'item' with pytest versions between 3.0.0 and 3.7.4. Fixed #136
"},{"location":"changelog/#222-parametrize_with_cases-compatibility-improvements","title":"2.2.2 - @parametrize_with_cases compatibility improvements","text":" @parametrize_with_cases now supports that argnames is a list or tuple, just as @pytest.mark.parametrize does. PR #132, by @saroad2.
"},{"location":"changelog/#221-setuppy-fix-to-enforce-dependency-version","title":"2.2.1 - setup.py fix to enforce dependency version","text":" - Now enforcing usage of
makefun 1.9.3 or above to avoid issue AttributeError: 'functools.partial' object has no attribute '__module__' mentioned in #128
"},{"location":"changelog/#220-doc-improvements-bugfix-for-cases-requiring-fixtures","title":"2.2.0 - Doc improvements + bugfix for cases requiring fixtures","text":" -
Improved documentation to explain why @fixture should be used instead of @pytest.fixture. Fixed #125
-
Fixed ValueError: fixture is being applied more than once to the same function when two functions parametrized with the same cases were sitting in the same file. Improved robustness when cases require fixtures, in particular when parametrized test/fixture sits in a class or when several of them sit in a class/module. Fixed #126
"},{"location":"changelog/#213-missing-deprecation-warning","title":"2.1.3 - Missing deprecation warning","text":" -
Added missing deprecation warning on @cases_generator. Fixes #124.
-
Removed target and tags arguments of @cases_generator (deprecated api anyway) that were added by mistake in version 2.0.0 but never used.
"},{"location":"changelog/#212-compatibility-fix","title":"2.1.2 - Compatibility fix","text":" - Added support for pytest items without funcargs. Fixes interoperability with other pytest plugins such as
pytest-black or pytest-flake8. Fixes #122
"},{"location":"changelog/#211-fixed-issue-with-pytest-6","title":"2.1.1 - Fixed issue with pytest 6","text":"pytest 6 is now supported. Fixes #121
"},{"location":"changelog/#210-internal-engine-improvements-bugfixes","title":"2.1.0 - Internal engine improvements + bugfixes","text":"Fixed issue with @parametrize_with_cases when two cases with the same id and both requiring a fixture were to be created. Fixed #117.
Fixture closure engine refactoring:
-
When no fixture unions are present, the fixture closure is now identical to the default one in pytest, to avoid issues originating from other plugins fiddling with the closure. Fixes #116
-
New SuperClosure class representing the \"list\" facade on top of the fixture tree (instead of FixtureClosureNode). In addition, this list facade now better handles editing the order of fixtures when possible. Fixes #111.
-
Session and Module-scoped fixtures that are not used in all union alternatives are not any more torn town/setup across union alternatives. Fixes #120
"},{"location":"changelog/#204-bugfix","title":"2.0.4 - Bugfix","text":" - Fixed
TypeError with iterable argvalue in standard parametrize. Fixed #115.
"},{"location":"changelog/#203-bugfixes","title":"2.0.3 - Bugfixes","text":""},{"location":"changelog/#202-better-string-representation-for-lazy-values","title":"2.0.2 - Better string representation for lazy values","text":"Lazy values (so, test cases) now have a much nicer string representation ; in particular in pytest-harvest results tables. Fixes #112
"},{"location":"changelog/#201-better-test-ids-and-theory-page","title":"2.0.1 - Better test ids and theory page","text":""},{"location":"changelog/#200-less-boilerplate-full-pytest-alignment","title":"2.0.0 - Less boilerplate & full pytest alignment","text":"I am very pleased to announce this new version of pytest-cases, providing a lot of major improvements. Creating powerful and complex test suites have never been so easy and intuitive !
Below is a complete list of changes, but the user guide has also been updated accordingly so feel free to have a look to get a complete example-based walkthrough.
A/ More powerful and flexible cases collection
New @parametrize_with_cases decorator to replace @cases_data (deprecated).
-
Aligned with pytest:
-
now argnames can contain several names, and the case functions are automatically unpacked into it. You don't need to perform a case.get() in the test anymore !
@parametrize_with_cases(\"a,b\")\ndef test_foo(a, b):\n # use a and b directly !\n ...\n
-
cases are unpacked at test setup time, so the clock does not run while the case is created - in case you use pytest-harvest to collect the timings.
-
@parametrize_with_cases can be used on test functions as well as fixture functions (it was already the case in v1)
-
Easier to configure:
-
the decorator now has a single cases argument to indicate the cases, wherever they come from (no module argument anymore)
-
default (cases=AUTO) automatically looks for cases in the associated case module named test_xxx_cases.py. Users can easily switch to alternate pattern cases_xxx.py with cases=AUTO2. Fixes #91.
-
cases can sit inside a class, like what you're used to do with pytest. This additional style makes it much more convenient to organize cases and associated them with tests, when cases sit in the same file than the tests. Fixes #93.
-
an explicit sequence can be provided, it can mix all kind of sources: functions, classes, modules, and module names as strings (even relative ones!).
@parametrize_with_cases(\"a\", cases=(CasesClass, '.my_extra_cases'))\ndef test_foo(a):\n ...\n
-
More powerful API for filtering:
-
a new prefix argument (default case_) can be used to define case functions for various type of parameters: welcome user_<id>, data_<id>, algo_<id>, model_<id> ! Fixes #108
-
a new glob argument receiving a glob-like string can be used to further filter cases based on their names. For example you can distinguish *_success from *_failure case ids, so as to dispatch them to the appropriate positive or negative test. Fixes #108
-
finally you can still use has_tag and/or provide a filter callable, but now the callable will receive the case function, and this case function has a f._pytestcase attribute containing the id, tags and marks - it is therefore much easier to implement custom filtering.
B/ Easier-to-define case functions
-
Case functions can start with different prefixes to denote different kind of data: e.g. data_<id>, user_<id>, model_<id>, etc.
-
Case functions can now be parametrized with @parametrize or @pytest.mark.parametrize, just as in pytest ! This includes the ability to put pytest marks on the whole case, or on some specific parameter values using pytest.param. @cases_generator is therefore now deprecated but its alternate style for ids and arguments definition was preserved in @parametrize, see below.
-
Now case functions can require fixtures ! In that case they will be transformed into fixtures and injected as fixture_ref in the parametrization. Fixes #56.
-
New single optional @case(id=None, tags=(), marks=()) decorator to replace @case_name and @case_tags (deprecated): a single simple way to customize all aspects of a case function. Also, @test_target completely disappears from the picture as it was just a tag like others - this could be misleading.
C/ Misc / pytest goodies
-
New aliases for readability: @fixture for @fixture_plus, and@parametrize for @parametrize_plus (both aliases will coexist with the old names). Fixes #107.
-
@parametrize was improved in order to support the alternate parametrization mode that was previously offered by @cases_generator, see api reference. That way, users will be able to choose the style of their choice. Fixes #57 and #106.
-
@parametrize now raises an explicit error message when the user makes a mistake with the argnames. Fixes #105.
-
More readable error messages in @parametrize when lazy_value does not return the same number of argvalues than expected from the argnames.
-
Any error message associated to a lazy_value function call is not caught and hidden anymore but is emitted to the user, for easier debugging.
-
Fixed issue with lazy_value when a single mark is passed in the constructor.
-
lazy_value used as a tuple for several arguments now have a correct id generated even in old pytest version 2.
-
New pytest goodie assert_exception that can be used as a context manager. Fixes #104.
"},{"location":"changelog/#1170-lazy_value-improvements-annoying-warnings-suppression","title":"1.17.0 - lazy_value improvements + annoying warnings suppression","text":" -
lazy_value are now resolved at pytest setup stage, not pytest call stage. This is important for execution time recorded in the reports (see also pytest-harvest plugin). Fixes #102
-
A function used as a lazy_value can now be marked with pytest marks. Fixes #99
-
A lazy_value now has a nicer id when it is a partial. Fixes #97
-
Removed annoying PytestUnknownMarkWarning warning message when a mark was used on a case. Fixes #100
"},{"location":"changelog/#1160-new-lazy_value-for-parameters","title":"1.16.0 - New lazy_value for parameters","text":" - New marker
lazy_value for parametrize_plus. Fixes #92
"},{"location":"changelog/#1150-better-parametrize_plus-and-smaller-dependencies","title":"1.15.0 - better parametrize_plus and smaller dependencies","text":" -
Better support for pytest.param in parametrize_plus and also in fixture_union and fixture_param[s]. Improved corresponding ids. Fixed #79 and #86
-
New @ignore_unused decorator to protect a fixture function from the \"NOT_USED\" case happening when the fixture is used in a fixture union.
-
Removed six, wrapt and enum34 dependencies
-
(Internal) submodules reorganization for readability
- (Internal) suppressed a lot of code quality warnings
"},{"location":"changelog/#1140-bugfixes-and-hook-feature","title":"1.14.0 - bugfixes and hook feature","text":" -
Fixed ids precedence order when using pytest.mark.parametrize in a fixture_plus. Fixed #87
-
Fixed issue with fixture_union when using the same fixture twice in it. Fixes #85
-
Added the possibility to pass a hook function in all API where fixtures are created behind the scenes, so as to ease debugging and/or save fixtures (with stored_fixture from pytest harvest). Fixes #83
-
Fixture closures now support reordering when no unions are present. This suppressed the annoying warning \"WARNING the new order is not taken into account !!\" when it was not relevant. Fixes #81
"},{"location":"changelog/#1131-packaging-improvements","title":"1.13.1 - packaging improvements","text":" - packaging improvements: set the \"universal wheel\" flag to 1, and cleaned up the
setup.py. In particular removed dependency to six for setup and added py.typed file. Fixes #78
"},{"location":"changelog/#1130-cases_generator-default-names","title":"1.13.0 - @cases_generator default names","text":"@cases_generator now has a default value for the names template, based on the parameters. Fixes #77.
"},{"location":"changelog/#1124-bugfix","title":"1.12.4 - Bugfix","text":"Fixed ValueError when a product of unions was used on a test node, for example when two parametrize_plus using fixture_refs were used on the same fixture or test function. Fixed #76
"},{"location":"changelog/#1123-improved-error-messages","title":"1.12.3 - Improved error messages","text":"Improved error message when something that is not a fixture is used in unpack_fixture or fixture_union. Fixed #75
"},{"location":"changelog/#1122-warning-fix","title":"1.12.2 - Warning fix","text":"Fixed deprecation warning #74
"},{"location":"changelog/#1121-bugfixes","title":"1.12.1 - Bugfixes","text":" -
Now using module name and not file path to detect symbols in cases files that are imported from elsewhere and not created locally. Indeed that was causing problems on some ^platforms where a .pyc cache file is created. Fixes #72
-
Fixed PluginValidationError when pytest_fixture_plus or pytest_parametrize_plus were used in a conftest.py file. Fixes #71. According to discussion in pytest#6475, pytest_fixture_plus and pytest_parametrize_plus are now renamed to fixture_plus and parametrize_plus in order for pytest (pluggy) not to think they are hooks. Old aliases will stay around for a few versions, with a deprecation warning.
"},{"location":"changelog/#1120-better-test-ids-for-parametrized-tests-with-fixture-refs-bugfix","title":"1.12.0 - better test ids for parametrized tests with fixture refs + bugfix","text":" -
Improved test ids for the cases where fixture_ref is used in the parameters list in @pytest_parametrize_plus. Fixed #69. Thanks last-partizan for the suggestion !
-
Fixed TypeError: got an unexpected keyword argument 'indirect' with pytest 5+. Fixed #70.
"},{"location":"changelog/#1119-bugfix","title":"1.11.9 - bugfix","text":"FixtureClosureNode is now able to properly handle ignore_args, and now supports that plugins append fixtures to the closure, such as pytest-asyncio. Added corresponding tests. Fixes #68
"},{"location":"changelog/#1118-bugfix","title":"1.11.8 - bugfix","text":"Fixed KeyError issue happening when a fixture is not found. Now users will see the \"normal\" error message from pytest (\"fixture <name> not found\"). Fixed #67.
"},{"location":"changelog/#1117-bugfix","title":"1.11.7 - bugfix","text":"Fixed ValueError issue happening with indirectly parametrized fixtures. Fixed #64.
"},{"location":"changelog/#1116-pyprojecttoml","title":"1.11.6 - pyproject.toml","text":"raddessi added a pyproject.toml - thanks! Fixed #65.
"},{"location":"changelog/#1115-bugfix","title":"1.11.5 - bugfix","text":"pytest_parametrize_plus was not working correctly with test classes, leading to fixture 'self' not found. Fixed #63.
"},{"location":"changelog/#1114-python-2-bugfix","title":"1.11.4 - python 2 bugfix","text":"Fixed issue happening with @pytest.mark.parametrize with python 2. Fixed #62.
"},{"location":"changelog/#1113-minor-improvements","title":"1.11.3 - minor improvements","text":"Better error message when users use THIS_MODULE in cases= instead of module=.
Added __version__ package-level attribute.
"},{"location":"changelog/#1112-increased-tolerance-to-other-plugins-bugfix","title":"1.11.2 - Increased tolerance to other plugins + bugfix","text":"Now when other plugins try to manipulate the fixture closure, warning messages are emitted but no error is raised. Fixed #55.
Also fixed issue #58 happening with doctest.
"},{"location":"changelog/#1111-added-six-dependency-explicitly","title":"1.11.1 - Added six dependency explicitly","text":"It was missing from setup.py.
"},{"location":"changelog/#1110-fixture_ref-can-now-be-used-inside-tuples-leading-to-cross-products","title":"1.11.0 - fixture_ref can now be used inside tuples, leading to cross-products","text":"Fixes #47.
"},{"location":"changelog/#1102-more-intuitive-error-messages","title":"1.10.2 - More intuitive error messages","text":"Now raising an explicit InvalidParamsList when pytest parametrize argvalues are incorrect. See #54
"},{"location":"changelog/#1101-bugfix","title":"1.10.1 - Bugfix","text":"Fixed #52.
"},{"location":"changelog/#1100-new-feature-fixtures-unpacking","title":"1.10.0 - New feature: fixtures unpacking","text":"You can now unpack a fixture iterable into several individual fixtures using unpack_fixture or using @pytest_fixture_plus(unpack_into=<names>). This is also available in union_fixture(unpack_into=<names>). Fixed #50 and #51.
"},{"location":"changelog/#193-bugfix","title":"1.9.3 - Bugfix","text":"Fixed issues when parametrize argnames contains a list. This fixed #49
"},{"location":"changelog/#192-bugfix-with-pytest-37","title":"1.9.2 - Bugfix with pytest 3.7","text":"Fixed #48.
"},{"location":"changelog/#191-bugfix-with-pytest-37","title":"1.9.1 - Bugfix with pytest 3.7","text":"Fixed #48.
"},{"location":"changelog/#190-new-with-reorder-commandline-option","title":"1.9.0 - New --with-reorder commandline option","text":"New commandline option '--with-reorder' to change the reordering strategy currently in application. Fixes #45.
The --with-reorder \"skip\" mode was not working correctly in presence of marks, fixed it. Fixed #46.
"},{"location":"changelog/#181-bugfixes","title":"1.8.1 - BugFixes","text":"Ids should not be used when setting a NOT_USED parametrization. Fixes #43
Fixed issue with ordering and setup/teardown for higher-level scope fixtures (session and module scopes) when using union fixtures. Fixes #44
"},{"location":"changelog/#180-better-ids-for-fixture-unions","title":"1.8.0 - Better ids for fixture unions","text":"New:
fixture_union now accept a non-None value for ids. It also has a new idstyle argument allowing users to change the style of ids used. Finally pytest_parametrize_plus relies on this ids argument to set a more readable list of ids for the created union. Fixes #41.
Misc:
- Added non-regression test for fixture order. It passes already for all recent pytest versions (after 3.3). Fixes #42
"},{"location":"changelog/#170-new-pytest_parametrize_plus-allowing-fixture-references-to-be-used-in-parameter-values","title":"1.7.0 - New @pytest_parametrize_plus allowing fixture references to be used in parameter values","text":"New decorator @pytest_parametrize_plus able to handle the case where a fixture_ref(<fixture_name>) is present in the parameter values list. This decorator can be applied both on test functions and fixtures (if they are decorated with @pytest_fixture_plus). Fixes #40
Major refactoring of the \"union fixtures\" mechanism.
- The
NOT_USED status is now correctly propagated between dependent fixtures. This should fix a few cases where user fixtures were setup/teardown while not used in the current test node. - Empty fixture unions are not permitted anymore.
- The way unions are handled in test parametrization was redesigned. The new design is based on a two-steps approach: first build the fixture closure for each node as a tree (and not a list as in
pytest), and then apply parametrization intelligently based on this tree structure. This fixes several unintuitive behaviours that were happening with unions.
Note: interestingly this also fixes pytest#5054.
"},{"location":"changelog/#163-minor-exception-enhancement","title":"1.6.3 - Minor exception enhancement","text":"Improved the error message when the name template is wrong in @cases_generator. Fixes #39.
"},{"location":"changelog/#162-bug-fixes","title":"1.6.2 - bug fixes","text":"fixture_union:
param_fixtures:
-
param_fixtures now delegates to param_fixture when a single parameter name is provided. This is more consistent. Fixed #36.
-
param_fixture[s] now support all arguments from fixture (scope and autouse in particular).
"},{"location":"changelog/#161-pytest_fixture_plus-improvement-to-handle-not_used-cases","title":"1.6.1 - @pytest_fixture_plus improvement to handle NOT_USED cases","text":"Fixed issue where fixtures get called with NOT_USED as a parameter when using a fixture_union. This issue is actually only fixed in @pytest_fixture_plus, if you use @pytest.fixture you have to handle it manually. Fixes #37.
"},{"location":"changelog/#160-fixture_union-and-param_fixtures-bugfix","title":"1.6.0 - fixture_union and param_fixture[s] bugfix","text":"New fixture_union method to create a fixture that is the union/combination of other fixtures. This is an attempt to solve this pytest proposal.
Also, param_fixture and param_fixtures can now be used without necessarily storing the return value into a variable: they will automatically register the created fixtures in the calling module.
Finally, fixed a bug with param_fixtures when called to create a fixture for a single parameter.
"},{"location":"changelog/#151-param_fixtures-bugfix","title":"1.5.1 - param_fixtures bugfix","text":"Fixed param_fixtures issue: all parameter values were identical to the last parameter of the tuple. Fixes #32.
"},{"location":"changelog/#150-new-helpers-param_fixture-and-param_fixtures","title":"1.5.0 - new helpers param_fixture and param_fixtures","text":"Following Sup3rGeo's proposal, introduced two helper methods to create simple \"parameter fixtures\". Fixes #31.
"},{"location":"changelog/#142-parametrized-pytest_fixture_plus-minor-bug-fix","title":"1.4.2 - parametrized @pytest_fixture_plus minor bug fix","text":"@pytest_fixture_plus now correctly honors parameter id and marks overridden at single parameter level using pytest.param. Fixed #30.
"},{"location":"changelog/#141-parametrized-pytest_fixture_plus-minor-bug-fix","title":"1.4.1 - parametrized @pytest_fixture_plus minor bug fix","text":"Fixed @pytest_fixture_plus in case it is used with parametrize and one parameter is itself customized using pytest.param. Fixed #29.
"},{"location":"changelog/#140-pytest_fixture_plus-major-improvement","title":"1.4.0 - @pytest_fixture_plus major improvement","text":" -
Major improvement of @pytest_fixture_plus: instead of generating fixtures, it now correctly parametrizes the fixture. Skip/fail Marks are correctly copied too. Fixes #28.
-
pytest_fixture_plus does not accept the params and ids arguments any more, it only relies on parametrization marks.
"},{"location":"changelog/#133-parametrized-pytest_fixture_plus-bugfix","title":"1.3.3 - parametrized @pytest_fixture_plus Bugfix","text":"Fixed minor bug with parametrized @pytest_fixture_plus: spaces are now correctly removed when multiple parameter names are provided in the same parametrize call. Fixes #27.
"},{"location":"changelog/#132-parametrized-pytest_fixture_plus-bugfix","title":"1.3.2 - parametrized @pytest_fixture_plus Bugfix","text":"Fixed bug with @pytest_fixture_plus when used in parametrized mode. Fixes #26. Thanks Sup3rGeo!
"},{"location":"changelog/#131-minor-dependency-change","title":"1.3.1 - Minor dependency change","text":"Now using decopatch to create the decorators.
"},{"location":"changelog/#130-more-flexible-case-generators-names-minor-dependency-change","title":"1.3.0 - More flexible case generators names + Minor dependency change","text":"Cases generators can now support explicit name lists, and name generator callables, in addition to the name template strings. Fixed #24.
Dependency to decorator has been dropped and replaced with makefun. Fixed #25.
"},{"location":"changelog/#122-fixed-bug-with-marks-on-cases-with-pytest-33","title":"1.2.2 - fixed bug with marks on cases with pytest 3.3","text":"Marks on cases are now also working with pytest 3.3. Fixed #23.
Ids for marked tests are now better managed. A new function get_pytest_parametrize_args is now used to transform the list of cases obtained by get_all_cases(module), into the list of marked cases and ids required by @pytest.mark.parametrize. The doc has been updated to explain this for advanced users wishing to perform this step manually.
"},{"location":"changelog/#121-fixed-id-of-test-cases-with-marks","title":"1.2.1 - fixed id of test cases with marks","text":"Id of test cases with marks was appearing as ParameterSet. Fixed it.
"},{"location":"changelog/#120-pytestmark-can-be-used-on-cases-pytest_fixture_plus-parametrization-order-bugfix","title":"1.2.0 - @pytest.mark can be used on cases + @pytest_fixture_plus parametrization order bugfix","text":"Pytest marks such as @pytest.mark.skipif can now be used on case functions. As a consequence, get_all_cases is now the recommended function to use instead of extract_cases_from_module to perform manual collection. Indeed get_all_cases correctly prepares the resulting parameters list so that pytest sees the marks. Fixed #21.
Fixed parametrization order when @pytest_fixture_plus is used with several @pytest.mark.parametrize. Fixed #22.
"},{"location":"changelog/#111-improved-generated-fixture-names-for-pytest_fixture_plus","title":"1.1.1 - Improved generated fixture names for @pytest_fixture_plus","text":"When @pytest_fixture_plus is used on a function marked as parametrized, some fixtures are generated (one for each parameter). Generated fixture names now follow the pattern <fixturename>__<paramname>. Fixed #20.
"},{"location":"changelog/#110-new-pytest_fixture_plus","title":"1.1.0 - New @pytest_fixture_plus","text":"New decorator @pytest_fixture_plus allows to use several @pytest.mark.parametrize on a fixture. Therefore one can use multiple @cases_data decorators, too. Fixes #19. Note: this is a temporary feature, that will be removed if/when pytest supports it.
"},{"location":"changelog/#100-cases_fixture-pytest-2x-support","title":"1.0.0 - @cases_fixture + pytest 2.x support","text":"Pytest 2.x is now supported. Fixes #14.
New feature: @cases_fixture ! Now you can put your cases data retrieval in a fixture so that its duration does not enter into the test duration. This is particularly interesting if you use pytest-harvest to create benchmarks: you probably do not want the case data retrieval/parsing to be counted in the test duration, especially if you use caching on the case function to accelerate subsequent retrievals. Fixes #15.
"},{"location":"changelog/#0101-minor-encoding-issue-in-setuppy","title":"0.10.1 - minor encoding issue in setup.py","text":""},{"location":"changelog/#0100-support-for-python-2","title":"0.10.0 - support for python 2","text":"Python 2 is now supported. Fixed #3.
- Note:
CaseData, Given, ExpectedNormal, ExpectedError, and MultipleStepsCaseData type hints is not created in python 2 and python<3.5
"},{"location":"changelog/#091-pytest-steps-is-now-an-independent-project","title":"0.9.1 - pytest-steps is now an independent project","text":" - Light refactoring: some internal function names are now private, and there are now two submodules.
- pytest-steps is now an independent project. Examples in the documentation have been updated
- New documentation page: API reference
"},{"location":"changelog/#080-filtering-can-now-be-done-using-a-callable","title":"0.8.0 - Filtering can now be done using a callable.","text":" @cases_data: the filter argument now contains a filtering function. WARNING: the previous behaviour is still available but has been renamed has_tag. Fixes #8.
"},{"location":"changelog/#070-hardcoded-cases-selection-and-multi-module-selection","title":"0.7.0 - Hardcoded cases selection, and multi-module selection","text":" @cases_data has a new parameters cases that can be used to hardcode a case or a list of cases. Its module parameter can also now take a list of modules
"},{"location":"changelog/#060-case-parameters-and-better-test-suites","title":"0.6.0 - Case parameters and better test suites","text":" get_for is deprecated: it was too specific to a given case data format.MultipleStepsCaseData was fixed to also support multiple inputs.- Case functions can now have parameters (even case generators). This is particularly useful for test suites. Fixes #9.
"},{"location":"changelog/#050-support-for-test-suites","title":"0.5.0 - support for test suites","text":" - test functions can now be decorated with
@test_steps to easily define a test suite with several steps. This fixes #7.
"},{"location":"changelog/#040-support-for-data-caching-with-lru_cache","title":"0.4.0 - support for data caching with lru_cache","text":" - cases can now be decorated with
@lru_cache. @cases_generator also provides a lru_cache parameter to enable caching. Fixes #6.
"},{"location":"changelog/#030-case-generators","title":"0.3.0 - case generators","text":" - New decorator
@cases_generator to define case generators. Fixes #1. - Also, removed unused functions
is_expected_error_instance and assert_exception_equal
"},{"location":"changelog/#020-this_module-constant-taggingfiltering-doc","title":"0.2.0 - THIS_MODULE constant + Tagging/Filtering + doc","text":" -
New constant THIS_MODULE so that cases and test functions can coexist in the same file. This fixes #5.
-
Added @test_target and @case_tags decorators for case functions, and added filter parameter in @cases_data. This allows users to :
- tag a case function with any item (and in particular with the reference to the function it relates to),
- and to filter the case functions used by a test function according to a particular tag.
This fixes #4.
"},{"location":"changelog/#010-first-public-version","title":"0.1.0 - First public version","text":" - Initial fork from private repo
"},{"location":"examples/","title":"Examples","text":"Our examples are hosted in a separate repository so that they can be updated independently.
- data science benchmark demonstrates how
pytest can be used as a benchmarking engine thanks to pytest-cases and pytest-harvest, to compare the performances of several regression algorithms on several datasets and produce various reports (plots, csv table...).
"},{"location":"long_description/","title":"pytest-cases","text":"Separate test code from test cases in pytest.
The documentation for users is available here: https://smarie.github.io/python-pytest-cases/
A readme for developers is available here: https://github.com/smarie/python-pytest-cases
"},{"location":"pytest_goodies/","title":"pytest Goodies","text":"Many pytest features were missing to make pytest_cases work with such a \"no-boilerplate\" experience. Many of these can be of interest to the general pytest audience, so they are exposed in the public API.
"},{"location":"pytest_goodies/#fixture","title":"@fixture","text":"@fixture is similar to pytest.fixture but without its param and ids arguments. Instead, it is able to pick the parametrization from @pytest.mark.parametrize marks applied on fixtures. This makes it very intuitive for users to parametrize both their tests and fixtures. As a bonus, its name argument works even in old versions of pytest (which is not the case for fixture).
Finally it now supports unpacking, see unpacking feature.
@fixture deprecation if/when @pytest.fixture supports @pytest.mark.parametrize
The ability for pytest fixtures to support the @pytest.mark.parametrize annotation is a feature that clearly belongs to pytest scope, and has been requested already. It is therefore expected that @fixture will be deprecated in favor of @pytest_fixture if/when the pytest team decides to add the proposed feature. As always, deprecation will happen slowly across versions (at least two minor, or one major version update) so as for users to have the time to update their code bases.
"},{"location":"pytest_goodies/#unpack_fixture-unpack_into","title":"unpack_fixture / unpack_into","text":"In some cases fixtures return a tuple or a list of items. It is not easy to refer to a single of these items in a test or another fixture. With unpack_fixture you can easily do it:
import pytest\nfrom pytest_cases import unpack_fixture, fixture\n\n@fixture\n@pytest.mark.parametrize(\"o\", ['hello', 'world'])\ndef c(o):\n return o, o[0]\n\na, b = unpack_fixture(\"a,b\", c)\n\ndef test_function(a, b):\n assert a[0] == b\n
Note that you can also use the unpack_into= argument of @fixture to do the same thing:
import pytest\nfrom pytest_cases import fixture\n\n@fixture(unpack_into=\"a,b\")\n@pytest.mark.parametrize(\"o\", ['hello', 'world'])\ndef c(o):\n return o, o[0]\n\ndef test_function(a, b):\n assert a[0] == b\n
And it is also available in fixture_union:
import pytest\nfrom pytest_cases import fixture, fixture_union\n\n@fixture\n@pytest.mark.parametrize(\"o\", ['hello', 'world'])\ndef c(o):\n return o, o[0]\n\n@fixture\n@pytest.mark.parametrize(\"o\", ['yeepee', 'yay'])\ndef d(o):\n return o, o[0]\n\nfixture_union(\"c_or_d\", [c, d], unpack_into=\"a, b\")\n\ndef test_function(a, b):\n assert a[0] == b\n
"},{"location":"pytest_goodies/#param_fixtures","title":"param_fixture[s]","text":"If you wish to share some parameters across several fixtures and tests, it might be convenient to have a fixture representing this parameter. This is relatively easy for single parameters, but a bit harder for parameter tuples.
The two utilities functions param_fixture (for a single parameter name) and param_fixtures (for a tuple of parameter names) handle the difficulty for you:
import pytest\nfrom pytest_cases import param_fixtures, param_fixture\n\n# create a single parameter fixture\nmy_parameter = param_fixture(\"my_parameter\", [1, 2, 3, 4])\n\n@pytest.fixture\ndef fixture_uses_param(my_parameter):\n ...\n\ndef test_uses_param(my_parameter, fixture_uses_param):\n ...\n\n# -----\n# create a 2-tuple parameter fixture\narg1, arg2 = param_fixtures(\"arg1, arg2\", [(1, 2), (3, 4)])\n\n@pytest.fixture\ndef fixture_uses_param2(arg2):\n ...\n\ndef test_uses_param2(arg1, arg2, fixture_uses_param2):\n ...\n
You can mark any of the argvalues with pytest.mark to pass a custom id or a custom \"skip\" or \"fail\" mark, just as you do in pytest. See pytest documentation.
"},{"location":"pytest_goodies/#fixture_union","title":"fixture_union","text":"As of pytest 5, it is not possible to create a \"union\" fixture, i.e. a parametrized fixture that would first take all the possible values of fixture A, then all possible values of fixture B, etc. Indeed all fixture dependencies (a.k.a. \"closure\") of each test node are grouped together, and if they have parameters a big \"cross-product\" of the parameters is done by pytest.
The topic has been largely discussed in pytest-dev#349 and a request for proposal has been finally made.
fixture_union is an implementation of this proposal. It is also used by @parametrize to support fixture_ref in parameter values, see below. The theory is presented in more details in this page, while below are more practical examples.
from pytest_cases import fixture, fixture_union\n\n@fixture\ndef first():\n return 'hello'\n\n@fixture(params=['a', 'b'])\ndef second(request):\n return request.param\n\n# c will first take all the values of 'first', then all of 'second'\nc = fixture_union('c', [first, second])\n\ndef test_basic_union(c):\n print(c)\n
yields
<...>::test_basic_union[\\first] hello PASSED\n<...>::test_basic_union[\\second-a] a PASSED\n<...>::test_basic_union[\\second-b] b PASSED\n
"},{"location":"pytest_goodies/#idstyle","title":"idstyle","text":"As you can see the ids of union fixtures are slightly different from standard ids, so that you can easily understand what is going on. You can change this feature with \u00ecdstyle, see API documentation for details.
"},{"location":"pytest_goodies/#marks-and-ids","title":"marks and ids","text":"You can mark any of the alternatives with pytest.mark to pass a custom id or a custom \"skip\" or \"fail\" mark, just as you do in pytest. See pytest documentation.
"},{"location":"pytest_goodies/#unpacking","title":"unpacking","text":"Fixture unions also support unpacking with the unpack_into argument, see unpacking feature.
"},{"location":"pytest_goodies/#to-conclude","title":"to conclude","text":"Fixture unions are a major change in the internal pytest engine, as fixture closures (the ordered set of all fixtures required by a test node to run - directly or indirectly) now become trees where branches correspond to alternative paths taken in the \"unions\", and leafs are the alternative fixture closures. This feature has been tested in very complex cases (several union fixtures, fixtures that are not selected by a given union but that is requested by the test function, etc.). But if you find some strange behaviour don't hesitate to report it in the issues page !
IMPORTANT if you do not use @fixture but only @pytest.fixture, then you will see that your fixtures are called even when they are not used, with a parameter NOT_USED. This symbol is automatically ignored if you use @fixture, otherwise you have to handle it. Alternatively you can use @ignore_unused on your fixture function.
fixture unions vs. cases
If you're familiar with pytest-cases already, you might note that @cases_data is not so different than a fixture union: we do a union of all case functions. If one day union fixtures are directly supported by pytest, we will probably refactor this lib to align all the concepts.
"},{"location":"pytest_goodies/#parametrize","title":"@parametrize","text":"@parametrize is a replacement for @pytest.mark.parametrize with many additional features to make the most of parametrization. See API reference for details about all the new features. In particular it allows you to include references to fixtures and to value-generating functions in the parameter values.
- Simply use
fixture_ref(<fixture>) in the parameter values, where <fixture> can be the fixture name or fixture function. New: from version 3.2 on, if auto_refs=True (default), @parametrize will automatically detect fixture symbols in the list of argvalues, and will create fixture_refs automatically around them so that you don't need to. - if you do not wish to create a fixture, you can also use
lazy_value(<function>) - Note that when parametrizing several argnames, both
fixture_ref and lazy_value can be used as the tuple, or in the tuple. Several fixture_ref and/or lazy_value can be used in the same tuple, too. - By default the id associated with a
fixture_ref or a lazy_value is the name of the fixture or function. Custom ids can be passed with the id=<id> parameter.
For example, with a single argument:
import pytest\nfrom pytest_cases import parametrize, fixture, fixture_ref, lazy_value\n\n@pytest.fixture\ndef world_str():\n return 'world'\n\ndef whatfun():\n return 'what'\n\n@fixture\n@parametrize('who', [world_str, 'you'])\ndef greetings(who):\n return 'hello ' + who\n\n@parametrize('main_msg', ['nothing', \n fixture_ref(world_str),\n lazy_value(whatfun),\n \"1\",\n fixture_ref(greetings)], \n auto_refs=False)\n@pytest.mark.parametrize('ending', ['?', '!'])\ndef test_prints(main_msg, ending):\n print(main_msg + ending)\n
yields the following
> pytest -s -v\ncollected 12 items\ntest_prints[nothing-?] PASSED [ 8%]nothing?\ntest_prints[nothing-!] PASSED [ 16%]nothing!\ntest_prints[world_str-?] PASSED [ 25%]world?\ntest_prints[world_str-!] PASSED [ 33%]world!\ntest_prints[whatfun-?] PASSED [ 41%]what?\ntest_prints[whatfun-!] PASSED [ 50%]what!\ntest_prints[1-?] PASSED [ 58%]1?\ntest_prints[1-!] PASSED [ 66%]1!\ntest_prints[greetings-world_str-?] PASSED [ 75%]hello world?\ntest_prints[greetings-world_str-!] PASSED [ 83%]hello world!\ntest_prints[greetings-you-?] PASSED [ 91%]hello you?\ntest_prints[greetings-you-!] PASSED [100%]hello you!\n
"},{"location":"pytest_goodies/#ids-and-marks","title":"ids and marks","text":"You can also mark any of the argvalues with pytest.param to pass a custom id or a custom \"skip\" or \"fail\" mark, just as you do in pytest. See pytest documentation.
You can also pass a custom callable or generator in ids as in @pytest.mark.parametrize.
"},{"location":"pytest_goodies/#idstyle-customization","title":"idstyle customization","text":"As you can see in the example above, the default ids are similar to what you would intuitively expect, even when you use fixture_ref.
This is because by default idstyle=None, to preserve test ids very close to standard pytest by default. But still, a fixture_union is generated behind the scenes when there is a fixture reference. So this is actually non-standard. You may therefore prefer to see explicit ids showing the various fixture alternatives, as in fixture_union. For this simply set the idstyle to 'compact', 'explicit' or to a callable such as str.
For example, changing the previous example to add idstyle=\"explicit\":
(...same as above...)\n\n@parametrize('main_msg', ['nothing',\n fixture_ref(world_str),\n lazy_value(whatfun),\n \"1\",\n fixture_ref(greetings)], idstyle=\"explicit\")\n@pytest.mark.parametrize('ending', ['?', '!'])\ndef test_prints(main_msg, ending):\n print(main_msg + ending)\n
yields to
> pytest -s -v\ncollected 12 items\ntest_prints[main_msg\\nothing-?] PASSED [ 8%]nothing?\ntest_prints[main_msg\\nothing-!] PASSED [ 16%]nothing!\ntest_prints[main_msg\\world_str-?] PASSED [ 25%]world?\ntest_prints[main_msg\\world_str-!] PASSED [ 33%]world!\ntest_prints[main_msg\\P2:4-whatfun-?] PASSED [ 41%]what?\ntest_prints[main_msg\\P2:4-whatfun-!] PASSED [ 50%]what!\ntest_prints[main_msg\\P2:4-1-?] PASSED [ 58%]1?\ntest_prints[main_msg\\P2:4-1-!] PASSED [ 66%]1!\ntest_prints[main_msg\\greetings-world_str-?] PASSED [ 75%]hello world?\ntest_prints[main_msg\\greetings-world_str-!] PASSED [ 83%]hello world!\ntest_prints[main_msg\\greetings-you-?] PASSED [ 91%]hello you?\ntest_prints[main_msg\\greetings-you-!] PASSED [100%]hello you!\n
You can see that with this explicit style, the various \"alternatives\" in the fixture union generated behind the scenes for the main_msg parameter appear explicitly. In particular you see that there is an alternative main_msg\\P2:4 covering several parameters in a row.
Note that this idstyle is not taken into account if you only use lazy_values but no fixture_ref, as lazy_values do not require to create a fixture union behind the scenes.
"},{"location":"pytest_goodies/#parametrization-order","title":"parametrization order","text":"Another consequence of using fixture_ref is that the priority order of the parameters, relative to other standard pytest.mark.parametrize parameters that you would place on the same function, will get impacted. You may solve this by replacing your other @pytest.mark.parametrize calls with param_fixtures so that all the parameters are fixtures (see param_fixture).
"},{"location":"pytest_goodies/#passing-a-hook","title":"passing a hook","text":"As per version 1.14, all the above functions now support passing a hook argument. This argument should be a callable. It will be called every time a fixture is about to be created by pytest_cases on your behalf. The fixture function is passed as the argument of the hook, and the hook should return it as the result.
You can use this fixture to better understand which fixtures are created behind the scenes, and also to decorate the fixture functions before they are created. For example you can use hook=saved_fixture (from pytest-harvest) in order to save the created fixtures in the fixture store.
"},{"location":"pytest_goodies/#assert_exception","title":"assert_exception","text":"assert_exception context manager is an alternative to pytest.raises to check exceptions in your tests. You can either check type, instance equality, repr string pattern, or use custom validation functions. See API reference.
"},{"location":"pytest_goodies/#-with-reorder","title":"--with-reorder","text":"pytest postprocesses the order of the collected items in order to optimize setup/teardown of session, module and class fixtures. This optimization algorithm happens at the pytest_collection_modifyitems stage, and is still under improvement, as can be seen in pytest#3551, pytest#3393, #2846...
Besides other plugins such as pytest-reorder can modify the order as well.
This new commandline is a goodie to change the reordering:
-
--with-reorder normal is the default behaviour: it lets pytest and all the plugins execute their reordering in each of their pytest_collection_modifyitems hooks, and simply does not interact
-
--with-reorder skip allows you to restore the original order that was active before pytest_collection_modifyitems was initially called, thus not taking into account any reordering done by pytest or by any of its plugins.
"},{"location":"unions_theory/","title":"Theory behind fixture_union","text":""},{"location":"unions_theory/#1-how-pytest-works-today","title":"1. How pytest works today","text":"As of pytest 5, there are three kind of concepts at play to generate the list of test nodes and their received parameters (\"call spec\" in pytest internals).
-
test functions are the functions defined with def test_<name>(<args>).
-
they can be parametrized using @pytest.mark.parametrize (or our enhanced version @parametrize). That means that some of the <args> will take several values, and for each combination a distinct test node will be created
-
they can require fixtures, that is, functions decorated with @pytest.fixture (or our enhanced version @fixture). That means that some of the <args> will take the value of the corresponding fixture(s).
-
fixtures can be parametrized too (with @fixture it is easier :) ), and can require other fixtures.
-
finally fixtures can enable an \"auto-use\" mode, so that they are called even when not explicitly required by anything.
Therefore, a test plan can be represented as an acyclic directed graph of fixtures, where nodes are fixtures and edges represent dependencies. On top of this layout, we can overlay the information of which fixture nodes are parametrized, which ones are required by which test function, and which test function is parametrized. The resulting figure is presented below:
The following code can be used to easily check the number of tests run. Note that we use @fixture and @parametrize from pytest-cases to ease code readability here but you would get a similar behaviour with @pytest.fixture and @pytest.mark.parametrize (the test ids would not show the parameter names by default though, which is helpful for our demonstration here).
from pytest_cases import fixture, parametrize\n\n@fixture(autouse=True)\n@parametrize(ie=[-1, 1])\ndef e(ie):\n return \"e%s\" % ie\n\n@fixture\ndef d():\n return \"d\"\n\n@fixture\ndef c():\n return \"c\"\n\n@fixture\n@parametrize(ia=[0, 1])\ndef a(c, d, ia):\n return \"a%s\" % ia + c + d\n\n@parametrize(i2=['x', 'z'])\ndef test_2(a, i2):\n assert (a + i2) in (\"a0cdx\", \"a0cdz\", \"a1cdx\", \"a1cdz\")\n\n@fixture\n@parametrize(ib=['x', 'z'])\ndef b(a, c, ib):\n return \"b%s\" % ib + c + a\n\ndef test_1(a, b):\n assert a in (\"a0cd\", \"a1cd\")\n assert a == b[-4:]\n assert b[:-4] in (\"bxc\", \"bzc\")\n
calling pytest yields:
============================= test session starts =============================\ncollecting ... collected 16 items\n\ntest_doc_fixture_graph.py::test_2[ie=-1-ia=0-i2=x] \ntest_doc_fixture_graph.py::test_2[ie=-1-ia=0-i2=z] \ntest_doc_fixture_graph.py::test_2[ie=-1-ia=1-i2=x] \ntest_doc_fixture_graph.py::test_2[ie=-1-ia=1-i2=z] \ntest_doc_fixture_graph.py::test_2[ie=1-ia=0-i2=x] \ntest_doc_fixture_graph.py::test_2[ie=1-ia=0-i2=z] \ntest_doc_fixture_graph.py::test_2[ie=1-ia=1-i2=x] \ntest_doc_fixture_graph.py::test_2[ie=1-ia=1-i2=z] \ntest_doc_fixture_graph.py::test_1[ie=-1-ia=0-ib=x] \ntest_doc_fixture_graph.py::test_1[ie=-1-ia=0-ib=z] \ntest_doc_fixture_graph.py::test_1[ie=-1-ia=1-ib=x] \ntest_doc_fixture_graph.py::test_1[ie=-1-ia=1-ib=z] \ntest_doc_fixture_graph.py::test_1[ie=1-ia=0-ib=x] \ntest_doc_fixture_graph.py::test_1[ie=1-ia=0-ib=z] \ntest_doc_fixture_graph.py::test_1[ie=1-ia=1-ib=x] \ntest_doc_fixture_graph.py::test_1[ie=1-ia=1-ib=z] \n\n============================= 16 passed in 0.14s ==============================\n
So each test is called 8 times. How are these calls computed ?
- first for each test,
pytest computes the set of all fixtures that are directly or indirectly required to run it. This is known as the \"fixture closure\". So for test_1 this closure is {a, b, c, d, e} while for test 2 it is {a, c, d, e}. We can show this on the following picture:
- then a cartesian product is made across the parameters of all parametrization marks found on any item in the closure (including parameters of the test itself), So for
test_1 the cartesian product is <ie> x <ia> x <ib> while for test_2 it is <ie> x <ia> x <i2>. This is why both tests result in having 8 variants being called (see details in the test ids above).
"},{"location":"unions_theory/#2-extension-to-fixture-unions","title":"2. Extension to fixture unions.","text":"A fixture union is by definition a fixture that is parametrized to alternately depend on other fixtures. We will represent this in the figures with a special dashed orange arrow, to remind that a special parameter is associated with the selection of which arrow is activated.
Let's consider the following modification of the above example, where we introduce two \"unions\": one as an explicit fixture u, and the other implicitly created by using fixture_refs in the parametrization of b.
We can create such a configuration with a slight modification to the above example:
from pytest_cases import fixture, parametrize, fixture_ref, fixture_union\n\n(... same as above ...)\n\n@fixture\n@parametrize(ib=['x', 'z'])\n@parametrize(ub=(fixture_ref(a), fixture_ref(c)), idstyle=\"explicit\")\ndef b(ub, ib):\n return \"b%s\" % ib + ub\n\nu = fixture_union(\"u\", (a, b), idstyle=\"explicit\")\n\ndef test_1(u):\n pass\n
Note the idstyle=\"explicit\" keyword arguments, that will help us get more details in the test ids.
Calling pytest yields:
============================= test session starts =============================\ncollecting ... collected 24 items\n\ntest_doc_fixture_graph_union.py::test_2[ie=-1-ia=0-i2=x] PASSED [ 4%]\ntest_doc_fixture_graph_union.py::test_2[ie=-1-ia=0-i2=z] PASSED [ 8%]\ntest_doc_fixture_graph_union.py::test_2[ie=-1-ia=1-i2=x] PASSED [ 12%]\ntest_doc_fixture_graph_union.py::test_2[ie=-1-ia=1-i2=z] PASSED [ 16%]\ntest_doc_fixture_graph_union.py::test_2[ie=1-ia=0-i2=x] PASSED [ 20%]\ntest_doc_fixture_graph_union.py::test_2[ie=1-ia=0-i2=z] PASSED [ 25%]\ntest_doc_fixture_graph_union.py::test_2[ie=1-ia=1-i2=x] PASSED [ 29%]\ntest_doc_fixture_graph_union.py::test_2[ie=1-ia=1-i2=z] PASSED [ 33%]\ntest_doc_fixture_graph_union.py::test_1[ie=-1-u\\a-ia=0] PASSED [ 37%]\ntest_doc_fixture_graph_union.py::test_1[ie=-1-u\\a-ia=1] PASSED [ 41%]\ntest_doc_fixture_graph_union.py::test_1[ie=-1-u\\b-ib=x-ub\\a-ia=0] PASSED [ 45%]\ntest_doc_fixture_graph_union.py::test_1[ie=-1-u\\b-ib=x-ub\\a-ia=1] PASSED [ 50%]\ntest_doc_fixture_graph_union.py::test_1[ie=-1-u\\b-ib=x-ub\\c] PASSED [ 54%]\ntest_doc_fixture_graph_union.py::test_1[ie=-1-u\\b-ib=z-ub\\a-ia=0] PASSED [ 58%]\ntest_doc_fixture_graph_union.py::test_1[ie=-1-u\\b-ib=z-ub\\a-ia=1] PASSED [ 62%]\ntest_doc_fixture_graph_union.py::test_1[ie=-1-u\\b-ib=z-ub\\c] PASSED [ 66%]\ntest_doc_fixture_graph_union.py::test_1[ie=1-u\\a-ia=0] PASSED [ 70%]\ntest_doc_fixture_graph_union.py::test_1[ie=1-u\\a-ia=1] PASSED [ 75%]\ntest_doc_fixture_graph_union.py::test_1[ie=1-u\\b-ib=x-ub\\a-ia=0] PASSED [ 79%]\ntest_doc_fixture_graph_union.py::test_1[ie=1-u\\b-ib=x-ub\\a-ia=1] PASSED [ 83%]\ntest_doc_fixture_graph_union.py::test_1[ie=1-u\\b-ib=x-ub\\c] PASSED [ 87%]\ntest_doc_fixture_graph_union.py::test_1[ie=1-u\\b-ib=z-ub\\a-ia=0] PASSED [ 91%]\ntest_doc_fixture_graph_union.py::test_1[ie=1-u\\b-ib=z-ub\\a-ia=1] PASSED [ 95%]\ntest_doc_fixture_graph_union.py::test_1[ie=1-u\\b-ib=z-ub\\c] PASSED [100%]\n\n======================== 24 passed, 1 warning in 0.30s ========================\n
Now 24 tests were created ! test_2 still has 8 runs, which is normal as it does not depend on any union fixture. Let's try to understand what happened to parametrization of test_1. It is actually fairly simple:
-
first a global fixture closure is created as usual, consisting in {u, a, b, c, d, e}
-
then for each union fixture in test_1's closure, starting from the bottom of the graph, we generate several closures by activating each of the arrows in turn. We progress upwards through the graph of remaining dependencies for each alternative:
So the result consists in 3 alternate fixture closures for test_1:
-
finally, as usual, for each closure a cartesian product is made across the parameters of all parametrization marks found on any item in the closure (including parameters of the test itself), So
- for
test_1 alternative u\\a, the cartesian product is <ie> x <ia> (4 tests) - for
test_1 alternative u\\b-ub\\a, the cartesian product is <ie> x <ia> x <ib> (8 tests) - for
test_1 alternative u\\b-ub\\c, the cartesian product is <ie> x <ib> (4 tests) - for
test_2 it is <ie> x <ia> x <i2>. (8 tests).
The total is indeed 4 + 8 + 4 + 8 = 24 tests. Once again the test ids may be used to check that everything is correct, see above.
"}]}
\ No newline at end of file
diff --git a/sitemap.xml.gz b/sitemap.xml.gz
index fb05e5c..25b6d92 100644
Binary files a/sitemap.xml.gz and b/sitemap.xml.gz differ