[Python] Compatibility with pytest 7.4

[h-vetinari]

Describe the bug, including details regarding any error messages, version, and platform.

When running the pyarrow test suite with pytest 7.4, I get:

=========================== short test summary info ============================
ERROR test_substrait.py::test_udf_via_substrait[True]
ERROR test_substrait.py::test_udf_via_substrait[False]
ERROR test_substrait.py::test_scalar_aggregate_udf_basic
ERROR test_substrait.py::test_hash_aggregate_udf_basic
ERROR test_udf.py::test_udf_array_unary
ERROR test_udf.py::test_scalar_udf_context
ERROR test_udf.py::test_scalar_input
ERROR test_udf.py::test_input_lifetime
ERROR test_udf.py::test_scalar_agg_basic
ERROR test_udf.py::test_scalar_agg_empty
ERROR test_udf.py::test_scalar_agg_varargs
ERROR test_udf.py::test_hash_agg_basic
ERROR test_udf.py::test_hash_agg_empty

It seems to be something about fixture discovery, in particular unary_func_fixture is not being found anymore.

More details

==================================== ERRORS ====================================
________________ ERROR at setup of test_udf_via_substrait[True] ________________
file $PREFIX/lib/python3.8/site-packages/pyarrow/tests/test_substrait.py, line 325
  @pytest.mark.parametrize("use_threads", [True, False])
  def test_udf_via_substrait(unary_func_fixture, use_threads):
E       fixture 'unary_func_fixture' not found
>       available fixtures: base_datadir, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, disable_aws_metadata, doctest_namespace, gcs_server, hdfs_connection, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, s3_connection, s3_server, tempdir, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

$PREFIX/lib/python3.8/site-packages/pyarrow/tests/test_substrait.py:325
_______________ ERROR at setup of test_udf_via_substrait[False] ________________
file $PREFIX/lib/python3.8/site-packages/pyarrow/tests/test_substrait.py, line 325
  @pytest.mark.parametrize("use_threads", [True, False])
  def test_udf_via_substrait(unary_func_fixture, use_threads):
E       fixture 'unary_func_fixture' not found
>       available fixtures: base_datadir, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, disable_aws_metadata, doctest_namespace, gcs_server, hdfs_connection, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, s3_connection, s3_server, tempdir, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

$PREFIX/lib/python3.8/site-packages/pyarrow/tests/test_substrait.py:325
______________ ERROR at setup of test_scalar_aggregate_udf_basic _______________
file $PREFIX/lib/python3.8/site-packages/pyarrow/tests/test_substrait.py, line 610
  def test_scalar_aggregate_udf_basic(varargs_agg_func_fixture):
E       fixture 'varargs_agg_func_fixture' not found
>       available fixtures: base_datadir, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, disable_aws_metadata, doctest_namespace, gcs_server, hdfs_connection, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, s3_connection, s3_server, tempdir, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

$PREFIX/lib/python3.8/site-packages/pyarrow/tests/test_substrait.py:610
_______________ ERROR at setup of test_hash_aggregate_udf_basic ________________
file $PREFIX/lib/python3.8/site-packages/pyarrow/tests/test_substrait.py, line 758
  def test_hash_aggregate_udf_basic(varargs_agg_func_fixture):
E       fixture 'varargs_agg_func_fixture' not found
>       available fixtures: base_datadir, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, disable_aws_metadata, doctest_namespace, gcs_server, hdfs_connection, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, s3_connection, s3_server, tempdir, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

$PREFIX/lib/python3.8/site-packages/pyarrow/tests/test_substrait.py:758
____________________ ERROR at setup of test_udf_array_unary ____________________
file $PREFIX/lib/python3.8/site-packages/pyarrow/tests/test_udf.py, line 334
  def test_udf_array_unary(unary_func_fixture):
E       fixture 'unary_func_fixture' not found
>       available fixtures: base_datadir, binary_func_fixture, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, disable_aws_metadata, doctest_namespace, exception_agg_func_fixture, gcs_server, hdfs_connection, monkeypatch, nullary_func_fixture, pytestconfig, raising_func_fixture, record_property, record_testsuite_property, record_xml_attribute, recwarn, s3_connection, s3_server, sum_agg_func_fixture, tempdir, ternary_func_fixture, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, varargs_func_fixture, wrong_output_datatype_func_fixture, wrong_output_dtype_agg_func_fixture, wrong_output_type_agg_func_fixture, wrong_output_type_func_fixture, wrong_signature_func_fixture
>       use 'pytest --fixtures [testpath]' for help on them.

$PREFIX/lib/python3.8/site-packages/pyarrow/tests/test_udf.py:334
__________________ ERROR at setup of test_scalar_udf_context ___________________
file $PREFIX/lib/python3.8/site-packages/pyarrow/tests/test_udf.py, line 532
  def test_scalar_udf_context(unary_func_fixture):
E       fixture 'unary_func_fixture' not found
>       available fixtures: base_datadir, binary_func_fixture, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, disable_aws_metadata, doctest_namespace, exception_agg_func_fixture, gcs_server, hdfs_connection, monkeypatch, nullary_func_fixture, pytestconfig, raising_func_fixture, record_property, record_testsuite_property, record_xml_attribute, recwarn, s3_connection, s3_server, sum_agg_func_fixture, tempdir, ternary_func_fixture, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, varargs_func_fixture, wrong_output_datatype_func_fixture, wrong_output_dtype_agg_func_fixture, wrong_output_type_agg_func_fixture, wrong_output_type_func_fixture, wrong_signature_func_fixture
>       use 'pytest --fixtures [testpath]' for help on them.

$PREFIX/lib/python3.8/site-packages/pyarrow/tests/test_udf.py:532
_____________________ ERROR at setup of test_scalar_input ______________________
file $PREFIX/lib/python3.8/site-packages/pyarrow/tests/test_udf.py, line 553
  def test_scalar_input(unary_func_fixture):
E       fixture 'unary_func_fixture' not found
>       available fixtures: base_datadir, binary_func_fixture, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, disable_aws_metadata, doctest_namespace, exception_agg_func_fixture, gcs_server, hdfs_connection, monkeypatch, nullary_func_fixture, pytestconfig, raising_func_fixture, record_property, record_testsuite_property, record_xml_attribute, recwarn, s3_connection, s3_server, sum_agg_func_fixture, tempdir, ternary_func_fixture, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, varargs_func_fixture, wrong_output_datatype_func_fixture, wrong_output_dtype_agg_func_fixture, wrong_output_type_agg_func_fixture, wrong_output_type_func_fixture, wrong_signature_func_fixture
>       use 'pytest --fixtures [testpath]' for help on them.

$PREFIX/lib/python3.8/site-packages/pyarrow/tests/test_udf.py:553
____________________ ERROR at setup of test_input_lifetime _____________________
file $PREFIX/lib/python3.8/site-packages/pyarrow/tests/test_udf.py, line 559
  def test_input_lifetime(unary_func_fixture):
E       fixture 'unary_func_fixture' not found
>       available fixtures: base_datadir, binary_func_fixture, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, disable_aws_metadata, doctest_namespace, exception_agg_func_fixture, gcs_server, hdfs_connection, monkeypatch, nullary_func_fixture, pytestconfig, raising_func_fixture, record_property, record_testsuite_property, record_xml_attribute, recwarn, s3_connection, s3_server, sum_agg_func_fixture, tempdir, ternary_func_fixture, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, varargs_func_fixture, wrong_output_datatype_func_fixture, wrong_output_dtype_agg_func_fixture, wrong_output_type_agg_func_fixture, wrong_output_type_func_fixture, wrong_signature_func_fixture
>       use 'pytest --fixtures [testpath]' for help on them.

$PREFIX/lib/python3.8/site-packages/pyarrow/tests/test_udf.py:559
___________________ ERROR at setup of test_scalar_agg_basic ____________________
file $PREFIX/lib/python3.8/site-packages/pyarrow/tests/test_udf.py, line 683
  def test_scalar_agg_basic(unary_agg_func_fixture):
E       fixture 'unary_agg_func_fixture' not found
>       available fixtures: base_datadir, binary_func_fixture, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, disable_aws_metadata, doctest_namespace, exception_agg_func_fixture, gcs_server, hdfs_connection, monkeypatch, nullary_func_fixture, pytestconfig, raising_func_fixture, record_property, record_testsuite_property, record_xml_attribute, recwarn, s3_connection, s3_server, sum_agg_func_fixture, tempdir, ternary_func_fixture, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, varargs_func_fixture, wrong_output_datatype_func_fixture, wrong_output_dtype_agg_func_fixture, wrong_output_type_agg_func_fixture, wrong_output_type_func_fixture, wrong_signature_func_fixture
>       use 'pytest --fixtures [testpath]' for help on them.

$PREFIX/lib/python3.8/site-packages/pyarrow/tests/test_udf.py:683
___________________ ERROR at setup of test_scalar_agg_empty ____________________
file $PREFIX/lib/python3.8/site-packages/pyarrow/tests/test_udf.py, line 690
  def test_scalar_agg_empty(unary_agg_func_fixture):
E       fixture 'unary_agg_func_fixture' not found
>       available fixtures: base_datadir, binary_func_fixture, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, disable_aws_metadata, doctest_namespace, exception_agg_func_fixture, gcs_server, hdfs_connection, monkeypatch, nullary_func_fixture, pytestconfig, raising_func_fixture, record_property, record_testsuite_property, record_xml_attribute, recwarn, s3_connection, s3_server, sum_agg_func_fixture, tempdir, ternary_func_fixture, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, varargs_func_fixture, wrong_output_datatype_func_fixture, wrong_output_dtype_agg_func_fixture, wrong_output_type_agg_func_fixture, wrong_output_type_func_fixture, wrong_signature_func_fixture
>       use 'pytest --fixtures [testpath]' for help on them.

$PREFIX/lib/python3.8/site-packages/pyarrow/tests/test_udf.py:690
__________________ ERROR at setup of test_scalar_agg_varargs ___________________
file $PREFIX/lib/python3.8/site-packages/pyarrow/tests/test_udf.py, line 709
  def test_scalar_agg_varargs(varargs_agg_func_fixture):
E       fixture 'varargs_agg_func_fixture' not found
>       available fixtures: base_datadir, binary_func_fixture, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, disable_aws_metadata, doctest_namespace, exception_agg_func_fixture, gcs_server, hdfs_connection, monkeypatch, nullary_func_fixture, pytestconfig, raising_func_fixture, record_property, record_testsuite_property, record_xml_attribute, recwarn, s3_connection, s3_server, sum_agg_func_fixture, tempdir, ternary_func_fixture, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, varargs_func_fixture, wrong_output_datatype_func_fixture, wrong_output_dtype_agg_func_fixture, wrong_output_type_agg_func_fixture, wrong_output_type_func_fixture, wrong_signature_func_fixture
>       use 'pytest --fixtures [testpath]' for help on them.

$PREFIX/lib/python3.8/site-packages/pyarrow/tests/test_udf.py:709
____________________ ERROR at setup of test_hash_agg_basic _____________________
file $PREFIX/lib/python3.8/site-packages/pyarrow/tests/test_udf.py, line 727
  def test_hash_agg_basic(unary_agg_func_fixture):
E       fixture 'unary_agg_func_fixture' not found
>       available fixtures: base_datadir, binary_func_fixture, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, disable_aws_metadata, doctest_namespace, exception_agg_func_fixture, gcs_server, hdfs_connection, monkeypatch, nullary_func_fixture, pytestconfig, raising_func_fixture, record_property, record_testsuite_property, record_xml_attribute, recwarn, s3_connection, s3_server, sum_agg_func_fixture, tempdir, ternary_func_fixture, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, varargs_func_fixture, wrong_output_datatype_func_fixture, wrong_output_dtype_agg_func_fixture, wrong_output_type_agg_func_fixture, wrong_output_type_func_fixture, wrong_signature_func_fixture
>       use 'pytest --fixtures [testpath]' for help on them.

$PREFIX/lib/python3.8/site-packages/pyarrow/tests/test_udf.py:727
____________________ ERROR at setup of test_hash_agg_empty _____________________
file $PREFIX/lib/python3.8/site-packages/pyarrow/tests/test_udf.py, line 745
  def test_hash_agg_empty(unary_agg_func_fixture):
E       fixture 'unary_agg_func_fixture' not found
>       available fixtures: base_datadir, binary_func_fixture, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, disable_aws_metadata, doctest_namespace, exception_agg_func_fixture, gcs_server, hdfs_connection, monkeypatch, nullary_func_fixture, pytestconfig, raising_func_fixture, record_property, record_testsuite_property, record_xml_attribute, recwarn, s3_connection, s3_server, sum_agg_func_fixture, tempdir, ternary_func_fixture, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, varargs_func_fixture, wrong_output_datatype_func_fixture, wrong_output_dtype_agg_func_fixture, wrong_output_type_agg_func_fixture, wrong_output_type_func_fixture, wrong_signature_func_fixture
>       use 'pytest --fixtures [testpath]' for help on them.

$PREFIX/lib/python3.8/site-packages/pyarrow/tests/test_udf.py:745

Component(s)

Python

[pitrou]

@danepitkin @AlenkaF

[AlenkaF]

I am not able to reproduce the errors locally with Python 3.10 and pytest 7.4.
The last nightly build (6h ago) with Python 3.8 and pytest 7.4. is successful (some tests are skipped, but most of the tests with the fixture in test_udf.py passed: https://github.com/ursacomputing/crossbow/actions/runs/5994083247/job/16255251709#step:6:5897

I see the error is only triggered for the three fixtures unary_func_fixture, unary_agg_func_fixture and varargs_agg_func_fixture all defined in parent pyarrow/conftest.py with session scope (as opposed to other fixtures defined in pyarrow/tests/conftest.py). That should not be an issue as pytest looks for fixtures in parent directories also, but we could try to change?

[h-vetinari]

This had something to do with the way the recipe was executing the tests (from within $SP_DIR/pyarrow/tests, which didn't find the conftest.py one layer further out; pytest 7.4 just got stricter about erroring on missing fixtures it seems). I changed the invocation now so the pytest cap could be removed again.

Sorry for the hassle.