Failed hypothesis health check

[simonetgordon]

Hi, I have encountered an hypothesis heath check error for the __ifloordiv__ and other similar arithmetic operator methods in test_iop when using tensorflow backend functions:

____________ test_iop[__ifloordiv__(x1_i is -0 and x2_i > 0) -> -0] ____________
iop_name = '__ifloordiv__', iop = <built-in function ifloordiv>
case = BinaryCase(<x1_i is -0 and x2_i > 0 -> -0>)
    @pytest.mark.parametrize("iop_name, iop, case", iop_params)
>   @given(
        oneway_dtypes=oneway_promotable_dtypes(dh.float_dtypes),
        oneway_shapes=oneway_broadcastable_shapes(),
        data=st.data(),
    )
E   hypothesis.errors.FailedHealthCheck: It looks like your strategy is filtering out a lot of data. Health check found 50 filtered examples but only 0 good ones. This will make your tests much slower, and also will probably distort the data generation quite a lot. You should adapt your strategy to filter less. This can also be caused by a low max_leaves parameter in recursive() calls
E   See https://hypothesis.readthedocs.io/en/latest/healthchecks.html for more information about this. If you want to disable just this health check, add HealthCheck.filter_too_much to the suppress_health_check settings for this test.
ivy/ivy_tests/test_array_api/array_api_tests/test_special_cases.py:1283: FailedHealthCheck

What is the best way to handle this?

[asmeurer]

A health check error generally indicates a bug in the test suite. It means we are filtering too much in one of the strategies.

[simonetgordon]

In that case, should I skip the test or disable the health check for now?

[honno]

A health check error generally indicates a bug in the test suite. It means we are filtering too much in one of the strategies.

Yep, although when it comes to ivy we have to make sure there's not an error on their end. We'd want to try running the extensive test suite for hypothesis.extra.array_api for ivy's PyTorch namespace. I had done something similiar before for ivy's JAX namespace #115 (comment).

In that case, should I skip the test or disable the health check for now?

If you can figure using Hypothesis' internal suite as mentioned above, I'd try and explore that. But yeah otherwise I'd skip for now if you're running this on CI—I'd disable the health check only when exploring things locally.

[asmeurer]

To be sure, this particular health check error means that the test isn't actually being run at all, so you'd effectively need to treat this test as an XFAIL.

[simonetgordon]

Is there a way to skip one of the special case functions for a particular method? e.g. skip test_binary for add? (also getting the heath check message for this)

[honno]

Depends how, but yes. Specifying the test case externally (e.g. how we specify skips in CI), you'll need the unique test case id e.g. array_api_tests/test_special_cases.py::test_iop[__ifloordiv__(x1_i is -0 and x2_i > 0) -> -0].

[simonetgordon]

I'm attempting to implement a similar set of functions to here down in the ivy conftest.py file. In principal, should this work? (currently not having any luck). Or is it necessary to have the skips.txt file within the array-api submodule directory itself as only the array api conftest.py can skip tests?

[honno]

I'm attempting to implement a similar set of functions to here down in the ivy conftest.py file. In principal, should this work? (currently not having any luck). Or is it necessary to have the skips.txt file within the array-api submodule directory itself as only the array api conftest.py can skip tests?

Yeah, in principal... but specifying skips/xfails is a bit annoying to do with pytest, outside of them being literally coded in the test heh.

Or is it necessary to have the skips.txt file within the array-api submodule directory itself as only the array api conftest.py can skip tests?

I think you could have skips.txt at the top level of this repo (not the array-api submodule) for it to be registered with the sibling conftest.py file.

[simonetgordon]

Cheers, I think I have it working now!

[asmeurer]

I'm getting a lot of health check errors testing against the NumPy compat library https://github.com/data-apis/numpy-array-api-compat, specifically for floordiv and mod operators. Plain NumPy without the compat library fails because np.asarray doesn't have the copy keyword. Is that a requirement for the iop tests to work?

Maybe we should add a plain NumPy build to the CI to check for test suite errors like this. Also, if you run the tests against the development NumPy with NPY_PROMOTION_STATE=weak the type promotion tests all pass, so the number of test failures is much lower.

[honno]

Plain NumPy without the compat library fails because np.asarray doesn't have the copy keyword. Is that a requirement for the iop tests to work?

Yep, on the in-place branch copy is used, as IIRC it's necessary to actually test the elements of resulting arrays.

array-api-tests/array_api_tests/test_operators_and_elementwise_functions.py

Lines 410 to 415 in 8ad6b62

	def func(l: Array, r: Union[Scalar, Array]) -> Array:
	locals_ = {}
	locals_[left_sym] = xp.asarray(l, copy=True) # prevents mutating l
	locals_[right_sym] = r
	exec(expr, locals_)
	return locals_[left_sym]