matrix_power issues

[mattbarrett98]

Hi, I'm working with Ivy where we are wrapping various ML frameworks and trying to align them with the Array API standard https://github.com/unifyai/ivy. The first problem I'm having is slight numerical issues. For example, when using TensorFlow I have gotten the error:

x = ivy.array([[2.5418657e+12, 2.5418657e+12, 2.5418657e+12],
           [2.5418657e+12, 2.5418657e+12, 2.5418657e+12],
           [2.5418657e+12, 2.5418657e+12, 2.5418657e+12]], dtype=float32)
y = ivy.array([[2.541866e+12, 2.541866e+12, 2.541866e+12],
           [2.541866e+12, 2.541866e+12, 2.541866e+12],
           [2.541866e+12, 2.541866e+12, 2.541866e+12]], dtype=float32)
    
>       assert all(exactly_equal(x, y)), "The input arrays have different values"
E       AssertionError: The input arrays have different values

For PyTorch, I have gotten the error:

x = ivy.array([[0., 0.],
           [0., 0.]], dtype=float32)
y = ivy.array([[ 0.,  0.],
           [ 0., -0.]], dtype=float32)

>       assert all(exactly_equal(x, y)), "The input arrays have different values"
E       AssertionError: The input arrays have different values

These types of failures don't appear each time I run the tests, only appearing every few runs. I wonder if something similar to NumPy.allclose would be better for value checking, as matrix_power seems like a function susceptible to these very small errors.

Also, when testing against JAX I get a failure caused by the following falsifying example:

Falsifying example: test_matrix_power(
    x=ivy.array([[[1., 1.],
                [1., 1.]]], dtype=float32), n=-1,
)

which is strange given that here x isn't invertible and n is negative, which it says in test_matrix_power shouldn't occur.

[asmeurer]

The exactly equal thing is a known problem which I plan to fix soon. See #44. The matrix_power generating a non-invertible matrix for n=-1 is a bug. It's supposed to avoid doing that, so I'll need to take a look as to why that is happening.

[asmeurer]

I'm not able to reproduce the matrix_power n=-1 issue. The code should only be generating invertible matrices when n < 0. Can you run the test with -k matrix_power -s --hypothesis-verbosity=verbose and paste the output somewhere (@honno do you have any other suggestions on how to get figure out what is going on here?).

[asmeurer]

I should add that the NumPy implementation raises an exception for matrix_power(asarray([[[1., 1.], [1., 1.]]]), -1). So if the test can generate this example then NumPy should be failing as well, but I'm not able to get it to fail, even after running thousands of examples.

[asmeurer]

I'll also just add to make sure you are using the latest versions of both the array_api_tests and hypothesis.

[honno]

I can have a look at this later, just to clarify this example

For PyTorch, I have gotten the error:

x = ivy.array([[0., 0.],
           [0., 0.]], dtype=float32)
y = ivy.array([[ 0.,  0.],
           [ 0., -0.]], dtype=float32)

>       assert all(exactly_equal(x, y)), "The input arrays have different values"
E       AssertionError: The input arrays have different values

It's failing because the 4th zero is positive in x and negative in y. For many Array API functions the sign of a zero shouldn't matter... but there are cases where we do care! We don't do this in test_matrix_power though do we? I don't think we even use y as a variable name anywhere—did someone make a modification for an internal test suite that ivy uses?

[asmeurer]

The first two failures are coming from testing exact equality in stacking (I think, @mattbarrett98 didn't show the full stack trace), which is something we do want to test differently. A difference between positive and negative zero can be something that is affected by the numerical details of a specific algorithm. I don't know what the actual inputs were here (again, @mattbarrett98 it would be helpful to provide that detail), but there aren't any cases in linalg where such things are specified.

[mattbarrett98]

Thanks I think this should resolve the first 2 issues! As for the +-0, in what situations do we care about them being different and why?

For the invertible matrix issue, yes we test against numpy also and this has never had this issue. I wonder if this is because in numpy it throws an exception when trying to invert the given matrix. However jax.numpy actually gives you a result when asking it to invert that same non-invertible matrix?

[honno]

As for the +-0, in what situations do we care about them being different and why?

From the top of my head: manipulation functions because those shouldn't change the actual elements of an array, and some elementwise functions because of special cases.

[djl11]

This is a stack trace for the non-invertible matrix error:

=================================== FAILURES ===================================
______________________________ test_matrix_power _______________________________

    @pytest.mark.xp_extension('linalg')
>   @given(
        # Generate any square matrix if n >= 0 but only invertible matrices if n < 0
        x=matrix_power_n.flatmap(lambda n: invertible_matrices() if n < 0 else
                                 xps.arrays(dtype=xps.floating_dtypes(),
                                            shape=square_matrix_shapes)),
        n=matrix_power_n,
    )

ivy/ivy_tests/test_array_api/array_api_tests/test_linalg.py:339: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
ivy/ivy_tests/test_array_api/array_api_tests/test_linalg.py:358: in test_matrix_power
    _test_stacks(func, x, res=res, true_val=true_val)
ivy/ivy_tests/test_array_api/array_api_tests/test_linalg.py:81: in _test_stacks
    assert_equal(res_stack, decomp_res_stack)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

x = ivy.array([[ 2., -1.],
           [-1.,  1.]], dtype=float32)
y = ivy.array([[ inf, -inf],
           [-inf,  inf]], dtype=float32)

    def assert_exactly_equal(x, y):
        """
        Test that the arrays x and y are exactly equal.
    
        If x and y do not have the same shape and dtype, they are not considered
        equal.
    
        """
        assert x.shape == y.shape, f"The input arrays do not have the same shapes ({x.shape} != {y.shape})"
    
        assert x.dtype == y.dtype, f"The input arrays do not have the same dtype ({x.dtype} != {y.dtype})"
    
>       assert all(exactly_equal(x, y)), "The input arrays have different values"
E       AssertionError: The input arrays have different values

ivy/ivy_tests/test_array_api/array_api_tests/array_helpers.py:179: AssertionError
---------------------------------- Hypothesis ----------------------------------
Falsifying example: test_matrix_power(
    x=ivy.array([[[1., 1.],
                [1., 1.]]], dtype=float32), n=-1,
)

This is the commit where the test failed. At the time, we were testing against this commit.

Perhaps the commits since 28th March have fixed this?

[asmeurer]

My guess is it has to do with the fact that it's Jax, which works differently from the other array libraries due to it being graph based. I don't think we've used Jax much in the test suite so far, so it's not surprising if there are some issues.

Either way, I'll take a look to see if I can reproduce the issue.

[asmeurer]

I was able to reproduce the matrix_power issue. I'm still not clear what is going on, but it appears to be fixed by #101.

[asmeurer]

I believe this should be fixed. Let me know if you still see these failures.