Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TST: bump rtol for equal_bounds #14878

Merged
merged 2 commits into from
Oct 19, 2021
Merged

TST: bump rtol for equal_bounds #14878

merged 2 commits into from
Oct 19, 2021

Conversation

andyfaff
Copy link
Contributor

This PR bumps the test tolerance for examining how different minimiser methods cope with equal bounds.
The test examines the output from a finite differences minimisation vs one that uses a (callable) analytic gradient. In the former factorisation is carried out, in the latter it isn't; this is probably enough to generate differences in the output. Indeed, the arm64 nightly wheels on MacPython were failing because of the specified tolerances:

___________ test_equal_bounds[None-constraints0-<lambda>-kwds0-TNC] ____________
[gw1] linux -- Python 3.10.0 /venv/bin/python
/venv/lib/python3.10/site-packages/scipy/optimize/tests/test_optimize.py:2416: in test_equal_bounds
    assert_allclose(res.x, expected.x, rtol=3e-6)
E   AssertionError: 
E   Not equal to tolerance rtol=3e-06, atol=0
E   
E   Mismatched elements: 2 / 4 (50%)
E   Max absolute difference: 0.00034846
E   Max relative difference: 0.00039608
E    x: array([ 1.413984,  2.      ,  0.879424, -1.      ])
E    y: array([ 1.413696,  2.      ,  0.879772, -1.      ])
        bound_type = <function setup_test_equal_bounds.<locals>.<lambda> at 0xffff7a514670>
        bounds     = [(0, 3), (2, 2), (-1, 2), (-1, -1)]
        callback   = None
        constraints = (None, None)
        expected   =      fun: 1289.4752548690583
     jac: array([-1.75951145e-05,             nan,  4.24487110e-05,             nan])
 me...  nfev: 19
     nit: 7
  status: 1
 success: True
       x: array([ 1.41369615,  2.        ,  0.87977221, -1.        ])
        fd_needed  = False
        i_eb       = array([False,  True, False,  True])
        kwds       = {'bounds': [(0, 3), (2, 2), (-1, 2), (-1, -1)], 'callback': None, 'constraints': None, 'fun': <function setup_test_equal_bounds.<locals>.func at 0xffff7a5141f0>, ...}
        lb         = array([ 0,  2, -1, -1])
        method     = 'TNC'
        reference_constraints = None
        res        =      fun: 1289.4754142650613
     jac: array([ 0.46138667,         nan, -0.53325948,         nan])
 message: 'Converge... nfev: 108
     nit: 9
  status: 1
 success: True
       x: array([ 1.41398439,  2.        ,  0.87942374, -1.        ])
        test_constraints = None
        ub         = array([ 3,  2,  2, -1])
        x0         = array([0.5488135 , 0.71518937, 0.60276338, 0.54488318])

I therefore relaxed the tolerances to allow the test to pass. This is quite a big bump, from 3e-6 to 5e-4, a couple of orders of magnitude. This might indicate that improvements are needed for TNC.

I therefore decided to add an extra test that would compare the output of two minimisation run that both use finite differences. The first test has two parameters with equal bounds, with two varying parameters (for a total of 4 params). The output of this is then compared to a minimisation that only has the two varying parameters from the first. I would expect the two runs to be pretty much identical.

@andyfaff
Copy link
Contributor Author

The underlying issue is with TNC, see #14879

@andyfaff andyfaff merged commit 94af3fb into scipy:master Oct 19, 2021
@andyfaff andyfaff deleted the fd branch October 19, 2021 06:22
@tylerjereddy tylerjereddy added this to the 1.8.0 milestone Oct 19, 2021
@tupui
Copy link
Member

tupui commented Oct 19, 2021

@andyfaff we still have some failures, see https://app.travis-ci.com/github/MacPython/scipy-wheels/jobs/544115570

/venv/lib/python3.10/site-packages/scipy/optimize/tests/test_optimize.py:2436: in test_equal_bounds
890    assert_allclose(res.fun, fd_res.fun)
891E   AssertionError: 
892E   Not equal to tolerance rtol=1e-07, atol=0
893E   
894E   Mismatched elements: 1 / 1 (100%)
895E   Max absolute difference: 0.00015936
896E   Max relative difference: 1.23582398e-07
897E    x: array(1289.475255)
898E    y: array(1289.475414)

@andyfaff
Copy link
Contributor Author

@tupui, the location for updates on that should now be #14879.
The fix will be contained in #14882.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants