MinTrace's protection to Schafer-Strimmer covariance, eliminated stat…

[kdgutier]

Solved eigenvalue indetermination in the presence of small percentage of zero predictions.

• Added protections for single series with constant values / std=0.
• Added protections to standardize residuals when std=0.
• Final epsilon diagonal for improved numerical stability.
• Zero residuals in MinTrace warning.
• Added stress test unit test for percentage of zero residuals scenarios in core.ipynb
• Added single insample value MinTrace-shr covariance stress test in methods.ipynb (ridge diagonal solved this problem).

I took out the statsmodels, the only function we used from it was cov2corr that is particularly simple and does not merit such a big dependency.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[AzulGarza]

I have left some comments. As we are not testing the nbs/examples files automatically, please just check that we are able to recover the previous outputs/replications. I'll work on an automatic test to help future development.

[AzulGarza]

Also, if this PR is going close #92, please let's add a test where it is verified that before the changes, the same error arises, and now with these changes, it is solved.

See, for example, this PR.

A user had an error using HierarchicalEvaluation when h=1, then a test was included showing that it works. In particular, these lines,

This particular case is a little bit tricky because we don't have the actual data. I'd start trying to reproduce the error with a small example. Just to be sure that these changes are actually solving the problem.

[AzulGarza]

Also, if this PR is going close #92, please let's add a test where it is verified that before the changes, the same error arises, and now with these changes, it is solved.

See, for example, this PR.

A user had an error using HierarchicalEvaluation when h=1, then a test was included showing that it works. In particular, these lines,

This particular case is a little bit tricky because we don't have the actual data. I'd start trying to reproduce the error with a small example. Just to be sure that these changes are actually solving the problem.

Just some thoughts on the test:

An element of the diagonal of the variance-covariance matrix would be zero if a time series has only one value, so to reproduce the error, we could use the objects used for testing purposes in nbs/methods.ipynb as follows,

diff_len_y_insample = S @ y_bottom
diff_len_y_hat_insample = S @ y_hat_bottom_insample
diff_len_y_insample[-1, :-1] = np.nan
diff_len_y_hat_insample[-1, :-1] = np.nan
print(diff_len_y_insample)
cls_min_trace = MinTrace(method='mint_shrink')
cls_min_trace(
    S=S, 
    y_hat=S @ y_hat_bottom, 
    y_insample=diff_len_y_insample,
    y_hat_insample=diff_len_y_hat_insample,
    idx_bottom=idx_bottom
)

The output of the previous code is,

This small example recovers the #92 bug. Observe that if we use ols, there is no problem (as expected),

The user's data has time series with a single value.