[BUG] Optimizers do not obey fit_intercept = False, or arg is ambiguous

[Jacob-Stevens-Haas]

Problem

See coefficient plots in example notebooks 1 and 5, which both show the coefficient matrix $\xi$, fit to an equation of the form:

$$ \begin{align*} \dot x = \xi_0 + \xi_1f_1(x) + \dots + \xi_n f_n(x) \end{align*} $$

Expected vs Actual

As I understood it, if fit_intercept=False (default) $\xi_0$ should always be zero. But it clearly isn't in those examples.

T/S

the fit_intercept parameter is used to calculate offsets in BaseOptimizer.fit(). Those results are used later in the function to explicitly self._set_intercept(). Might also be some interaction with normalize_columns.

Nevermind, this is because even if optimizer fit_intercept is false, PolynomialLibrary defaults to providing a zeroeth order term unless include_bias is false (true by default). WeakPDELibrary and SINDyPILibrary also has an inclue_bias term because they're doing more than just evaluating functions, but maybe that's another piece of evidence that they should be subclasses of SINDy, not BaseFeatureLibrary.

It's also unclear how these options interact - i.e. if both are true, does the model try to fit two intercepts? I'm planting a flag that we ought to remove one or both of these options as part of the breaking changes I discussed last week, slightly biased towards removing include_bias.

[znicolaou]

Just adding two cents here, I think that it may have been a mistake sometime early in development to include the include_bias and fit_intercept options.

The fit_intercept option is part of the sklearn.LinearRegression while include_bias is part of sklearn.preprocessing.PolynomialFeatures. The BaseFeatureLibrary inherits only sklearn.base.TransformerMixin, which has neither. I'm guessing the early development tried different sklearn inheritance structures before deciding on the current one. Since SINDy is not constrained to linear or polynomial features, it's a little strange to distinguish the constant feature in general, especially since it can be added multiple times on accident easily.

Most of the sklearn integration seems like unnecessary bloat to me, but I suppose it may be useful to some people to integrate with other parts of sklearn. I'd favor trying to eliminate the arguments and try to think more carefully about whether some libraries should inherit from sklearn. Maybe we could have a general **kwargs to pass to the super classes if needed in specific cases.

[Jacob-Stevens-Haas]

So what's your preference - do we deal with the coefficient term in the regression, and prohibit constant terms in feature libraries in order to prevent people from including it multiple times? Or do we get rid of fit_intercept and treat the constant function like any other function in a library?

[akaptano]

Adding my random opinions on the various pull requests now: I agree with Zach that the sklearn integration is not only unnecessary bloat, but actively harms future development. For instance, we have been thinking for a while of adding a multi-step prediction error functionality, but this requires using Jax, and somehow squaring all of this with the sklearn integration. Not sure how much effort it would be to minimize the sklearn dependency here. I agree that the two ways of specifying a constant term is not good and I'm happy however we deal with it.

[Jacob-Stevens-Haas]

I've settled on removing the argument from optimizers. It's a much harder task to remove it from all function libraries, and I believe that's where people will expect it (e.g. 0th order polynomials). It incurs a minor performance hit in some cases (because scikitlearn's LinearRegression calculates it as the average of the output data) that disappears with scale (Because one additional regression column means $\mathcal O(f(n+1)) = \mathcal O(f(n))$, probably )

That seems a reasonable sacrifice in order to prevent the tragic 😭 case of fitting both an intercept and a constant term, which could only otherwise be achieved with higher coupling and more tests.