ENH: Enable accelerated linear regression for overdetermined systems

[david-cortes-intel]

Description

As a follow-up from oneDAL PR: uxlfoundation/oneDAL#2930

The PR above enables fitting linear regressions to overdetermined systems, both in regular regression and multi-output regression. Currently, the sklearnex estimators fall back to scikit-learn for overdetermined systems:
https://github.com/intel/scikit-learn-intelex/blob/2fccf444f800fb8ae1cce844b35727efc623e6d3/sklearnex/linear_model/linear.py#L180

But after the PR above, those should be supported in oneDAL too. Hence, this PR enables using oneDAL for those situations and adds tests for them.

One thing to note though: the mechanism in oneDAL is implemented as a fallback from Cholesky - that is, it will first perform a Cholesky factorization, and if that fails, will then use the fallback solver which can handle overdetermined systems. But if e.g. there are more columns that rows, we'll already know beforehand that Cholesky will invariably fail, and the procedure could be made faster if such kind of solver could be requested straight away from oneDAL. Implementation of it as a selectable solver is left for future PRs.

Note that since the oneDAL PR hasn't been merged at the time of writing, this PR will of course end up failing tests, but will keep it here in the meantime.

---

Checklist to comply with before moving PR from draft:

PR completeness and readability

• I have reviewed my changes thoroughly before submitting this pull request.
• I have commented my code, particularly in hard-to-understand areas.
• I have updated the documentation to reflect the changes or created a separate PR with update and provided its number in the description, if necessary.
• Git commit message contains an appropriate signed-off-by string (see CONTRIBUTING.md for details).
• I have added a respective label(s) to PR if I have a permission for that.
• I have resolved any merge conflicts that might occur with the base branch.

Testing

• I have run it locally and tested the changes extensively.
• All CI jobs are green or I have provided justification why they aren't.
• I have extended testing suite if new functionality was introduced in this PR.

Performance

• I have measured performance for affected algorithms using scikit-learn_bench and provided at least summary table with measured data, if performance change is expected.
• I have provided justification why performance has changed or why changes are not expected.
• I have provided justification why quality metrics have changed or why changes are not expected.
• I have extended benchmarking suite and provided corresponding scikit-learn_bench PR if new measurable functionality was introduced in this PR.

[icfaust]

Due to the nature of using two repos together, you'll unfortunately need to if statement out the new functionality with a daal_check_version https://github.com/intel/scikit-learn-intelex/blob/main/daal4py/sklearn/_utils.py#L69 Once the onedal PR is merged, we can force run the nightly and then the GitHub Actions CI will reflect the change in oneDAL linear Regression, but the Azure Pipelines will have to wait to a minor release (i.e. 2025.1). This should solve the segfaulting in at least the azure pipelines-side of CI (and again apologies as this is undocumented convention).

[david-cortes-intel]

Due to the nature of using two repos together, you'll unfortunately need to if statement out the new functionality with a daal_check_version https://github.com/intel/scikit-learn-intelex/blob/main/daal4py/sklearn/_utils.py#L69 Once the onedal PR is merged, we can force run the nightly and then the GitHub Actions CI will reflect the change in oneDAL linear Regression, but the Azure Pipelines will have to wait to a minor release (i.e. 2025.1). This should solve the segfaulting in at least the azure pipelines-side of CI (and again apologies as this is undocumented convention).

Thanks, added a check for the oneDAL version that would enable features and tests accordingly.

[david-cortes-intel]

PR adding the functionality in oneDAL was merged, so this PR should start working by tomorrow as the builds finish I guess.

[david-cortes-intel]

CI jobs are passing now.

[david-cortes-intel]

/intelci: run

[david-cortes-intel]

/intelci: run

[david-cortes-intel]

/intelci: run

[david-cortes-intel]

/intelci: run

[Vika-F]

The changes look good to me, there are only couple of minor suggestions.

[Alexsandruss]

/intelci: run

[Alexsandruss]

PreCommit has few repeated failures of LinearRegression. Also, need to rule out all other failures as sporadic.

[david-cortes-intel]

/intelci: run

[david-cortes-intel]

/intelci: run

[david-cortes-intel]

/intelci: run

[david-cortes-intel]

/intelci: run

[david-cortes-intel]

/intelci: run

[david-cortes-intel]

/intelci: run

[david-cortes-intel]

/intelci: run

[david-cortes-intel]

All CI jobs are passing now.

Waiting for some comment from @syakov-intel or @napetrov on this comment from Ian about whether to disable multiple regression in cases where it can segfault or leave it as it is.

[icfaust]

@DDJHB please include these style changes into ridge out of preview, should enable some deselected tests.

[Alexsandruss]

Patching logging got broken due to incorrect names.

[Alexsandruss]

/intelci: run

[Alexsandruss]

http://intel-ci.intel.com/ef96e180-8fcb-f125-9c69-a4bf010d0e2d