API: run LearnerRanker using single model instead of CrossFit (#317)

* API: implement class ParameterSpace and related classes

* API: add initial version of CrossFit-free LearnerRanker

* API: remove previous LearnerRanker

* API: enable tests for new LearnerRanker

* REFACTOR: remove obsolete function _learner_type()

* REFACTOR: rename LearnerRanker2 to LearnerRanker

* REFACTOR: remove classes LearnerGrid and LearnerEvaluation

* API: introduce CandidateEstimatorDF

* API: add scoring preprocessing

* API: introduce CandidateEstimatorDF name parameter

* API: expect EstimatorDF arguments for BaseParameterSpace

* API: remove MultiRegressorParameterSpace, MultiClassifierParameterSpace

* API: add T_Candidate_co; update (Multi)ParamaterSpace class signatures

* API: add PARAM_CANDIDATE… constants to CandidateEstimatorDF

* API: add documentation for CandidateEstimatorDF attributes

* API: add call to super().__init__()

* API: suppress PEP warnings for 'X' arg name

* API: add property CandidateEstimatorDF._estimator_type

* DOC: tweak API docstring

* API: streamline/improve handling of estimator scoring in LearnerRanker

* TEST: update unit tests for API changes

* API: remove obsolete constants ParamaterSpace.STEP_…

* API: don't use CandidateEstimatorDF in ParameterSpace; add auto-naming

* API: add type validation for LearnerRanker parameter searcher_factory

* TEST: streamline and update unit tests

* TEST: eliminate test_crossfit.py

* API: remove module facet.crossfit

* API: rename LearnerRanker param searcher_factory to searcher_type

* API: eliminate LearnerRanker parameter random_state

* DOC: add docstrings for LearnerRanker attributes

* DOC: update release notes

* FIX: honour arg prefix in MultiEstimatorParameterSpace.get_parameters()

* FIX: improve type hints of class CandidateEstimatorDF

* REFACTOR: tweak approach to get BaseSearchCV class

* REFACTOR: reorder constants for better code legibility

* API: support all estimators in LearnerRanker, not just learner pipelines

* DOC: add comments

* API: remove CandidateEstimatorDF.empty()

* API: rename LearnerRanker to ModelSelector

* API: move CandidateEstimatorDF to facet.selection.base

* DOC: update the pydata sphinx theme to ~=0.7

* TEST: rename _ranker to _selector

Co-authored-by: Mateusz Sokół <mat646@gmail.com>
Co-authored-by: Mateusz Sokół <8431159+mtsokol@users.noreply.github.com>

RELEASE_NOTES.rst

@@ -7,6 +7,19 @@ FACET 2.0
 2.0.0
 ~~~~~
 
+``facet.data``
+^^^^^^^^^^^^^^
+
+- API: class :class:`.RangePartitioner` supports new optional arguments ``lower_bound``
+  and ``upper_bound`` in method :meth:`~.RangePartitioner.fit` and no longer accepts
+  them in the class initializer
+
+``facet.inspection``
+^^^^^^^^^^^^^^^^^^^^
+
+- API: :class:`.LearnerInspector` no longer uses learner crossfits and instead inspects
+  models using a single pass of SHAP calculations, usually leading to performance gains
+  of up to a factor of 50
 - API: return :class:`.LearnerInspector` matrix outputs as :class:`.Matrix` instances
 - API: diagonals of feature synergy, redundancy, and association matrices are now
   ``nan`` instead of 1.0
@@ -15,14 +28,34 @@ FACET 2.0
   row and column order of :class:`.Matrix` objects returned by the corresponding
   ``feature_…_matrix`` methods of :class:`.LearnerInspector`, minimizing the distance
   between adjacent leaves
-  The old sorting behaviour of FACET 1.x can be restored using method
+  The old sorting behaviour of FACET 1 can be restored using method
   :meth:`.LinkageTree.sort_by_weight`
-- API: class :class:`.RangePartitioner` adds optional arguments ``lower_bound`` and
-  ``upper_bound`` to method :meth:`~.RangePartitioner.fit` and no longer accepts them
-  in the class initializer
+
+``facet.selection``
+^^^^^^^^^^^^^^^^^^^
+
+- API: :class:`.ModelSelector` replaces FACET 1 class ``LearnerRanker``, and now
+  supports any CV searcher that supports `scikit-learn`'s CV search API, including
+  `scikit-learn`'s native searchers such as :class:`.GridSearchCV` or
+  :class:`.RandomizedSearchCV`
+- API: new classes :class:`.ParameterSpace` and :class:`MultiParameterSpace` offer an
+  a more convenient and robust mechanism for declaring options or distributions for
+  hyperparameter tuning
+
+``facet.simulation``
+^^^^^^^^^^^^^^^^^^^^
+
+- API: simulations no longer depend on learner crossfits and instead are carried out
+  as a single pass on the full dataset, using the *standard error of mean predictions*
+  to obtain confidence intervals that less conservative yet more realistic
 - VIZ: minor tweaks to simulation plots and reports generated by
   :class:`.SimulationDrawer`
 
+Other
+^^^^^
+
+- API: class ``LearnerCrossfit`` is no longer used in FACET 2 and has been removed
+
 
 FACET 1.2
 ---------
@@ -93,7 +126,7 @@ by the :class:`.LearnerInspector`.
   across matrices as an indication of confidence for each calculated value.
 - API: Method :meth:`.LearnerInspector.shap_plot_data` now returns SHAP values for the
   positive class of binary classifiers.
-- API: Increase efficiency of :class:`.LearnerRanker` parallelization by adopting the
+- API: Increase efficiency of :class:`.ModelSelector` parallelization by adopting the
   new :class:`pytools.parallelization.JobRunner` API provided by :mod:`pytools`
 - BUILD: add support for :mod:`shap` 0.38 and 0.39
 

environment.yml

@@ -27,7 +27,7 @@ dependencies:
   - m2r ~= 0.2
   - pluggy ~= 0.13
   - pre-commit ~= 2.7
-  - pydata-sphinx-theme ~= 0.4.0
+  - pydata-sphinx-theme ~= 0.7
   - pytest ~= 5.2
   - pytest-cov ~= 2.8
   - pyyaml ~= 5.1