Relax restrictions on model type in resampling (evaluate!)
#985
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Current behaviour
Currently a model in
evaluate/evaluate!must subtypeSupervisedorAnnotator. However, in the current implementation, all that's essential for the model is:predictmethodfitmethod consumes a targety(which must be provided in theevaluatecall, ormachineconstructor).operationis left unspecified,MLJModelnterface.prediction_typemust be one of::deterministic,:probabilistic,:interval.Subtypes of
AnnotatorandSupervised(e.g,Probabilistic,Deterministic,Interval) have the obviousprediction_typefallbacks, but the fallback forUnsupervisedmodels is:unknown.What this PR does
This PR:
<:Modelis now allowed.yhas been provided.prediction_typeis not one of:deterministic,:probabilistic,:interval, the error which is already thrown in that case is made more informative. It now suggests two remedies: (i) explicitly specifyoperation(s), or (ii) post an issue to have the model'sprediction_typereviewed.Context
More and more we are relying on traits and less on the abstract model hierarchy. Currently, however, only
Unsupervisedmodels can have theirtransformmethod propagated in aPipeline(andpredictis propagated bySupervisedmodels). So some models want to beUnsupervisedfor pipeline use, despite the fact they are also "supervised" - a case in point isRecursiveFeatureElimination(currentlySupervisedand so not pipelinable).To do
Just to be sure, let's wait for
Afterword
We should ultimately like to evaluate models that do not consume a target in training, such as clustering models. In that case, the target is only supplied for the test sets (e.g., for computing the Rand index). However, that requires a more substantial redesign of
evaluate.