Relax restrictions on model type in resampling (evaluate!
)
#985
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Current behaviour
Currently a model in
evaluate
/evaluate!
must subtypeSupervised
orAnnotator
. However, in the current implementation, all that's essential for the model is:predict
methodfit
method consumes a targety
(which must be provided in theevaluate
call, ormachine
constructor).operation
is left unspecified,MLJModelnterface.prediction_type
must be one of::deterministic
,:probabilistic
,:interval
.Subtypes of
Annotator
andSupervised
(e.g,Probabilistic
,Deterministic
,Interval
) have the obviousprediction_type
fallbacks, but the fallback forUnsupervised
models is:unknown
.What this PR does
This PR:
<:Model
is now allowed.y
has been provided.prediction_type
is not one of:deterministic
,:probabilistic
,:interval
, the error which is already thrown in that case is made more informative. It now suggests two remedies: (i) explicitly specifyoperation
(s), or (ii) post an issue to have the model'sprediction_type
reviewed.Context
More and more we are relying on traits and less on the abstract model hierarchy. Currently, however, only
Unsupervised
models can have theirtransform
method propagated in aPipeline
(andpredict
is propagated bySupervised
models). So some models want to beUnsupervised
for pipeline use, despite the fact they are also "supervised" - a case in point isRecursiveFeatureElimination
(currentlySupervised
and so not pipelinable).To do
Just to be sure, let's wait for
Afterword
We should ultimately like to evaluate models that do not consume a target in training, such as clustering models. In that case, the target is only supplied for the test sets (e.g., for computing the Rand index). However, that requires a more substantial redesign of
evaluate
.