Continue training on OOM error && add subsampling support for trainValidationDatasetManager #6714
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
LittleLittleCloud
changed the title
Closed
|
/azp run |
|
Azure Pipelines successfully started running 2 pipeline(s). |
| @@ -12,34 +15,83 @@ public interface IDatasetManager | |||
| { | |||
| } | |||
|
|
|||
| internal interface ICrossValidateDatasetManager | |||
| public interface ICrossValidateDatasetManager | |||
There was a problem hiding this comment.
Can you add some documentation for these since they are now public?
| } | ||
|
|
||
| internal class CrossValidateDatasetManager : IDatasetManager, ICrossValidateDatasetManager | ||
| { | ||
| public IDataView Dataset { get; set; } | ||
| public IDataView? Dataset { get; set; } |
There was a problem hiding this comment.
Do you need this to be a nullable type? Can an IDataView directly have null as a value that you can just check?
There was a problem hiding this comment.
This should not be nullable. I'll make the change
michaelgsharp
approved these changes
Jun 12, 2023
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## main #6714 +/- ##
==========================================
+ Coverage 68.84% 68.88% +0.04%
==========================================
Files 1215 1216 +1
Lines 250691 250890 +199
Branches 26256 26258 +2
==========================================
+ Hits 172580 172823 +243
+ Misses 71286 71245 -41
+ Partials 6825 6822 -3
Flags with carried forward coverage won't be shown. Click here to find out more.
|
This was referenced Jun 14, 2023
ghost
locked as resolved and limited conversation to collaborators
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We are excited to review your PR.
So we can do the best job, please check:
Fixes #nnnnin your description to cause GitHub to automatically close the issue(s) when your PR is merged.Training on a sub-set of train dataset will help mitigate
OOMerror.This will be helpful if the training dataset is huge, in which case subsampling won't hurt metric a lot.
Because this feature is mostly useful for massive training dataset, crossValidationDatasetManger won't need it so the subsampling strategy is only added to trainValidationDatasetManager.
fix #dotnet/machinelearning-modelbuilder#2645