[Issue, ML.net C#] 330GB csv file of data cause a OutOfMemoryException (2/2)

[wil70]

System Information (please complete the following information):

• OS & Version: Win8, latest version as of this bug entry
• ML.NET Version: 16.13.9
• .NET Version:6.0.303

Describe the bug
When I start c# AutoML in c# I get a OutOfMemoryException after the memory reach the maximum of 64GB.
I have 64GB Ram, I have a 330GB csv file of data.

Note: I couldn't do it with ML.net CLI due to this bug #6288 so I tried to do it with c# AutoML package. I'm totally new at ML.NET, sorry in adavance for the code quality

To Reproduce
Steps to reproduce the behavior:

1. Generate a 330GB file with 4209 columns with random data
2. create a c# project and paste the code below
3. See error log at the end of this message with the OutOfMemoryException

Expected behavior
I expect to be able to be able to handle 2TB files and 100K columns without any issue with ML.Net CLI and also with c# on a 64GB ram computer by streaming the data instead of loading all in memeory.

Screenshots, Code, Sample Projects
If applicable, add screenshots, code snippets, or sample projects to help explain your problem.

Additional context
Add any other context about the problem here.

I have a 330gb file (64 gb ram). I tried ML.NET CLI but hit a bug see.
So I'm now trying with c#, the bug is different than the ML.NET CLI issue as it seems to try to load everything in memory

IDataView trainingData = mlContext.Data.LoadFromTextFile(
"c:\data.csv",
separatorChar: ',', hasHeader: true, trimWhitespace: true);

        var cts = new CancellationTokenSource();
        var experimentSettings = new MulticlassExperimentSettings();
        //experimentSettings.TrainingData = trainingData;
        experimentSettings.MaxExperimentTimeInSeconds = 3600;
        experimentSettings.CancellationToken = cts.Token;
        experimentSettings.CacheBeforeTrainer = CacheBeforeTrainer.Auto;

        // Cancel experiment after the user presses any key
        //CancelExperimentAfterAnyKeyPress(cts);
        experimentSettings.CacheDirectoryName = null;

        MulticlassClassificationExperiment experiment = mlContext.Auto().CreateMulticlassClassificationExperiment(experimentSettings);
        ExperimentResult<MulticlassClassificationMetrics> experimentResult = experiment.Execute(trainingData, "Entry(Text)");//, progressHandler: progressHandler);

.....
public class ModelInput
{
[LoadColumn(0), NoColumn]
public string _data0 { get; set; }

        [LoadColumn(1), NoColumn]
        public float ignoreData1 { get; set; }

        [LoadColumn(2, 4205)]
        public float _data { get; set; }

        [LoadColumn(4206),NoColumn]//(4206,4208)]
        public float _ignoreData4206 { get; set; }         
	[LoadColumn(4207), NoColumn]//(4206,4208)]
        public float _ignoreData4207 { get; set; }
        [LoadColumn(4208), NoColumn]//(4206,4208)]
        public float _ignoreData4208 { get; set; }

        [LoadColumn(4209),ColumnName("Entry(Text)")]
        public string _label { get; set; }
    }

---

There is a Exception of type 'System.OutOfMemoryException' was thrown.
(new System.Collections.Generic.Mscorlib_CollectionDebugView<Microsoft.ML.AutoML.RunDetail<Microsoft.ML.Data.MulticlassClassificationMetrics>>(experimentResult.RunDetails).Items[0]).Exception.StackTrace
at Microsoft.ML.Internal.Utilities.OrderedWaiter.Wait(Int64 position, CancellationToken token)
at Microsoft.ML.Data.CacheDataView.GetPermutationOrNull(Random rand)
at Microsoft.ML.Data.CacheDataView.GetRowCursorSetWaiterCore[TWaiter](TWaiter waiter, Func2 predicate, Int32 n, Random rand) at Microsoft.ML.Data.CacheDataView.GetRowCursorSet(IEnumerable1 columnsNeeded, Int32 n, Random rand)
at Microsoft.ML.Data.OneToOneTransformBase.GetRowCursorSet(IEnumerable1 columnsNeeded, Int32 n, Random rand) at Microsoft.ML.Data.DataViewUtils.TryCreateConsolidatingCursor(DataViewRowCursor& curs, IDataView view, IEnumerable1 columnsNeeded, IHost host, Random rand)
at Microsoft.ML.Data.TransformBase.GetRowCursor(IEnumerable1 columnsNeeded, Random rand) at Microsoft.ML.Trainers.TrainingCursorBase.FactoryBase1.Create(Random rand, Int32[] extraCols)
at Microsoft.ML.Trainers.OnlineLinearTrainer2.TrainCore(IChannel ch, RoleMappedData data, TrainStateBase state) at Microsoft.ML.Trainers.OnlineLinearTrainer2.TrainModelCore(TrainContext context)
at Microsoft.ML.Trainers.TrainerEstimatorBase2.TrainTransformer(IDataView trainSet, IDataView validationSet, IPredictor initPredictor) at Microsoft.ML.Trainers.OneVersusAllTrainer.TrainOne(IChannel ch, ITrainerEstimator2 trainer, RoleMappedData data, Int32 cls)
at Microsoft.ML.Trainers.OneVersusAllTrainer.Fit(IDataView input)
at Microsoft.ML.Data.EstimatorChain1.Fit(IDataView input) at Microsoft.ML.Data.EstimatorChain1.Fit(IDataView input)
at Microsoft.ML.AutoML.RunnerUtil.TrainAndScorePipeline[TMetrics](MLContext context, SuggestedPipeline pipeline, IDataView trainData, IDataView validData, String groupId, String labelColumn, IMetricsAgent`1 metricsAgent, ITransformer preprocessorTransform, FileInfo modelFileInfo, DataViewSchema modelInputSchema, IChannel logger)

---

There is a Exception of type 'System.OutOfMemoryException' was thrown.
(new System.Collections.Generic.Mscorlib_CollectionDebugView<Microsoft.ML.AutoML.RunDetail<Microsoft.ML.Data.MulticlassClassificationMetrics>>(experimentResult.RunDetails).Items[0]).Exception.InnerException.StackTrace
at Microsoft.ML.Internal.Utilities.ArrayUtils.EnsureSize[T](T[]& array, Int32 min, Int32 max, Boolean keepOld, Boolean& resized)
at Microsoft.ML.Internal.Utilities.BigArray1.AddRange(ReadOnlySpan1 src)
at Microsoft.ML.Data.CacheDataView.ColumnCache.ImplVec`1.CacheCurrent()
at Microsoft.ML.Data.CacheDataView.Filler(DataViewRowCursor cursor, ColumnCache[] caches, OrderedWaiter waiter)

[luisquintanilla]

This looks like the old version of AutoML. @wil70 can you please try to use the latest preview version of AutoML.

https://www.nuget.org/packages/Microsoft.ML.AutoML/0.20.0-preview.22313.1

Here are some samples that use the new API.

https://github.com/dotnet/csharp-notebooks/blob/main/machine-learning/03-Training%20and%20AutoML.ipynb

https://github.com/dotnet/csharp-notebooks/blob/main/machine-learning/04-Model%20Evaluation.ipynb

cc: @LittleLittleCloud

[LittleLittleCloud]

@wil70 let us know if @luisquintanilla comments resolve your issue, and a tip:

• you can disable LightGbm trainer as it uses a lot of memory when training data goes up

This issue has been marked needs-author-action and may be missing some important information.

[wil70]

Hello,

Sorry for the delay, it takes time to reproduce the issue and get the error message with big files.

1. Every trainer completed fine with 16GB csv input file. I can start AutoML and run all the algo
   1.1. For the same entry csv file, I observed a strange thing where some algo have a confusion Matrix that look like this

    	ConfusionMatrix.PerClassPrecision: Results from All trainers with the exception of FastForestOva and FastTreeOva
    	- class 1: 952, 0, 0,
    	- class 2: 0, 971, 0,
    	- class 3: 0, 0, 805,
    
    	While other are like this
    	ConfusionMatrix.PerClassPrecision: Results from trainers FastForestOva and FastTreeOva
    	- class 1: 4487, 0, 0,
    	- class 2: 0, 4429, 0,
    	- class 3: 0, 0, 3667,
    
    1.1.a. This is a big difference in the confusion Matrix. There are 125,766 lines in the input files (no duplicates), so the algos are ignoring a lot of rows, some much more than others.
    1.1.b. I have some NaN here and there, I'm wondering if some of the trainer do not like those.
    What shall I use instead of NaN? 
    	1.1.b.1. For Data none available, ie. NaN?
    	1.1.b.2. For Infinity? I can use double.Max
    	1.1.b.3. For -Infinity?
   

2. For files of 330GB there are a few issues
   2.1. I needed to run the Algo 1 by 1, as many crash and can't complete
   2.2 LightGbm is the only one that completed - Yeah!

    	LightGbm
    	Processing LightGbm
    	BestRun TrainerName:        LightGbmMulti
    		- MicroAccuracy:        1
    		- MacroAccuracy:        1
    		- LogLoss:              3.589831285873676E-05
    		- CLogLossReduction:    0.9999669040026492
    	
    	Class log loss:
    		- class 1: 4.208176463648898E-05
    		- class 2: 3.385601134085665E-05
    		- class 3: 3.3677191368241515E-05
    	
    	ConfusionMatrix.PerClassPrecision:
    	class 1: 1
    	class 2: 1
    	class 3: 1
    	
    	ConfusionMatrix.PerClassPrecision:
    		- class 1: 145805, 0, 0,
    		- class 2: 0, 211129, 0,
    		- class 3: 0, 0, 211780,
    	Saving model!!!
    	Done saving...
   
    	Duration: 04:25:05.6115268
   
            2.2.a. Looking at the confusion matrix, I can see it is missing a lot of rows, this is probably the same as in 1.1.b (probably NaN)
   

   2.3 All the other crash but didn't really give some stack trace except for

    	Processing LbfgsMaximumEntropy
    	Exception of type 'System.OutOfMemoryException' was thrown.
    	at System.Linq.GroupedEnumerable`2.GetEnumerator()
    	at System.Linq.Enumerable.SelectEnumerableIterator`2.ToArray()
    	at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
    	at System.Linq.OrderedEnumerable`1.ToArray()
    	at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
    	at System.Linq.Enumerable.ReverseIterator`1.MoveNext()
    	at System.Linq.Enumerable.EnumerablePartition`1.MoveNext()
    	at System.Linq.Enumerable.SelectIPartitionIterator`2.MoveNext()
    	at System.Collections.Generic.HashSet`1.UnionWith(IEnumerable`1 other)
    	at System.Collections.Generic.HashSet`1..ctor(IEnumerable`1 collection, IEqualityComparer`1 comparer)
    	at System.Collections.Generic.HashSet`1..ctor(IEnumerable`1 collection)
    	at Microsoft.ML.AutoML.PipelineSuggester.OrderTrainersByNumTrials(IEnumerable`1 history, IEnumerable`1 selectedTrainers)
    	at Microsoft.ML.AutoML.PipelineSuggester.GetNextInferredPipeline(MLContext context, IEnumerable`1 history, DatasetColumnInfo[] columns, TaskKind task, Boolean isMaximizingMetric, CacheBeforeTrainer cacheBeforeTrainer, IChannel logger, IEnumerable`1 trainerAllowList)
    	at Microsoft.ML.AutoML.Experiment`2.Execute()
    	at Microsoft.ML.AutoML.ExperimentBase`2.Execute(ColumnInformation columnInfo, DatasetColumnInfo[] columns, IEstimator`1 preFeaturizer, IProgress`1 progressHandler, IRunner`1 runner)
    	at Microsoft.ML.AutoML.ExperimentBase`2.ExecuteTrainValidate(IDataView trainData, ColumnInformation columnInfo, IDataView validationData, IEstimator`1 preFeaturizer, IProgress`1 progressHandler)
    	at Microsoft.ML.AutoML.ExperimentBase`2.Execute(IDataView trainData, ColumnInformation columnInformation, IEstimator`1 preFeaturizer, IProgress`1 progressHandler)
    	at Microsoft.ML.AutoML.ExperimentBase`2.Execute(IDataView trainData, String labelColumnName, String samplingKeyColumn, IEstimator`1 preFeaturizer, IProgress`1 progressHandler)
    	at ML1.Program.ML4(MulticlassClassificationTrainer trainerID) in G:\Users\Wilhelm\dev\MachineLearning\ML2\Program.cs:line 76
    	
    	Duration: 01:41:49.7475104
   

   2.4. I tried to catch more details error for all the other aglo in debug mode via Visual Studio 2022 but that doesn't go well, VS crash etc...
   I also tried with "debug only my code" but same...

3. I would love to be able to run on 2TB csv files with more than 20,000+ columns/attributes.

4. It seems the CLI is having some memory issue too - see: [Issue, ML.net CLI] 330GB csv file of data cause a OutOfMemoryException (1/2) #6288

Thanks

Wilhelm

cc: @dakersnar @LittleLittleCloud @luisquintanilla @michaelgsharp

[luisquintanilla]

Thanks for that detailed report @wil70.

@LittleLittleCloud can you please take a look.

[wil70]

More or less related issues:

• [ML.Net c#, CLI, VS Builder] 1KB csv input file. Not sure what to do + NoColumn how-to #6309
• [Issue, ML.net C#] 330GB csv file of data cause a OutOfMemoryException (2/2) #6297
• [ML.Net c#, CLI, VS Builder] 1KB csv input file. Not sure what to do + NoColumn how-to #6309