[Issue, ML.net CLI] 330GB csv file of data cause a OutOfMemoryException (1/2)

[wil70]

System Information (please complete the following information):

• OS & Version: Win8, latest version as of this bug entry
• ML.NET Version: 16.13.9
• .NET Version:6.0.303

Describe the bug
When I start ML.net from CLI, I get a OutOfMemoryException
I have 64GB Ram, I have a 330GB csv file of data.

I tried with
To Reproduce
Steps to reproduce the behavior:

1. Generate a 330GB file with 4209 columns with random data
2. open prompt
3. type in command line:
   mlnet classification --train-time 75600 --name SampleClassification --log-file-path c:\Log_data.txt --has-header true --label-col 4209 --ignore-cols 0,1,4206,4207,4208 --dataset "c:\data.csv" --test-dataset "c:\test_data.csv"
4. See error log at the end of this message with the OutOfMemoryException

Expected behavior
I expect ml.net to continue and feed the data as it stream it, so there should be no OutOfMemoryException
When I monitor the mknet.exe prices with task manager, the mlnet.exe process doesn't go high at all, like less than ~14GB. So something is not right as I have 64GB and also it shouldn't matter isn't it as .

Screenshots, Code, Sample Projects
Additional context
Here is the log
Start Training
start nni training
Experiment output folder: C:\Users\W\AppData\Local\Temp\AutoML-NNI\Experiment-GET3JS
System.FormatException: Parsing failed with an exception: Stream reading encountered exception
---> System.FormatException: Stream reading encountered exception
---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Text.StringBuilder.ToString()
at System.IO.StreamReader.ReadLine()
at Microsoft.ML.Data.TextLoader.Cursor.LineReader.ThreadProc()
--- End of inner exception stack trace ---
at Microsoft.ML.Data.TextLoader.Cursor.LineReader.GetBatch()
at Microsoft.ML.Data.TextLoader.Cursor.ParallelState.Parse(Int32 tid)
at Microsoft.ML.Data.TextLoader.Cursor.ParallelState.ThreadProc(Object obj)
--- End of inner exception stack trace ---
at Microsoft.ML.Data.TextLoader.Cursor.ParseParallel(ParallelState state)+MoveNext()
at Microsoft.ML.Data.TextLoader.Cursor.MoveNextCore()
at Microsoft.ML.Data.RootCursorBase.MoveNext()
at Microsoft.ML.ModelBuilder.AutoMLService.Proposer.Controller.CountRows(IDataView data, Int64 maxRows) in //src/Microsoft.ML.ModelBuilder.AutoMLService/Proposer/Controller.cs:line 174
at Microsoft.ML.ModelBuilder.AutoMLService.Proposer.Controller.Initialize() in //src/Microsoft.ML.ModelBuilder.AutoMLService/Proposer/Controller.cs:line 111
at Microsoft.ML.ModelBuilder.AutoMLService.Experiments.LocalAutoMLExperiment.ExecuteAsync(IDataView trainData, IDataView validateData, ColumnInformation columnInformation, CancellationToken cancellationToken, CancellationToken timeout) in //src/Microsoft.ML.ModelBuilder.AutoMLService/Experiments/LocalAutoMLExperiment.cs:line 138
at Microsoft.ML.ModelBuilder.AutoMLEngine.StartTrainingAsync(TrainingConfiguration config, PathConfiguration pathConfig, CancellationToken userCancellationToken) in //src/Microsoft.ML.ModelBuilder.AutoMLService/AutoMLEngineService/AutoMLEngine.cs:line 160
at Microsoft.ML.CLI.Runners.AutoMLRunner.ExecuteAsync() in //src/mlnet/Runners/AutoMLRunner.cs:line 88
at Microsoft.ML.CLI.Program.TrainAsync(TrainingConfiguration trainingConfiguration, PathConfiguration pathConfig, AutoMLServiceLogLevel logLevel) in //src/mlnet/Program.cs:line 348
at Microsoft.ML.CLI.Program.AutoMLCommandRunner(AutoMLCommand command, Boolean skipGenerateConsoleApp) in //src/mlnet/Program.cs:line 329
at Microsoft.ML.CLI.Program.<>c.<b__4_0>d.MoveNext() in //src/mlnet/Program.cs:line 89
--- End of stack trace from previous location ---
at System.CommandLine.Invocation.CommandHandler.GetExitCodeAsync(Object value, InvocationContext context)
at System.CommandLine.Invocation.ModelBindingCommandHandler.InvokeAsync(InvocationContext context)
at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass23_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.ML.CLI.Program.<>c__DisplayClass4_0.<b__9>d.MoveNext() in /_/src/mlnet/Program.cs:line 290
--- End of stack trace from previous location ---
at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<b__24_0>d.MoveNext()
--- End of stack trace from previous location ---
at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass22_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass11_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<b__10_0>d.MoveNext()
--- End of stack trace from previous location ---
at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass14_0.<b__0>d.MoveNext()
Check out log file for more information: c:\Log_data.txt
Exiting ...

C:\Users\W>'

---

[LittleLittleCloud]

Looks like the OOM happens even before training start
@michaelgsharp is CountRows using stream data or it load IDataview in memory

[wil70]

I also tried with MLnet --version: 16.13.9+3066041f1761935763bdd267a9bcbb64cb3ec543
and I got the same Exception as above

---

Sometimes, I get a different OutOfMemoryException stack trace:

Start Training
start nni training
Experiment output folder: C:\Users\W\AppData\Local\Temp\AutoML-NNI\Experiment-GNJIOK
|     Trainer                              MicroAccuracy  MacroAccuracy  Duration #Iteration                     |
System.InvalidOperationException: Event we were waiting on was subject to an exception
 ---> System.FormatException: Parsing failed with an exception: Stream reading encountered exception
 ---> System.FormatException: Stream reading encountered exception
 ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Text.StringBuilder.ToString()
   at System.IO.StreamReader.ReadLine()
   at Microsoft.ML.Data.TextLoader.Cursor.LineReader.ThreadProc()
   --- End of inner exception stack trace ---
   at Microsoft.ML.Data.TextLoader.Cursor.LineReader.GetBatch()
   at Microsoft.ML.Data.TextLoader.Cursor.ParallelState.Parse(Int32 tid)
   at Microsoft.ML.Data.TextLoader.Cursor.ParallelState.ThreadProc(Object obj)
   --- End of inner exception stack trace ---
   at Microsoft.ML.Data.TextLoader.Cursor.ParseParallel(ParallelState state)+MoveNext()
   at Microsoft.ML.Data.TextLoader.Cursor.MoveNextCore()
   at Microsoft.ML.Data.RootCursorBase.MoveNext()
   at Microsoft.ML.Data.CacheDataView.Filler(DataViewRowCursor cursor, ColumnCache[] caches, OrderedWaiter waiter)
   --- End of inner exception stack trace ---
   at Microsoft.ML.Internal.Utilities.OrderedWaiter.Wait(Int64 position, CancellationToken token)
   at Microsoft.ML.Data.CacheDataView.GetPermutationOrNull(Random rand)
   at Microsoft.ML.Data.CacheDataView.GetRowCursorWaiterCore[TWaiter](TWaiter waiter, Func`2 predicate, Random rand)
   at Microsoft.ML.Data.CacheDataView.GetRowCursor(IEnumerable`1 columnsNeeded, Random rand)
   at Microsoft.ML.Transforms.RowShufflingTransformer.GetRowCursorCore(IEnumerable`1 columnsNeeded, Random rand)
   at Microsoft.ML.Data.TransformBase.GetRowCursor(IEnumerable`1 columnsNeeded, Random rand)
   at Microsoft.ML.Transforms.GenerateNumberTransform.GetRowCursorSet(IEnumerable`1 columnsNeeded, Int32 n, Random rand)
   at Microsoft.ML.Transforms.RangeFilter.GetRowCursorSet(IEnumerable`1 columnsNeeded, Int32 n, Random rand)
   at Microsoft.ML.Data.DataViewUtils.TryCreateConsolidatingCursor(DataViewRowCursor& curs, IDataView view, IEnumerable`1 columnsNeeded, IHost host, Random rand)
   at Microsoft.ML.Data.TransformBase.GetRowCursor(IEnumerable`1 columnsNeeded, Random rand)
   at Microsoft.ML.Transforms.ColumnSelectingTransformer.SelectColumnsDataTransform.GetRowCursor(IEnumerable`1 columnsNeeded, Random rand)
   at Microsoft.ML.Data.IO.BinarySaver.RowsPerBlockHeuristic(IDataView data, ColumnCodec[] actives)
   at Microsoft.ML.Data.IO.BinarySaver.SaveData(Stream stream, IDataView data, Int32[] colIndices)
   at Microsoft.ML.Data.DataSaverUtils.SaveDataView(IChannel ch, IDataSaver saver, IDataView view, Stream stream, Boolean keepHidden)
   at Microsoft.ML.BinaryLoaderSaverCatalog.SaveAsBinary(DataOperationsCatalog catalog, IDataView data, Stream stream, Boolean keepHidden)
   at Microsoft.ML.ModelBuilder.AutoMLService.Proposer.ResampleStrategyProposer.Initialize(String outputFolder)
   at Microsoft.ML.ModelBuilder.AutoMLService.Proposer.ResampleStrategyProposer.Propose(TrialParameter trialParameter)
   at Microsoft.ML.ModelBuilder.AutoMLService.Proposer.Controller.Propose()
   at Microsoft.ML.ModelBuilder.AutoMLService.Experiments.LocalAutoMLExperiment.ExecuteAsync(IDataView trainData, IDataView validateData, ColumnInformation columnInformation, CancellationToken cancellationToken, CancellationToken timeout)
   at Microsoft.ML.ModelBuilder.AutoMLEngine.StartTrainingAsync(TrainingConfiguration config, PathConfiguration pathConfig, CancellationToken userCancellationToken)
   at Microsoft.ML.CLI.Runners.AutoMLRunner.ExecuteAsync() in /_/src/mlnet/Runners/AutoMLRunner.cs:line 88
   at Microsoft.ML.CLI.Program.TrainAsync(TrainingConfiguration trainingConfiguration, PathConfiguration pathConfig, AutoMLServiceLogLevel logLevel) in /_/src/mlnet/Program.cs:line 348
   at Microsoft.ML.CLI.Program.AutoMLCommandRunner(AutoMLCommand command, Boolean skipGenerateConsoleApp) in /_/src/mlnet/Program.cs:line 329
   at Microsoft.ML.CLI.Program.<>c.<<CreateRootCommandLineBuilder>b__4_0>d.MoveNext() in /_/src/mlnet/Program.cs:line 89
--- End of stack trace from previous location ---
   at System.CommandLine.Invocation.CommandHandler.GetExitCodeAsync(Object value, InvocationContext context)
   at System.CommandLine.Invocation.ModelBindingCommandHandler.InvokeAsync(InvocationContext context)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<<BuildInvocationChain>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass23_0.<<UseParseErrorReporting>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.ML.CLI.Program.<>c__DisplayClass4_0.<<CreateRootCommandLineBuilder>b__9>d.MoveNext() in /_/src/mlnet/Program.cs:line 290
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseSuggestDirective>b__24_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass22_0.<<UseParseDirective>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass11_0.<<UseDebugDirective>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<RegisterWithDotnetSuggest>b__10_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass14_0.<<UseExceptionHandler>b__0>d.MoveNext()
Check out log file for more information: G:\log_data.txt
Exiting ...

[wil70]

Related issue on c# instead of CLI
#6297

[wil70]

Any ETA for a fix? TY!

[tarekgh]

@wil70 what kind of data do you have inside the csv file? does it contain strings with length greater than 2G? In .NET, the string length is limited to 2G (int.MaxValue).

[wil70]

Hi @tarekgh ,
All the fields are "double" except 1 of the fields is a "string" which has 3 characters maximum
ie.
DOO,-0.85748191341578595,-1.0763921379331161,-3,-1,...
UUU,-1.153315959380012,-1.2316112856397838,0,1,...
...
I hope this help
TY!

[tarekgh]

Thanks @wil70. In this case it looks to me you are running into legitimate OutOfMemoryException https://docs.microsoft.com/en-us/dotnet/standard/garbage-collection/performance#issue-an-out-of-memory-exception-is-thrown. I suggest you try to check the GC state when getting this exception. either through a profiler or a debugger or using the GC APIs https://docs.microsoft.com/en-us/dotnet/api/system.gcmemoryinfo?view=net-6.0.