Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ML.net cli] Resume a cli training after a crash #6286

Open
wil70 opened this issue Aug 17, 2022 · 1 comment
Open

[ML.net cli] Resume a cli training after a crash #6286

wil70 opened this issue Aug 17, 2022 · 1 comment
Labels
enhancement New feature or request

Comments

@wil70
Copy link

wil70 commented Aug 17, 2022

Hello, is there a way to resume an ML.net cli training to where it was before a crash?
I have a lot of data in the folder C:\Users\wwww\AppData\Local\Temp\AutoML-NNI\Experiment-9K67B4 but I do not know how to make mlnet start from there.

Detail:
I used the cli, ie "mlnet classicfiaction...."
I trained for a few days, but I made a mistake which used a lot of memory on my computer, which stopped the mlnet process. I would like to start mlnet to where it left so it can continue from there

Thanks
w

2022-08-10 15:03:24.3091 DEBUG System.InvalidOperationException: Event we were waiting on was subject to an exception
---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Array.Resize[T](T[]& array, Int32 newSize)
at Microsoft.ML.Internal.Utilities.ArrayUtils.EnsureSize[T](T[]& array, Int32 min, Int32 max, Boolean keepOld, Boolean& resized)
at Microsoft.ML.Data.CacheDataView.ColumnCache.ImplOne`1.CacheCurrent()
at Microsoft.ML.Data.CacheDataView.Filler(DataViewRowCursor cursor, ColumnCache[] caches, OrderedWaiter waiter)

@wil70 wil70 added the enhancement New feature or request label Aug 17, 2022
@ghost ghost added the untriaged New issue has not been triaged label Aug 17, 2022
@wil70 wil70 changed the title Resume a cli training ML.net cli -- Resume a cli training Aug 17, 2022
@wil70 wil70 changed the title ML.net cli -- Resume a cli training ML.net cli -- Resume a cli training after a crash Aug 17, 2022
@wil70 wil70 changed the title ML.net cli -- Resume a cli training after a crash [ML.net cli] Resume a cli training after a crash Aug 18, 2022
@dakersnar
Copy link
Contributor

I don't believe this functionality is currently supported.

cc @luisquintanilla for confirmation. Is this something that we are tracking support for in the future?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants