add pipelineitem for Ova #363

Ivanidzo4ka · 2018-06-14T19:38:44Z

Address #34.
This PR add wrapper on top of OVA to make it pipeline friendly.

Ivanidzo4ka · 2018-06-20T17:27:48Z

Can anyone take a look?

zeahmed · 2018-06-20T19:12:35Z

test/Microsoft.ML.Tests/Scenarios/IrisPlantClassificationTests.cs

+
+        private void CheckMetrics(ClassificationMetrics metrics)
+        {
+            Assert.Equal(.96, metrics.AccuracyMacro);


Assert.Equal(.96, metrics.AccuracyMacro); [](start = 11, length = 42)

It would also be better here to match on fixed decimal places.

zeahmed

sfilipi

* Bump master to v0.3 (#269) * RocketEngine fix for selecting top learners (#270) * Changes to RocketEngine to fix take top k logic. * Add namespace information to allow file to reference correct version of Formatting object. * small code cleanup (#271) * Preparation for syncing sources with internal repo (#275) * make class partial so I can add constuctor in separate file. add constructros for testing * formatting * Changes to use evaluator metrics names in PipelineSweeperSupportedMetrics. Made the private const strings in two classes public. (#276) * add missing subcomponents to sweepers (#278) * add missing subcomponents * right one * more cleanup * remove lotus references. (#252) * Random seed and concurrency for tests (#277) * first attempt * add comments * specify seed for random. make constructor internal. * Fix SupportedMetric.ByName() method (#280) * Fix for SupportedMetric.ByName() method. Include new unit test for function. * Fix for SupportedMetric.ByName() method. Include new unit test for function. * Fix for SupportedMetric.ByName() method. Include new unit test for function. * Removed unnecessary field filter, per review comment. * ML.NET-242: FastTreeRanking per-iteration loss metrics are empty (#289) When training a FastTreeRanker using the `testFrequency` parameter, it is expected that NDCG is prented every testFrequency iterations. However, instead of NDCG, only empty strings are printed. The root cause was that the MaxDCG property of the dataset was never calculated, so the NDCG calculation is aborted, leaving an empty string as a result. This PR fixes the problem by computing the MaxDCG for the dataset when the Tests are defined (so that if the tests are not defined, the MaxDCG will never be calculated). Closes #242 * Fixed typo in the method summary (#296) * Remove stale line of code from test. (#297) * Update release notes link to use aka.ms. (#294) Our release notes link is broken because the `Documentation` was renamed to `docs`. Fix this for the future to use a redirection link. * Add release notes for ML.NET 0.2 (#301) * Add release notes for ML.NET 0.2 * Adding release note about TextLoader changes and additional issue/PR references * Addressing comments: fixing typos, changing formatting, and adding references * Get the cross validation macro to work with non-default column names (#291) * Add label/grou/weight column name arguments to CV and train-test macros * Fix unit test. * Merge. * Update CSharp API. * Fix EntryPointCatalog test. * Address PR comments. * update sample in README.MD with 0.2 features. (#304) * update sample with new text loader API. * update with 0.2 stuff. * OVA should respect normalization in underlying learner (#310) * Respect normalization in OVA. * some cleanup * fix copypaste issues * Export to ONNX and cross-platform command-line tool to script ML.NET training and inference (#248) * Export to ONNX and Maml cross-platform executable. * Add Cluster evaluator (#316) * Add Cluster evaluator * fix copypaste * address comments * formatting * Fixes locale dependent test output comparisons (#109) The tests do not pass on systems with locale other than en-US. The error happens since the results are written to files and the contents of the files are compared to set of correct results produced under en-US locale. The fix is to imbue en-US culture to the test thread so that results will be output in format that is comparable with the test format. This patch fixes only tests, but do not guarantee calculation will be correct in production systems using a locale different than en-US. In particular, there can be problems in reading data and then conversing data from characters to numeric format. Fixes #74 * Add PartitionedFileLoader (#61) * Remove unexisting project from solution (#335) * GetSummaryDataView/Row implementation for Pca and Linear Predictors (#185) * Implement `ICanGetSummaryAsIDataView` on `PcaPredictor` class * Implement `ICanGetSummaryAsIRow` on `LinearPredictor` class * Disable ordinary least squares by removing the entry point (#286) * Disable ols by temporarily removing the entry point. It may be added again once we figure out how to ship MKL as part of this project. * add append function to pipeline (#284) Add `Append` function to pipeline for more fluent API than that allowed by `Add` * Removed field/column name checking of input type in TextLoader. (#327) * fix namespace issue in CSharpGenerator and some refactoring (#339) fix namespace issue and refactoring * Using named-tuple in OneToOneTransforms' constructor to make API more readable. (#324) * Minor formatting in CollectionDataSourceTests.cs (#348) * Create CalibratedPredictor instead of SchemaBindableCalibratedPredictor (#338) `CalibratorUtils.TrainCalibrator` and `TrainCalibratorIfNeeded` now creates `CalibratedPredictor` instead of `SchemaBindableCalibratedPredictor` whenever the predictor implements `IValueMapper`. * Remove reference and dependency on System.ValueTuple (#351) * Add link to samples (#355) * Use HideEnumValueAttribute for both manifest and C# API generation. (#356) * Use HideEnumValueAttribute for both manifest and C# API generation. * Unhide NAReplaceTransform.ReplacementKind.SpecifiedValue. This may require some other PR to resolve the corresponding issues. * Move the NuGet package build files into a TFM specific directory. (#370) When installing Microsoft.ML on an unsupported framework (like net452), it is currently getting installed successfully. However, users should be getting an error stating that net452 is not supported by this package. The cause is the build files exist for any TFM, which NuGet interprets as this package supports any TFM. Moving the build files to be consistent with the 'lib' folder support. Fix #357 * `Stream` subclasses now have `Close` call `base.Close` to ensure disposal. (#369) * Subclasses of `Stream` now have `Close` call `base.Close` to ensure disposal. * Add DeleteOnClose to File opening. * Remove explicit delete of file. * Remove explicit close of substream. * Since no longer deleting explicitly, no longer need `_overflowPath` member. * Return distinct array of ParameterSet when ProposeSweep is called (#368) * Changed List to HashSet to ensure that there are no duplicates * Update fast tree argument help text (#372) * Update fast tree argument help text * Update wording * Update API to fix test * Update core manifest JSON to update help text * Combine multiple tree ensemble models into a single tree ensemble (#364) * Add a way to create a single tree ensemble model from multiple tree ensemble models. * Address PR comments, and fix bugs in serializing/deserializing RegressionTrees. * Address PR comments. * add pipelineitem for Ova (#363) add pipelineitem for Ova * Fix CV macro to output the warnings data view properly. (#385) * Link to an example on using converting ML.NET model to ONNX. (#386) * Adding documentation about entry points, and entry points graphs: EntryPoints.md and GraphRunner.md (#295) * Adding EntryPoints.md and GraphRunner.md * addressing PR feedback * Updating the title of the GraphRunner.md file * adressing Tom's feedback * adressing feedback * code formatting for class names * Addressing Gal's comments * Adding an example of an entry point. Fixing casing on ML.NET * fixing link * Adding LDA Transform (#377) * Revert to using the native code (#413) Corrects an unintentional "typo" in FastTreeRanking.cs where there was mistakenly a USE_FASTTREENATIVE2 instead of USE_FASTTREENATIVE. This resulted in some obscure hidden ranking options (distance weighting, normalize query lambdas, and a few others) being unavailable. These are important for some applications. * LightGBM (#392) * LightGBM and test. * add test baselines and nuget source for lightGBM binaries. * Add entrypoint for lightGBM. * add unsafe flag for release build. * update nuget version. * make lightgbm test single threaded. * install gcc on OS machines to resolve dependencies on openmp thatis needed by lightgbm native code. * PR comments. Leave BREW and GCC in bash script to verify macOS tests work. * remove brew and gcc from build script. * PR feedback. * disable test on macOS. * disable test on macOS. * PR feedback. * Adding Factorization Machines (#383) * Adding Factorization Machines * ONNX API documentation. (#419) * ONNX API documentation. * Bring ensembles into codebase (#379) Introduce Ensemble codebase * enable macOS tests for LightGBM. (#422) * Create a shorter temp file name for model loading. (#397) Create a shorter temp file name for model loading, as well as remove the potential for a race condition among multiple openings by using the creation of a lock file. * removing extraneous character that broke the linux build, and with it unecessary cmake version requirement (#425) * EvaluatorUtils to handle label column of type key without text key values (#394) * Fix EvaluatorUtils to handle label column of type key without text key values. * Removing non source files from solution (#362)

add pipelineitem for Ova

@daholste

…ature branch (#3324) * Initial commit * ci test build * forgot to save this one file * Debug-Intrinsics isn't a valid config, trying windows-x64 * disabled tests for now * disable tests attempt 2 * initial code push, no history, test project not in the build so is the internal client * battling with warn as err * test build * test change * make params for MLContext data extensions match ML.NET default names and values; update gitignore; nit rev for Benchmarking.cs (#5) * Create README.md (#2) * API folder changes (#6) * comment out fast forest trainer, per discussion on ML.NET open issue #1983, for now, to run E2E w/o exceptions (#7) * Make validation data param mandatory; remove GetFirstPipeline sample (#10) * Make validation data param mandatory; remove GetFirstPipeline sample * remove deprecated todo * Create ISSUE_TEMPLATE.md & PULL_REQUEST_TEMPLATE.md (#12) * Create ISSUE_TEMPLATE.md * Create PULL_REQUEST_TEMPLATE.md * NestedObject For pipeline (#14) * add estimator extensions / catalog; add conversion from external to internal pipeline; transform clean-up; add back in test proj and fix build; refactor trainer ext name mappings (#15) * Make validation data param mandatory; remove GetFirstPipeline sample * remove deprecated todo * add estimator extensions / catalog; add ability to go from external to internal pipeline; a lot of transform clean-up; add back in test proj and get it building; refactor trainer ext name mappings * corrected the typo in readme (#16) * make GetNextPipeline API w/ public Pipeline method on PipelineSuggester; write GetNextPipeline API test; fix public Pipeline object serialization; fix header inferencing bug; write test utils for fetching datasets (#18) * get next pipeline API rev -- refactor API to consume column dimensions, purpose, type, and name instead of available trainers & transforms (#19) * mark get next pipeline test as ignore for now (#20) * fix dataview take util bug, add dataview skip util, add some UTs to increase code coverage (#21) * fix dataview take util bug, add dataview skip util, add some UTs to increase code coverage * add accuracy threshold on AutoFit test * add null check to best pipeline on autofit result * unit test additions (including user input validation testing); dead code removal for code coverage (including KDO & associated utils); misc fixes & revs (#22) * add trainer extension tests, & misc fixes (#23) * add estimator extension tests (#24) * add conversions tests (#25) * fix multiclass runs & add multiclass autofit UT (#27) * add basic autofit regression test (#28) * fix categorical transform bug (sometimes categorical features weren't concatenated to final features); add UT transforms; add PipelineNode equality & tests to serve as AutoML testing infra * add example to readme (#26) * add lightgbm args as nested properties (#33) * fix bug where if one pipeline hyperparam optimization converges, run terminates (#36) * add open-source headers to files; other nit clean-ups along the way (#35) * Ungroup Columns in Column Inference (#40) * Added sequential grouping of columns * added ungrouping of column option * reverted the file * Misc fixes (#39) * misc fixes -- fix bug where SMAC returning already-seen values; fix param encoding return bug in pipeline object model; nit clean-up AutoFit; return in pipeline suggester when sweeper has no next proposal; null ref fix in public object model pipeline suggester * fix in BuildPipelineNodePropsLightGbm test, fix / use correct 'newTrainer' variable in PipelneSuggester * SMAC perf improvement * Removing the nuget.config and have build.props mention the nuget package sources. (#38) * Added sequential grouping of columns * removed nuget.config and have only props mentions the nuget sources * reverted the file * transform inferencing concat / ignore fixes (#41) * make pipeline object model & other public classes internal (#43) * handle SMAC exception when fewer trees were trained than requested (#44) * Throw error on incorrect Label name in InferColumns API (#47) * Added sequential grouping of columns * reverted the file * addded infer columns label name checking * added column detection error * removed unsed usings * added quotes * replace Where with Any clause * replace Where with Any clause * Set Nullable Auto params to null values (#50) * Added sequential grouping of columns * reverted the file * added auto params as null * change to the update fields method * First public api propsal (#52) * Includes following 1) Final proposal for 0.1 public API surface 2) Prefeaturization 3) Splitting train data into train and validate when validation data is null 4) Providing end to end samples one each for regression, binaryclassification and multiclass classification * Incorporating code review feedbacks * Revert "Set Nullable Auto params to null values" (#53) * Revert "First public api propsal (#52)" This reverts commit e4a64cf. * Revert "Set Nullable Auto params to null values (#50)" This reverts commit 41c663c. * AutoFit return type is now an IEnumerable (#55) AutoFit returns is now an IEnumerable - this enables many good things Implementing variety of early stopping criteria (See sample) Early discard of models that are no good. This improves memory usage efficiency. (See sample) No need to implement a callback to get results back Getting best score is now outside of API implementation. It is a simple math function to compare scores (See sample). Also templatized the return type for better type safety through out the code. * misc fixes & test additions, towards 0.1 release (#56) * Enable UnitTests on build server (#57) * 1) Making trainer name public (#62) 2) Fixing up samples to reflect it * Initial version of CLI tool for mlnet (#61) * added global tool initial project * removed unneccesary files, renamed files * refactoring and added base abstract classes for trainer generator * removed unused class * Added classes for transforms * added transform generate dummy classes * more refactoring, added first transform * more refactoring and added classes * changed the project structure * restructing added options class * sln changes * refactored options to different class: * added more logic for code generation of class * misc changes * reverted file * added commandline api package * reverted sample * added new command line api parser * added normalization of column names * Added command defaults and error message * implementation of all trainers * changed auto to null * added all transform generators * added error handling when args is empty and minor changes due to change in AutoML api names * changed the name of param * added new command line options and restructuring code * renamed proj file and added solution * Added code to generate usings, Fixed few bugs in the code * added validation to the command line options * changed project name * Bug fixes due to API change in AutoML * changed directory structure * added test framework and basic tests * added more tests * added improvements to template and error handling * renamed the estimator name * fixed test case * added comments * added headers * changed namespace and removed unneccesary properties from project * Revert "changed namespace and removed unneccesary properties from project" This reverts commit 9edae033e9845e910f663f296e168f1182b84f5f. * fixed test cases and renamed namespaces * cleaned up proj file * added folder structure * added symbols/tokens for strings * added more tests * review comments * modified test cases * review comments * change in the exception message * normalized line endings * made method private static * simplified range building /optimization * minor fix * added header * added static methods in command where necessary * nit picks * made few methods static * review comments * nitpick * remove line pragmas * fix test case * Use better AutiFit overload and ignore Multiclass (#64) * Upgrading CLI to produce ML.NET V.10 APIs and bunch of Refactoring tasks (#65) * Added sequential grouping of columns * reverted the file * upgrade to v .10 and refactoring * added null check * fixed unit tests * review comments * removed the settings change * added regions * fixed unit tests * Upgrade ML.NET package to 0.10.0 (#70) * Change in template to accomodate new API of TextLoader (#72) * Added sequential grouping of columns * reverted the file * changed to new API of Text Loader * changed signature * added params for taking additional settings * changes to codegen params * refactoring of templates and fixing errors * Enable gated check for mlnet.tests (#79) * Added sequential grouping of columns * reverted the file * changed to new API of Text Loader * changed signature * added params for taking additional settings * changes to codegen params * refactoring of templates and fixing errors * added run-tests.proj and referred it in build.proj * CLI tool - make validation dataset optional and support for crossvalidation in generated code (#83) * Added sequential grouping of columns * reverted the file * bug fixes, more logic to templates to support cross-validate * formatting and fix type in consolehelper * Added logic in templates * revert settings * benchmarking related changes (#63) * Create test.txt * Create test.txt * changes needed for benchmarking * forgot one file * merge conflict fix * fix build break * back out my version of the fix for Label column issue and fix the original fix * bogus file removal * undo SuggestedPipeline change * remove labelCol from pipeline suggester * fix build break * fix fast forest learner (don't sweep over learning rate) (#88) * Made changes to Have non-calibrated scoring for binary classifiers (#86) * Added sequential grouping of columns * reverted the file * added calibration workaround * removed print probability * reverted settings * rev ColumnInference API: can take label index; rev output object types; add tests (#89) * rename AutoML to Microsoft.ML.Auto everywhere and a shot at publishing nuget package (#99) * Create test.txt * Create test.txt * changes needed for benchmarking * forgot one file * merge conflict fix * fix build break * back out my version of the fix for Label column issue and fix the original fix * bogus file removal * undo SuggestedPipeline change * remove labelCol from pipeline suggester * fix build break * rename AutoML to Microsoft.ML.Auto everywhere and a shot at publishing nuget package (will probably need tweaks once I try to use the pipleline) * publish nuget (#101) * use dotnet-internal-temp agent for internal build * use dotnet-internal feed * Fix Codegen for columnConvert and ValueToKeyMapping transform and add individual transform tests (#95) * Added sequential grouping of columns * reverted the file * fix usings for type convert * added transforms tests * review comments * When generating usings choose only distinct usings directives (#94) * Added sequential grouping of columns * reverted the file * Added code to have unique strings * refactoring * minor fix * minor fix * Autofit overloads + cancellation + progress callbacks 1) Introduce AutoFit overloads (basic and advanced) 2) AutoFit Cancellation 3) AutoFit progress callbacks * Default the kfolds to value 5 in CLI generated code (#115) * Added sequential grouping of columns * reverted the file * Set up CI with Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * remove file * added kfold param and defaulted to value * changed type * added for regression * Remove extra ; from generated code (#114) * Added sequential grouping of columns * reverted the file * Set up CI with Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * removed extra ; from generated code * removed file * fix unit tests * TimeoutInSeconds (#116) Specifying timeout in seconds instead of minutes * Added more command line args implementation to CLI tool and refactoring (#110) * Added sequential grouping of columns * reverted the file * Set up CI with Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * added git status * reverted change * added codegen options and refactoring * minor fixes' * renamed params, minor refactoring * added tests for commandline and refactoring * removed file * added back the test case * minor fixes * Update src/mlnet.Test/CommandLineTests.cs Co-Authored-By: srsaggam <41802116+srsaggam@users.noreply.github.com> * review comments * capitalize the first character * changed the name of test case * remove unused directives * Fail gracefully if unable to instantiate data view with swept parameters (#125) * gracefully fail if fail to parse a datai * rev * validate AutoFit 'Features' column must be of type R4 (#132) * Samples: exceptions / nits (#124) * Logging support in CLI + Implementation of cmd args [--name,--output,--verbosity] (#121) * addded logging and helper methods * fixing code after merge * added resx files, added logger framework, added logging messages * added new options * added spacing * minor fixes * change command description * rename option, add headers, include new param in test * formatted * build fix * changed option name * Added NlogConfig file * added back config package * fix tests * added correct validation check (#137) * Use CreateTextLoader<T>(..) instead of CreateTextLoader(..) (#138) * added support to loaddata by class in the generated code * fix tests * changed CreateTextLoader to ReadFromTextFile method. (#140) * changed textloader to readfromtextfile method * formatting * exception fixes (#136) * infer purpose of hidden columns as 'ignore' (#142) * Added approval tests and bunch of refactoring of code and normalizing namespaces (#148) * changed textloader to readfromtextfile method * formatting * added approval tests and refactoring of code * removed few comments * API 2.0 skeleton (#149) Incorporating API review feedback * The CV code should come before the training when there is no test dataset in generated code (#151) * reorder cv code * build fix * fixed structure * Format the generated code + bunch of misc tasks (#152) * added formatting and minor changes for reordering cv * fixing the template * minor changes * formatting changes * fixed approval test * removed unused nuget * added missing value replacing * added test for new transform * fix test * Update src/mlnet/Templates/Console/MLCodeGen.cs Co-Authored-By: srsaggam <41802116+srsaggam@users.noreply.github.com> * Sanitize the column names in CLI (#162) * added sanitization layer in CLI * fix test * changed exception.StackTrace to exception.ToString() * fix package name (#168) * Rev public API (#163) * Rename TransformGeneratorBase .cs to TransformGeneratorBase.cs (#153) * Fix minor version for the repository + remove Nlog config package (#171) * changed the minor version * removed the nlog config package * Added new test to columninfo and fixing up API (#178) * Make optimizing metric customizable and add trainer whitelist functionality (#172) * API rev (#181) * propagate root MLContext thru AutoML (instead of creating our own) (#182) * Enabling new command line args (#183) * fix package name * initial commit * added more commandline args * fixed tests * added headers * fix tests * fix test * rename 'AutoFitter' to 'Experiment' (#169) * added tests (#187) * rev InferColumns to accept ColumnInfo input param (#186) * Implement argument --has-header and change usage of dataset (#194) * added has header and fixed dataset and train dataset * fix tests * removed dummy command (#195) * Fix bug for regression and sanitize input label from user (#198) * removed dummy command * sanitize label and fix template * fix tests * Do not generate code concatenating columns when the dataset has a single feature column (#191) * Include some missed logging in the generated code. (#199) * added logging messages for generated code * added log messages * deleted file * cleaning up proj files (#185) * removed platform target * removed platform target * Some spaces and extra lines + bug in output path (#204) * nit picks * nit picks * fix test * accept label from user input and provide in generated code (#205) * Rev handling of weight / label columns (#203) * migrate to private ML.NET nuget for latest bug fixes (#131) * fix multiclass with nonstandard label (#207) * Multiclass nondefault label test (#208) * printing escaped chars + bug (#212) * delete unused internal samples (#211) * fix SMAC bug that causes multiclass sample to infinite loop (#209) * Rev user input validation for new API (#210) * added console message for exit and nit picks (#215) * exit when exception encountered (#216) * Seal API classes (and make EnableCaching internal) (#217) * Suggested sample nits (feel free to ask for any of these to be reverted) (#219) * User input column type validation (#218) * upgrade commandline and renaming (#221) * upgrade commandline and renaming * renaming fields * Make build.sh, init-tools.sh, & run.sh executable on OSX/Linux (#225) * CLI argument descriptions updated (#224) * CLI argument descriptions updated * No version in .csproj * added flag to disable training code (#227) * Exit if perfect model produced (#220) * removed header (#228) * removed header * added auto generated header * removed console read key (#229) * Fix model path in generated file (#230) * removed console read key * fix model path * fix test * reorder samples (#231) * remove rule that infers column purpose as categorical if # of distinct values is < 100 (#233) * Null reference exception fix for finding best model when some runs have failed (#239) * samples fixes (#238) * fix for defaulting Averaged Perceptron # of iterations to 10 (#237) * Bug bash feedback Feb 27. API changes and sample changes (#240) * Bug bash feedback Feb 27. API changes Sample changes Exception fix * Samples / API rev from 2/27 bug bash feedback (#242) * changed the directory structure for generated project (#243) * changed the directory structure for generated project * changed test * upgraded commandline package * Fix test file locations on OSX (#235) * fix test file locations on OSX * changing to Path.Combine() * Additional Path.Combine() * Remove ConsoleCodeGeneratorTests.GeneratedTrainCodeTest.received.txt * Additional Path.Combine() * add back in double comparison fix * remove metrics agent NaN returns * test fix * test format fix * mock out path Thanks to @daholste for additional fixes! * upgrade to latest ML.NET public surface (#246) * Upgrade to ML.NET 0.11 (#247) * initial changes * fix lightgbm * changed normalize method * added tests * fix tests * fix test * Private preview final API changes (#250) * .NET framework design guidelines applied to public surface * WhitelistedTrainers -> Trainers * Add estimator to public API iteration result (#248) * LightGBM pipeline serialization fix (#251) * Change order that we search for TextLoader's parameters (#256) * CLI IFileInfo null exception fix (#254) * Averaged Perceptron pipeline serialization fix (#257) * Upgrade command-line-api and default folder name change (#258) * change in defautl folderName * upgrade command line * Update src/mlnet/Program.cs Co-Authored-By: srsaggam <41802116+srsaggam@users.noreply.github.com> * eliminate IFileInfo from CLI (#260) * Rev samples towards private preview; ignored columns fix (#259) * remove unused methods in consolehelper and nit picks in generated code (#261) * nit picks * change in console helper * fix tests * add space * fix tests * added nuget sources in generated csproj (#262) * added nuget sources in csproj * changed the structure in generated code * space * upgrade to mlnet 0.11 (#263) * Formatting CLI metrics (#264) Ensures space between printed metrics (also model counter). Right aligned metrics. Extended AUC to four digits. * Add implementation of non -ova multi class trainers code gen (#267) * added non ova multi class learners * added tests * test cases * Add caching (#249) * AdvancedExperimentSettings sample nits (#265) * Add sampling key column (#268) * Initial work for multi-class classification support for CLI (#226) * Initial work for multi-class classification support for CLI * String updates * more strings * Whitelist non-OVA multi-class learners * Refactor the orchestration of AutoML calls (#272) * Do not auto-group columns with suggested purpose = 'Ignore' (#273) * Fix: during type inferencing, parse whitespace strings as NaN (#271) * Printing additional metrics in CLI for binary classification (#274) * Printing additional metrics in CLI for binary classification * Update src/mlnet/Utilities/ConsolePrinter.cs * Add API option to store models on disk (instead of in memory); fix IEstimator memory leak (#269) * Print failed iterations in CLI (#275) * change the type to float from double (#277) * cache arg implementation in CLI (#280) * cache implementation * corrected the null case * added tests for all cases * Remove duplicate value-to-key mapping transform for multiclass string labels (#283) * Add post-trainer transform SDK infra; add KeyToValueMapping transform to CLI; fix: for generated multiclass models, convert predicted label from key to original label column type (#286) * Implement ignore columns command line arg (#290) * normalize line endings * added --ignore-columns * null checks * unit tests * Print winning iteration and runtime in CLI (#288) * Print best metric and runtime * Print best metric and runtime * Line endings in AutoMLEngine.cs * Rename time column to duration to match Python SDK * Revert to MicroAccuracy and MacroAccuracy spellings * Revert spelling of BinaryClassificationMetricsAgent to BinaryMetricsAgent to reduce merge conflicts * Revert spelling of MulticlassMetricsAgent to MultiMetricsAgent to reduce merge conflicts * missed some files * Fix merge conflict * Update AutoMLEngine.cs * Add MacOS & Linux to CI; MacOS & Linux test fixes (#293) * MicroAccuracy as default for multi-class (#295) Change default optimization metric for multi-class classification to MicroAccuracy (accuracy). Previously it was set to MacroAccuracy. * Null exception for ignorecolumns in CLI (#294) * Null exception for ignorecolumns in CLI * Check if ignore-columns array has values (as the default is now a empty array) * Emit caching flag in pipeline object model. (Includes SuggestedPipelineBuilder refactor & debug string fixes / refactor) (#296) * removed sln (#297) * Caching enabling in code gen part -2 (#298) * add * added caching codegen * support comma separated values for --ignore-columns (#300) * default initialization for ignore columns (#302) * default initialization * adde null check * Codegen for multiclass non-ova (#303) * changes to template * multicalss codegen * test cases * fix test cases * Generated Project new structure. (#305) * added new templates * writing files to disck * change path * added new templates * misisng braces * fix bugs * format code * added util methods for solution file creation and addition of projects to it * added extra packages to project files * new tests * added correct path for sln * build fix * fix build * include using system in prediction class (#307) * added using * fix test * Random number generator is not thread safe (#310) * Random number generator is not thread safe * Another local random generator * Missed a few references * Referncing AutoMlUtils.random instead of a local RNG * More refs to mail RNG; remove Float as per #1669 * Missed Random.cs * Fix multiclass code gen (#314) * compile error in codegen * removes scores printing * fix bugs * fix test * Fix compile error in codegen project (#319) * removed redundant code * fix test case * Rev OVA pipeline node SDK output: wrap binary trainers as children inside parent OVA node (#317) * Ova Multi class codegen support (#321) * dummy * multiova implementation * fix tests * remove inclusion list * fix tests and console helper * Rev run result trainer name for OVA: output different trainer name for each OVA + binary learner combination (#322) * Rev run result trainer name for Ova: output different trainer name for each Ova + binary learner combination * test fixes * Console helper bug in generated code for multiclass (#323) * fix * fix test * looping perlogclass * fix test * Initial version of Progress bar impl and CLI UI experience (#325) * progressbar * added progressbar and refactoring * reverted * revert sign assembly * added headers and removed exception rethrow * Setting model directory to temp directory (#327) * Suggested changes to progress bar (#335) * progressbar * added progressbar and refactoring * reverted * revert sign assembly * added headers and removed exception rethrow * bug fixes and updates to UI * added friendly name printing for metric * formatting * Rev Samples (#334) * Telemetry2 (#333) * Create test.txt * Create test.txt * changes needed for benchmarking * forgot one file * merge conflict fix * fix build break * back out my version of the fix for Label column issue and fix the original fix * bogus file removal * undo SuggestedPipeline change * remove labelCol from pipeline suggester * fix build break * rename AutoML to Microsoft.ML.Auto everywhere and a shot at publishing nuget package (will probably need tweaks once I try to use the pipleline) * tweak queue in vsts-ci.yml * CLI telemetry implementation * Telemetry implementation * delete unnecessary file and change file size bucket to actually log log2 instead of nearest ceil value * add headers, remove comments * one more header missing * Fix progress bar in linux/osx (#336) * progressbar * added progressbar and refactoring * reverted * revert sign assembly * added headers and removed exception rethrow * bug fixes and updates to UI * added friendly name printing for metric * formatting * change from task to thread * Update src/mlnet/CodeGenerator/CodeGenerationHelper.cs Co-Authored-By: srsaggam <41802116+srsaggam@users.noreply.github.com> * Mem leak fix (#328) * Create test.txt * Create test.txt * changes needed for benchmarking * forgot one file * merge conflict fix * fix build break * back out my version of the fix for Label column issue and fix the original fix * bogus file removal * undo SuggestedPipeline change * remove labelCol from pipeline suggester * fix build break * rename AutoML to Microsoft.ML.Auto everywhere and a shot at publishing nuget package (will probably need tweaks once I try to use the pipleline) * tweak queue in vsts-ci.yml * there is still investigation to be done but this fix works and solves memory leak problems * minor refactor * Upgrade ML.NET package (#343) * Add cross-validation (CV), and auto-CV for small datasets; push common API experiment methods into base class (#287) * restore old yml for internal pipeline so we can publish nuget again to devdiv stream (#344) * Polishing the CLI UI part-1 (#338) * formatting of pbar message * Polishing the UI * optimization * rename variable * Update src/mlnet/AutoML/AutoMLEngine.cs Co-Authored-By: srsaggam <41802116+srsaggam@users.noreply.github.com> * Update src/mlnet/CodeGenerator/CodeGenerationHelper.cs Co-Authored-By: srsaggam <41802116+srsaggam@users.noreply.github.com> * new message * changed hhtp to https * added iteration num + 1 * change string name and add color to artifacts * change the message * build errors * added null checks * added exception messsages to log file * added exception messsages to log file * CLI ML.NET version upgrade (#345) * Sample revs; ColumnInformation property name revs; pre-featurizer fixes (#346) * CLI -- consume logs from AutoML SDK (#349) * Rename RunDetails --> RunDetail (#350) * command line api upgrade and progress bar rendering bug (#366) * added fix for all platforms progress bar * upgrade nuget * removed args from writeline * change in the version (#368) * fix few bugs in progressbar and verbosity (#374) * fix few bugs in progressbar and verbosity * removed unused name space * Fix for folders with space in it while generating project (#376) * support for folders with spaces * added support for paths with space * revert file * change name of var * remove spaces * SMAC fix for minimizing metrics (#363) * Formatting Regression metrics and progress bar display days. (#379) * added progress bar day display and fix regression metrics * fix formatting * added total time * formatted total time * change command name and add pbar message (#380) * change command name and add pbar message * fix tests * added aliases * duplicate alias * added another alias for task * UI missing features (#382) * added formatting changes * added accuracy specifically * downgrade the codepages (#384) * Change in project structure (#385) * initial changes * Change in project structure * correcting test * change variable name * fix tests * fix tests * fix more tests * fix codegen errors * adde log file message * changed name of args * change variable names * fix test * FileSizeBuckets in correct units (#387) * Minor telemetry change to log in correct units and make our life easier in the future * Use Ceiling instead of Round * changed order (#388) * prep work to transfer to ml.net (#389) * move test projects to top level test subdir * rename some projects to make naming consistent and make it build again * fix test project refs * Add AutoML components to build, fix issues related to that so it builds

* Fixed build errors resulting from upgrade to VS2019 compilers * Added additional message describing the previous fix * Syncing upstream fork (#10) * Throw error on incorrect Label name in InferColumns API (#47) * Added sequential grouping of columns * reverted the file * addded infer columns label name checking * added column detection error * removed unsed usings * added quotes * replace Where with Any clause * replace Where with Any clause * Set Nullable Auto params to null values (#50) * Added sequential grouping of columns * reverted the file * added auto params as null * change to the update fields method * First public api propsal (#52) * Includes following 1) Final proposal for 0.1 public API surface 2) Prefeaturization 3) Splitting train data into train and validate when validation data is null 4) Providing end to end samples one each for regression, binaryclassification and multiclass classification * Incorporating code review feedbacks * Revert "Set Nullable Auto params to null values" (#53) * Revert "First public api propsal (#52)" This reverts commit e4a64cf4aeab13ee9e5bf0efe242da3270241bd7. * Revert "Set Nullable Auto params to null values (#50)" This reverts commit 41c663cd14247d44022f40cf2dce5977dbab282d. * AutoFit return type is now an IEnumerable (#55) AutoFit returns is now an IEnumerable - this enables many good things Implementing variety of early stopping criteria (See sample) Early discard of models that are no good. This improves memory usage efficiency. (See sample) No need to implement a callback to get results back Getting best score is now outside of API implementation. It is a simple math function to compare scores (See sample). Also templatized the return type for better type safety through out the code. * misc fixes & test additions, towards 0.1 release (#56) * Enable UnitTests on build server (#57) * 1) Making trainer name public (#62) 2) Fixing up samples to reflect it * Initial version of CLI tool for mlnet (#61) * added global tool initial project * removed unneccesary files, renamed files * refactoring and added base abstract classes for trainer generator * removed unused class * Added classes for transforms * added transform generate dummy classes * more refactoring, added first transform * more refactoring and added classes * changed the project structure * restructing added options class * sln changes * refactored options to different class: * added more logic for code generation of class * misc changes * reverted file * added commandline api package * reverted sample * added new command line api parser * added normalization of column names * Added command defaults and error message * implementation of all trainers * changed auto to null * added all transform generators * added error handling when args is empty and minor changes due to change in AutoML api names * changed the name of param * added new command line options and restructuring code * renamed proj file and added solution * Added code to generate usings, Fixed few bugs in the code * added validation to the command line options * changed project name * Bug fixes due to API change in AutoML * changed directory structure * added test framework and basic tests * added more tests * added improvements to template and error handling * renamed the estimator name * fixed test case * added comments * added headers * changed namespace and removed unneccesary properties from project * Revert "changed namespace and removed unneccesary properties from project" This reverts commit 9edae033e9845e910f663f296e168f1182b84f5f. * fixed test cases and renamed namespaces * cleaned up proj file * added folder structure * added symbols/tokens for strings * added more tests * review comments * modified test cases * review comments * change in the exception message * normalized line endings * made method private static * simplified range building /optimization * minor fix * added header * added static methods in command where necessary * nit picks * made few methods static * review comments * nitpick * remove line pragmas * fix test case * Use better AutiFit overload and ignore Multiclass (#64) * Upgrading CLI to produce ML.NET V.10 APIs and bunch of Refactoring tasks (#65) * Added sequential grouping of columns * reverted the file * upgrade to v .10 and refactoring * added null check * fixed unit tests * review comments * removed the settings change * added regions * fixed unit tests * Upgrade ML.NET package to 0.10.0 (#70) * Change in template to accomodate new API of TextLoader (#72) * Added sequential grouping of columns * reverted the file * changed to new API of Text Loader * changed signature * added params for taking additional settings * changes to codegen params * refactoring of templates and fixing errors * Enable gated check for mlnet.tests (#79) * Added sequential grouping of columns * reverted the file * changed to new API of Text Loader * changed signature * added params for taking additional settings * changes to codegen params * refactoring of templates and fixing errors * added run-tests.proj and referred it in build.proj * CLI tool - make validation dataset optional and support for crossvalidation in generated code (#83) * Added sequential grouping of columns * reverted the file * bug fixes, more logic to templates to support cross-validate * formatting and fix type in consolehelper * Added logic in templates * revert settings * benchmarking related changes (#63) * Create test.txt * Create test.txt * changes needed for benchmarking * forgot one file * merge conflict fix * fix build break * back out my version of the fix for Label column issue and fix the original fix * bogus file removal * undo SuggestedPipeline change * remove labelCol from pipeline suggester * fix build break * fix fast forest learner (don't sweep over learning rate) (#88) * Made changes to Have non-calibrated scoring for binary classifiers (#86) * Added sequential grouping of columns * reverted the file * added calibration workaround * removed print probability * reverted settings * rev ColumnInference API: can take label index; rev output object types; add tests (#89) * rename AutoML to Microsoft.ML.Auto everywhere and a shot at publishing nuget package (#99) * Create test.txt * Create test.txt * changes needed for benchmarking * forgot one file * merge conflict fix * fix build break * back out my version of the fix for Label column issue and fix the original fix * bogus file removal * undo SuggestedPipeline change * remove labelCol from pipeline suggester * fix build break * rename AutoML to Microsoft.ML.Auto everywhere and a shot at publishing nuget package (will probably need tweaks once I try to use the pipleline) * publish nuget (#101) * use dotnet-internal-temp agent for internal build * use dotnet-internal feed * Fix Codegen for columnConvert and ValueToKeyMapping transform and add individual transform tests (#95) * Added sequential grouping of columns * reverted the file * fix usings for type convert * added transforms tests * review comments * When generating usings choose only distinct usings directives (#94) * Added sequential grouping of columns * reverted the file * Added code to have unique strings * refactoring * minor fix * minor fix * Autofit overloads + cancellation + progress callbacks 1) Introduce AutoFit overloads (basic and advanced) 2) AutoFit Cancellation 3) AutoFit progress callbacks * Default the kfolds to value 5 in CLI generated code (#115) * Added sequential grouping of columns * reverted the file * Set up CI with Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * remove file * added kfold param and defaulted to value * changed type * added for regression * Remove extra ; from generated code (#114) * Added sequential grouping of columns * reverted the file * Set up CI with Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * removed extra ; from generated code * removed file * fix unit tests * TimeoutInSeconds (#116) Specifying timeout in seconds instead of minutes * Added more command line args implementation to CLI tool and refactoring (#110) * Added sequential grouping of columns * reverted the file * Set up CI with Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * added git status * reverted change * added codegen options and refactoring * minor fixes' * renamed params, minor refactoring * added tests for commandline and refactoring * removed file * added back the test case * minor fixes * Update src/mlnet.Test/CommandLineTests.cs Co-Authored-By: srsaggam <41802116+srsaggam@users.noreply.github.com> * review comments * capitalize the first character * changed the name of test case * remove unused directives * Fail gracefully if unable to instantiate data view with swept parameters (#125) * gracefully fail if fail to parse a datai * rev * validate AutoFit 'Features' column must be of type R4 (#132) * Samples: exceptions / nits (#124) * Logging support in CLI + Implementation of cmd args [--name,--output,--verbosity] (#121) * addded logging and helper methods * fixing code after merge * added resx files, added logger framework, added logging messages * added new options * added spacing * minor fixes * change command description * rename option, add headers, include new param in test * formatted * build fix * changed option name * Added NlogConfig file * added back config package * fix tests * added correct validation check (#137) * Use CreateTextLoader<T>(..) instead of CreateTextLoader(..) (#138) * added support to loaddata by class in the generated code * fix tests * changed CreateTextLoader to ReadFromTextFile method. (#140) * changed textloader to readfromtextfile method * formatting * exception fixes (#136) * infer purpose of hidden columns as 'ignore' (#142) * Added approval tests and bunch of refactoring of code and normalizing namespaces (#148) * changed textloader to readfromtextfile method * formatting * added approval tests and refactoring of code * removed few comments * API 2.0 skeleton (#149) Incorporating API review feedback * The CV code should come before the training when there is no test dataset in generated code (#151) * reorder cv code * build fix * fixed structure * Format the generated code + bunch of misc tasks (#152) * added formatting and minor changes for reordering cv * fixing the template * minor changes * formatting changes * fixed approval test * removed unused nuget * added missing value replacing * added test for new transform * fix test * Update src/mlnet/Templates/Console/MLCodeGen.cs Co-Authored-By: srsaggam <41802116+srsaggam@users.noreply.github.com> * Sanitize the column names in CLI (#162) * added sanitization layer in CLI * fix test * changed exception.StackTrace to exception.ToString() * fix package name (#168) * Rev public API (#163) * Rename TransformGeneratorBase .cs to TransformGeneratorBase.cs (#153) * Fix minor version for the repository + remove Nlog config package (#171) * changed the minor version * removed the nlog config package * Added new test to columninfo and fixing up API (#178) * Make optimizing metric customizable and add trainer whitelist functionality (#172) * API rev (#181) * propagate root MLContext thru AutoML (instead of creating our own) (#182) * Enabling new command line args (#183) * fix package name * initial commit * added more commandline args * fixed tests * added headers * fix tests * fix test * rename 'AutoFitter' to 'Experiment' (#169) * added tests (#187) * rev InferColumns to accept ColumnInfo input param (#186) * Implement argument --has-header and change usage of dataset (#194) * added has header and fixed dataset and train dataset * fix tests * removed dummy command (#195) * Fix bug for regression and sanitize input label from user (#198) * removed dummy command * sanitize label and fix template * fix tests * Do not generate code concatenating columns when the dataset has a single feature column (#191) * Include some missed logging in the generated code. (#199) * added logging messages for generated code * added log messages * deleted file * cleaning up proj files (#185) * removed platform target * removed platform target * Some spaces and extra lines + bug in output path (#204) * nit picks * nit picks * fix test * accept label from user input and provide in generated code (#205) * Rev handling of weight / label columns (#203) * migrate to private ML.NET nuget for latest bug fixes (#131) * fix multiclass with nonstandard label (#207) * Multiclass nondefault label test (#208) * printing escaped chars + bug (#212) * delete unused internal samples (#211) * fix SMAC bug that causes multiclass sample to infinite loop (#209) * Rev user input validation for new API (#210) * added console message for exit and nit picks (#215) * exit when exception encountered (#216) * Seal API classes (and make EnableCaching internal) (#217) * Suggested sample nits (feel free to ask for any of these to be reverted) (#219) * User input column type validation (#218) * upgrade commandline and renaming (#221) * upgrade commandline and renaming * renaming fields * Make build.sh, init-tools.sh, & run.sh executable on OSX/Linux (#225) * CLI argument descriptions updated (#224) * CLI argument descriptions updated * No version in .csproj * added flag to disable training code (#227) * Exit if perfect model produced (#220) * removed header (#228) * removed header * added auto generated header * removed console read key (#229) * Fix model path in generated file (#230) * removed console read key * fix model path * fix test * reorder samples (#231) * remove rule that infers column purpose as categorical if # of distinct values is < 100 (#233) * Null reference exception fix for finding best model when some runs have failed (#239) * samples fixes (#238) * fix for defaulting Averaged Perceptron # of iterations to 10 (#237) * Bug bash feedback Feb 27. API changes and sample changes (#240) * Bug bash feedback Feb 27. API changes Sample changes Exception fix * Samples / API rev from 2/27 bug bash feedback (#242) * changed the directory structure for generated project (#243) * changed the directory structure for generated project * changed test * upgraded commandline package * Fix test file locations on OSX (#235) * fix test file locations on OSX * changing to Path.Combine() * Additional Path.Combine() * Remove ConsoleCodeGeneratorTests.GeneratedTrainCodeTest.received.txt * Additional Path.Combine() * add back in double comparison fix * remove metrics agent NaN returns * test fix * test format fix * mock out path Thanks to @daholste for additional fixes! * upgrade to latest ML.NET public surface (#246) * Upgrade to ML.NET 0.11 (#247) * initial changes * fix lightgbm * changed normalize method * added tests * fix tests * fix test * Private preview final API changes (#250) * .NET framework design guidelines applied to public surface * WhitelistedTrainers -> Trainers * Add estimator to public API iteration result (#248) * LightGBM pipeline serialization fix (#251) * Change order that we search for TextLoader's parameters (#256) * CLI IFileInfo null exception fix (#254) * Averaged Perceptron pipeline serialization fix (#257) * Upgrade command-line-api and default folder name change (#258) * change in defautl folderName * upgrade command line * Update src/mlnet/Program.cs Co-Authored-By: srsaggam <41802116+srsaggam@users.noreply.github.com> * eliminate IFileInfo from CLI (#260) * Rev samples towards private preview; ignored columns fix (#259) * remove unused methods in consolehelper and nit picks in generated code (#261) * nit picks * change in console helper * fix tests * add space * fix tests * added nuget sources in generated csproj (#262) * added nuget sources in csproj * changed the structure in generated code * space * upgrade to mlnet 0.11 (#263) * Formatting CLI metrics (#264) Ensures space between printed metrics (also model counter). Right aligned metrics. Extended AUC to four digits. * Add implementation of non -ova multi class trainers code gen (#267) * added non ova multi class learners * added tests * test cases * Add caching (#249) * AdvancedExperimentSettings sample nits (#265) * Add sampling key column (#268) * Initial work for multi-class classification support for CLI (#226) * Initial work for multi-class classification support for CLI * String updates * more strings * Whitelist non-OVA multi-class learners * Refactor the orchestration of AutoML calls (#272) * Do not auto-group columns with suggested purpose = 'Ignore' (#273) * Fix: during type inferencing, parse whitespace strings as NaN (#271) * Printing additional metrics in CLI for binary classification (#274) * Printing additional metrics in CLI for binary classification * Update src/mlnet/Utilities/ConsolePrinter.cs * Add API option to store models on disk (instead of in memory); fix IEstimator memory leak (#269) * Print failed iterations in CLI (#275) * change the type to float from double (#277) * cache arg implementation in CLI (#280) * cache implementation * corrected the null case * added tests for all cases * Remove duplicate value-to-key mapping transform for multiclass string labels (#283) * Add post-trainer transform SDK infra; add KeyToValueMapping transform to CLI; fix: for generated multiclass models, convert predicted label from key to original label column type (#286) * Implement ignore columns command line arg (#290) * normalize line endings * added --ignore-columns * null checks * unit tests * Print winning iteration and runtime in CLI (#288) * Print best metric and runtime * Print best metric and runtime * Line endings in AutoMLEngine.cs * Rename time column to duration to match Python SDK * Revert to MicroAccuracy and MacroAccuracy spellings * Revert spelling of BinaryClassificationMetricsAgent to BinaryMetricsAgent to reduce merge conflicts * Revert spelling of MulticlassMetricsAgent to MultiMetricsAgent to reduce merge conflicts * missed some files * Fix merge conflict * Update AutoMLEngine.cs * Add MacOS & Linux to CI; MacOS & Linux test fixes (#293) * MicroAccuracy as default for multi-class (#295) Change default optimization metric for multi-class classification to MicroAccuracy (accuracy). Previously it was set to MacroAccuracy. * Null exception for ignorecolumns in CLI (#294) * Null exception for ignorecolumns in CLI * Check if ignore-columns array has values (as the default is now a empty array) * Emit caching flag in pipeline object model. (Includes SuggestedPipelineBuilder refactor & debug string fixes / refactor) (#296) * removed sln (#297) * Caching enabling in code gen part -2 (#298) * add * added caching codegen * support comma separated values for --ignore-columns (#300) * default initialization for ignore columns (#302) * default initialization * adde null check * Codegen for multiclass non-ova (#303) * changes to template * multicalss codegen * test cases * fix test cases * Generated Project new structure. (#305) * added new templates * writing files to disck * change path * added new templates * misisng braces * fix bugs * format code * added util methods for solution file creation and addition of projects to it * added extra packages to project files * new tests * added correct path for sln * build fix * fix build * include using system in prediction class (#307) * added using * fix test * Random number generator is not thread safe (#310) * Random number generator is not thread safe * Another local random generator * Missed a few references * Referncing AutoMlUtils.random instead of a local RNG * More refs to mail RNG; remove Float as per https://github.com/dotnet/machinelearning/issues/1669 * Missed Random.cs * Fix multiclass code gen (#314) * compile error in codegen * removes scores printing * fix bugs * fix test * Fix compile error in codegen project (#319) * removed redundant code * fix test case * Rev OVA pipeline node SDK output: wrap binary trainers as children inside parent OVA node (#317) * Ova Multi class codegen support (#321) * dummy * multiova implementation * fix tests * remove inclusion list * fix tests and console helper * Rev run result trainer name for OVA: output different trainer name for each OVA + binary learner combination (#322) * Rev run result trainer name for Ova: output different trainer name for each Ova + binary learner combination * test fixes * Console helper bug in generated code for multiclass (#323) * fix * fix test * looping perlogclass * fix test * Initial version of Progress bar impl and CLI UI experience (#325) * progressbar * added progressbar and refactoring * reverted * revert sign assembly * added headers and removed exception rethrow * Setting model directory to temp directory (#327) * Suggested changes to progress bar (#335) * progressbar * added progressbar and refactoring * reverted * revert sign assembly * added headers and removed exception rethrow * bug fixes and updates to UI * added friendly name printing for metric * formatting * Rev Samples (#334) * Telemetry2 (#333) * Create test.txt * Create test.txt * changes needed for benchmarking * forgot one file * merge conflict fix * fix build break * back out my version of the fix for Label column issue and fix the original fix * bogus file removal * undo SuggestedPipeline change * remove labelCol from pipeline suggester * fix build break * rename AutoML to Microsoft.ML.Auto everywhere and a shot at publishing nuget package (will probably need tweaks once I try to use the pipleline) * tweak queue in vsts-ci.yml * CLI telemetry implementation * Telemetry implementation * delete unnecessary file and change file size bucket to actually log log2 instead of nearest ceil value * add headers, remove comments * one more header missing * Fix progress bar in linux/osx (#336) * progressbar * added progressbar and refactoring * reverted * revert sign assembly * added headers and removed exception rethrow * bug fixes and updates to UI * added friendly name printing for metric * formatting * change from task to thread * Update src/mlnet/CodeGenerator/CodeGenerationHelper.cs Co-Authored-By: srsaggam <41802116+srsaggam@users.noreply.github.com> * Mem leak fix (#328) * Create test.txt * Create test.txt * changes needed for benchmarking * forgot one file * merge conflict fix * fix build break * back out my version of the fix for Label column issue and fix the original fix * bogus file removal * undo SuggestedPipeline change * remove labelCol from pipeline suggester * fix build break * rename AutoML to Microsoft.ML.Auto everywhere and a shot at publishing nuget package (will probably need tweaks once I try to use the pipleline) * tweak queue in vsts-ci.yml * there is still investigation to be done but this fix works and solves memory leak problems * minor refactor * Upgrade ML.NET package (#343) * Add cross-validation (CV), and auto-CV for small datasets; push common API experiment methods into base class (#287) * restore old yml for internal pipeline so we can publish nuget again to devdiv stream (#344) * Polishing the CLI UI part-1 (#338) * formatting of pbar message * Polishing the UI * optimization * rename variable * Update src/mlnet/AutoML/AutoMLEngine.cs Co-Authored-By: srsaggam <41802116+srsaggam@users.noreply.github.com> * Update src/mlnet/CodeGenerator/CodeGenerationHelper.cs Co-Authored-By: srsaggam <41802116+srsaggam@users.noreply.github.com> * new message * changed hhtp to https * added iteration num + 1 * change string name and add color to artifacts * change the message * build errors * added null checks * added exception messsages to log file * added exception messsages to log file * CLI ML.NET version upgrade (#345) * Sample revs; ColumnInformation property name revs; pre-featurizer fixes (#346) * CLI -- consume logs from AutoML SDK (#349) * Rename RunDetails --> RunDetail (#350) * command line api upgrade and progress bar rendering bug (#366) * added fix for all platforms progress bar * upgrade nuget * removed args from writeline * change in the version (#368) * fix few bugs in progressbar and verbosity (#374) * fix few bugs in progressbar and verbosity * removed unused name space * Fix for folders with space in it while generating project (#376) * support for folders with spaces * added support for paths with space * revert file * change name of var * remove spaces * SMAC fix for minimizing metrics (#363) * Formatting Regression metrics and progress bar display days. (#379) * added progress bar day display and fix regression metrics * fix formatting * added total time * formatted total time * change command name and add pbar message (#380) * change command name and add pbar message * fix tests * added aliases * duplicate alias * added another alias for task * UI missing features (#382) * added formatting changes * added accuracy specifically * downgrade the codepages (#384) * Change in project structure (#385) * initial changes * Change in project structure * correcting test * change variable name * fix tests * fix tests * fix more tests * fix codegen errors * adde log file message * changed name of args * change variable names * fix test * FileSizeBuckets in correct units (#387) * Minor telemetry change to log in correct units and make our life easier in the future * Use Ceiling instead of Round * changed order (#388) * prep work to transfer to ml.net (#389) * move test projects to top level test subdir * rename some projects to make naming consistent and make it build again * fix test project refs * Add AutoML components to build, fix issues related to that so it builds * fix test cases, remove AppInsights ref from AutoML (#3329) * [AutoML] disable netfx build leg for now (#3331) * disable netfx build leg for now * disable netfx build leg for now. * [AutoML] Add AutoML XML documentation to all public members; migrate AutoML projects & tests into ML.NET solution; AutoML test fixes (#3351) * [AutoML] Rev AutoML public API; add required native references to AutoML projects (#3364) * [AutoML] Minor changes to generated project in CLI based on feedback (#3371) * nitpicks for generated project * revert back the target framework * [AutoML] Migrate AutoML back to its own solution, w/ NuGet dependencies (#3373) * Migrate AutoML back to its own solution, w/ NuGet dependencies * build project updates; parameter name revert * dummy change * Revert "dummy change" This reverts commit 3e8574266f556a4d5b6805eb55b4d8b8b84cf355. * [AutoML] publish AutoML package (#3383) * publish AutoML package * Only leave automl and mlnet tests to run * publish AutoML package * Only leave automl and mlnet tests to run * fix build issues when ml.net is not building * bump version to 0.3 since that's the one we're going to ship for build (#3416) * [AutoML] temporarily disable all but x64 platforms -- don't want to do native builds and can't find a way around that with the current VSTS pipeline (#3420) * disable steps but keep phases to keep vsts build pipeline happy (#3423) * API docs for experimentation (#3484) * fixed path bug and regression metrics correction (#3504) * changed the casing of option alias as it conflicts with --help (#3554) * [AutoML] Generated project - FastTree nuget package inclusion dynamically (#3567) * added support for fast tree nuget pack inclusion in generated project * fix testcase * changed the tool name in telemetry message * dummy commit * remove space * dummy commit to trigger build * [AutoML] Add AutoML example code (#3458) * AutoML PipelineSuggester: don't recommend pipelines from first-stage trainers that failed (#3593) * InferColumns API: Validate all columns specified in column info exist in inferred data view (#3599) * [AutoML] AutoML SDK API: validate schema types of input IDataView (#3597) * [AutoML] If first three iterations all fail, short-circuit AutoML experiment (#3591) * mlnet CLI nupkg creation/signing (#3606) * mlnet CLI nupkg creation/signing * relmove includeinpackage from mlnet csproj * address PR comments -- some minor reshuffling of stuff * publish symbols for mlnet CLI * fix case in NLog.config * [AutoML] rename Auto to AutoML in namespace and nuget (#3609) * mlnet CLI nupkg creation/signing * [AutoML] take dependency on a specific ml.net version (#3610) * take dependency on a specific ml.net version * catch up to spelling fix for OptimizationTolerance * force a specific ml.net nuget version, fix typo (#3616) * [AutoML] Fix error handling in CLI. (#3618) * fix error handling * renaming variables * [AutoML] turn off line pragmas in .tt files to play nice with signing (#3617) * turn off line pragmas in .tt files to play nice with signing * dedupe tags * change the param name (#3619) * [AutoML] return null instead of null ref crash on Model property accessor (#3620) * return null instead of null ref crash on Model property accessor * [AutoML] Handling label column names which have space and exception logging (#3624) * fix case of label with space and exception logging * final handler * revert file * use Name instead of FullName for telemetry filename hash (#3633) * renamed classes (#3634) * change ML.NET dependency to 1.0 (#3639) [AutoML] undo pinning ML.NET dependency * set exploration time default in CLI to half hour (#3640) * [AutoML] step 2 of removing pinned nupkg versions (#3642) * InferColumns API that consumes label column index -- Only rename label column to 'Label' for headerless files (#3643) * [AutoML] Upgrade ml.net package in generated code (#3644) * upgrade the mlnet package in gen code * Update src/mlnet/Templates/Console/ModelProject.cs Co-Authored-By: srsaggam <41802116+srsaggam@users.noreply.github.com> * Update src/mlnet/Templates/Console/ModelProject.tt Co-Authored-By: srsaggam <41802116+srsaggam@users.noreply.github.com> * added spaces * [AutoML] Early stopping in CLI based on the exploration time (#3641) * early stopping in CLI * remove unused variables * change back to thread * remove sleep * fix review comments * remove ununsed usings * format message * collapse declaration * remove unused param * added environment.exit and removal of error message * correction in message * secs-> seconds * exit code * change value to 1 * reverse the declaration * [AutoML] Change wording for CouldNotFinshOnTime message (#3655) * set exploration time default in CLI to half hour * [AutoML] Change wording for CouldNotFinshOnTime message * [AutoML] Change wording for CouldNotFinshOnTime message * even better wording for CouldNotFinshOnTime * temp change to get around vsts publish failure (#3656) * [AutoML] bump version to 0.4.0 (#3658) * implement culture invariant strings (#3725) * reset culture (#3730) * [AutoML] Cross validation fixes; validate empty training / validation input data (#3794) * [AutoML] Enable style cop rules & resolve errors (#3823) * add task agnostic wrappers for autofit calls (#3860) * [AutoML] CLI telemetry rev (#3789) * delete automl .sln * CLI -- regenerate templated CS files (#3954) * [AutoML] Bump ML.NET package version to 1.2.0 in AutoML API and CLI; and AutoML package versions to 0.14.0 (#3958) * Build AutoML NuGet package (#3961) * Increment AutoML build version to 0.15.0 for preview. (#3968) * added culture independent parsing (#3731) * - convert tests to xunit - take project level dependency on ML.NET components instead of nuget - set up bestfriends relationship to ML.Core and remove some of the copies of util classes from AutoML.NET (more work needed to fully remove them, work item 4064) - misc build script changes to address PR comments * address issues only showing up in a couple configurations during CI build * fix cut&paste error * [AutoML] Bump version to ML.NET 1.3.1 in AutoML API and CLI and AutoML package version to 0.15.1 (#4071) * bumped version * change versions in nupkg * revert version bump in branch props * [AutoML] Fix for Exception thrown in cross val when one of the score equals infinity. (#4073) * bumped version * change versions in nupkg * revert version bump in branch props * added infinity fix * changes signing (#4079) * Addressed PR comments and build issues - sync block on creating test data file (failed intermittently) - removed classes we copied over from ML.Core and fixed their uses to de-dupe and use original ML.Core versions since we now have InternalsVisible and BestFriends - Fixed nupkg creation to use projects insted of public nuget version for AutoML - Fixed a bunch of unit tests that didn't actually test what they were supposed to test, while removing cut&past code and dependencies. - Few more misc small changes * minor nit - removed unused folder ref * Fix the .sln file for the right configurations. * Fix mistake in .sln file * test fixes and disable one test * fix tests, re-add AutoML samples csproj * bumped VS version to 16 in .sln, removed InternalsVisible for a dead assembly, removed unused references from AutoML test project * Updated docs to include PredictedLabel member (#4107) * Fixed build errors resulting from upgrade to VS2019 compilers * Added additional message describing the previous fix * Updated docs to include PredictedLabel member * Added CODEOWNERS file in the .github/ folder. (#4140) * Added CODEOWNERS file in the .github/ folder. This allows reviewers to review any changes in the machine learning repository * Updated .github/CODEOWNERS with the team instead of individual reviewers * Added AutoML team reviewers (#4144) * Added CODEOWNERS file in the .github/ folder. This allows reviewers to review any changes in the machine learning repository * Updated .github/CODEOWNERS with the team instead of individual reviewers * Added AutoML team reviwers to files owned by AutoML team * Added AutoML team reviwers to files owned by AutoML team * Removed two files that don't exist for AutoML team in CODEOWNERS * Build extension method to reload changes without specifying model name (#4146) * Image classification preview 2. (#4151) * Image classification preview 2. * PR feedback. * Add unit-test. * Add unit-test. * Add unit-test. * Add unit-test. * Use Path.Combine instead of Join. * fix test dataset path. * fix test dataset path. * Improve test. * Improve test. * Increase epochs in tests. * Disable test on Ubuntu. * Move test to its own project. * Move test to its own project. * Move test to its own project. * Move test to its own file. * cleanup. * Disable parallel execution of tensorflow tests. * PR feedback. * PR feedback. * PR feedback. * PR feedback. * Prevent TF test to execute in parallel. * PR feedback. * Build error. * clean up. * Added export functionality for LpNormNormalizingTransformer * Syncing upstream fork (#11) * Throw error on incorrect Label name in InferColumns API (#47) * Added sequential grouping of columns * reverted the file * addded infer columns label name checking * added column detection error * removed unsed usings * added quotes * replace Where with Any clause * replace Where with Any clause * Set Nullable Auto params to null values (#50) * Added sequential grouping of columns * reverted the file * added auto params as null * change to the update fields method * First public api propsal (#52) * Includes following 1) Final proposal for 0.1 public API surface 2) Prefeaturization 3) Splitting train data into train and validate when validation data is null 4) Providing end to end samples one each for regression, binaryclassification and multiclass classification * Incorporating code review feedbacks * Revert "Set Nullable Auto params to null values" (#53) * Revert "First public api propsal (#52)" This reverts commit e4a64cf4aeab13ee9e5bf0efe242da3270241bd7. * Revert "Set Nullable Auto params to null values (#50)" This reverts commit 41c663cd14247d44022f40cf2dce5977dbab282d. * AutoFit return type is now an IEnumerable (#55) AutoFit returns is now an IEnumerable - this enables many good things Implementing variety of early stopping criteria (See sample) Early discard of models that are no good. This improves memory usage efficiency. (See sample) No need to implement a callback to get results back Getting best score is now outside of API implementation. It is a simple math function to compare scores (See sample). Also templatized the return type for better type safety through out the code. * misc fixes & test additions, towards 0.1 release (#56) * Enable UnitTests on build server (#57) * 1) Making trainer name public (#62) 2) Fixing up samples to reflect it * Initial version of CLI tool for mlnet (#61) * added global tool initial project * removed unneccesary files, renamed files * refactoring and added base abstract classes for trainer generator * removed unused class * Added classes for transforms * added transform generate dummy classes * more refactoring, added first transform * more refactoring and added classes * changed the project structure * restructing added options class * sln changes * refactored options to different class: * added more logic for code generation of class * misc changes * reverted file * added commandline api package * reverted sample * added new command line api parser * added normalization of column names * Added command defaults and error message * implementation of all trainers * changed auto to null * added all transform generators * added error handling when args is empty and minor changes due to change in AutoML api names * changed the name of param * added new command line options and restructuring code * renamed proj file and added solution * Added code to generate usings, Fixed few bugs in the code * added validation to the command line options * changed project name * Bug fixes due to API change in AutoML * changed directory structure * added test framework and basic tests * added more tests * added improvements to template and error handling * renamed the estimator name * fixed test case * added comments * added headers * changed namespace and removed unneccesary properties from project * Revert "changed namespace and removed unneccesary properties from project" This reverts commit 9edae033e9845e910f663f296e168f1182b84f5f. * fixed test cases and renamed namespaces * cleaned up proj file * added folder structure * added symbols/tokens for strings * added more tests * review comments * modified test cases * review comments * change in the exception message * normalized line endings * made method private static * simplified range building /optimization * minor fix * added header * added static methods in command where necessary * nit picks * made few methods static * review comments * nitpick * remove line pragmas * fix test case * Use better AutiFit overload and ignore Multiclass (#64) * Upgrading CLI to produce ML.NET V.10 APIs and bunch of Refactoring tasks (#65) * Added sequential grouping of columns * reverted the file * upgrade to v .10 and refactoring * added null check * fixed unit tests * review comments * removed the settings change * added regions * fixed unit tests * Upgrade ML.NET package to 0.10.0 (#70) * Change in template to accomodate new API of TextLoader (#72) * Added sequential grouping of columns * reverted the file * changed to new API of Text Loader * changed signature * added params for taking additional settings * changes to codegen params * refactoring of templates and fixing errors * Enable gated check for mlnet.tests (#79) * Added sequential grouping of columns * reverted the file * changed to new API of Text Loader * changed signature * added params for taking additional settings * changes to codegen params * refactoring of templates and fixing errors * added run-tests.proj and referred it in build.proj * CLI tool - make validation dataset optional and support for crossvalidation in generated code (#83) * Added sequential grouping of columns * reverted the file * bug fixes, more logic to templates to support cross-validate * formatting and fix type in consolehelper * Added logic in templates * revert settings * benchmarking related changes (#63) * Create test.txt * Create test.txt * changes needed for benchmarking * forgot one file * merge conflict fix * fix build break * back out my version of the fix for Label column issue and fix the original fix * bogus file removal * undo SuggestedPipeline change * remove labelCol from pipeline suggester * fix build break * fix fast forest learner (don't sweep over learning rate) (#88) * Made changes to Have non-calibrated scoring for binary classifiers (#86) * Added sequential grouping of columns * reverted the file * added calibration workaround * removed print probability * reverted settings * rev ColumnInference API: can take label index; rev output object types; add tests (#89) * rename AutoML to Microsoft.ML.Auto everywhere and a shot at publishing nuget package (#99) * Create test.txt * Create test.txt * changes needed for benchmarking * forgot one file * merge conflict fix * fix build break * back out my version of the fix for Label column issue and fix the original fix * bogus file removal * undo SuggestedPipeline change * remove labelCol from pipeline suggester * fix build break * rename AutoML to Microsoft.ML.Auto everywhere and a shot at publishing nuget package (will probably need tweaks once I try to use the pipleline) * publish nuget (#101) * use dotnet-internal-temp agent for internal build * use dotnet-internal feed * Fix Codegen for columnConvert and ValueToKeyMapping transform and add individual transform tests (#95) * Added sequential grouping of columns * reverted the file * fix usings for type convert * added transforms tests * review comments * When generating usings choose only distinct usings directives (#94) * Added sequential grouping of columns * reverted the file * Added code to have unique strings * refactoring * minor fix * minor fix * Autofit overloads + cancellation + progress callbacks 1) Introduce AutoFit overloads (basic and advanced) 2) AutoFit Cancellation 3) AutoFit progress callbacks * Default the kfolds to value 5 in CLI generated code (#115) * Added sequential grouping of columns * reverted the file * Set up CI with Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * remove file * added kfold param and defaulted to value * changed type * added for regression * Remove extra ; from generated code (#114) * Added sequential grouping of columns * reverted the file * Set up CI with Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * removed extra ; from generated code * removed file * fix unit tests * TimeoutInSeconds (#116) Specifying timeout in seconds instead of minutes * Added more command line args implementation to CLI tool and refactoring (#110) * Added sequential grouping of columns * reverted the file * Set up CI with Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * added git status * reverted change * added codegen options and refactoring * minor fixes' * renamed params, minor refactoring * added tests for commandline and refactoring * removed file * added back the test case * minor fixes * Update src/mlnet.Test/CommandLineTests.cs Co-Authored-By: srsaggam <41802116+srsaggam@users.noreply.github.com> * review comments * capitalize the first character * changed the name of test case * remove unused directives * Fail gracefully if unable to instantiate data view with swept parameters (#125) * gracefully fail if fail to parse a datai * rev * validate AutoFit 'Features' column must be of type R4 (#132) * Samples: exceptions / nits (#124) * Logging support in CLI + Implementation of cmd args [--name,--output,--verbosity] (#121) * addded logging and helper methods * fixing code after merge * added resx files, added logger framework, added logging messages * added new options * added spacing * minor fixes * change command description * rename option, add headers, include new param in test * formatted * build fix * changed option name * Added NlogConfig file * added back config package * fix tests * added correct validation check (#137) * Use CreateTextLoader<T>(..) instead of CreateTextLoader(..) (#138) * added support to loaddata by class in the generated code * fix tests * changed CreateTextLoader to ReadFromTextFile method. (#140) * changed textloader to readfromtextfile method * formatting * exception fixes (#136) * infer purpose of hidden columns as 'ignore' (#142) * Added approval tests and bunch of refactoring of code and normalizing namespaces (#148) * changed textloader to readfromtextfile method * formatting * added approval tests and refactoring of code * removed few comments * API 2.0 skeleton (#149) Incorporating API review feedback * The CV code should come before the training when there is no test dataset in generated code (#151) * reorder cv code * build fix * fixed structure * Format the generated code + bunch of misc tasks (#152) * added formatting and minor changes for reordering cv * fixing the template * minor changes * formatting changes * fixed approval test * removed unused nuget * added missing value replacing * added test for new transform * fix test * Update src/mlnet/Templates/Console/MLCodeGen.cs Co-Authored-By: srsaggam <41802116+srsaggam@users.noreply.github.com> * Sanitize the column names in CLI (#162) * added sanitization layer in CLI * fix test * changed exception.StackTrace to exception.ToString() * fix package name (#168) * Rev public API (#163) * Rename TransformGeneratorBase .cs to TransformGeneratorBase.cs (#153) * Fix minor version for the repository + remove Nlog config package (#171) * changed the minor version * removed the nlog config package * Added new test to columninfo and fixing up API (#178) * Make optimizing metric customizable and add trainer whitelist functionality (#172) * API rev (#181) * propagate root MLContext thru AutoML (instead of creating our own) (#182) * Enabling new command line args (#183) * fix package name * initial commit * added more commandline args * fixed tests * added headers * fix tests * fix test * rename 'AutoFitter' to 'Experiment' (#169) * added tests (#187) * rev InferColumns to accept ColumnInfo input param (#186) * Implement argument --has-header and change usage of dataset (#194) * added has header and fixed dataset and train dataset * fix tests * removed dummy command (#195) * Fix bug for regression and sanitize input label from user (#198) * removed dummy command * sanitize label and fix template * fix tests * Do not generate code concatenating columns when the dataset has a single feature column (#191) * Include some missed logging in the generated code. (#199) * added logging messages for generated code * added log messages * deleted file * cleaning up proj files (#185) * removed platform target * removed platform target * Some spaces and extra lines + bug in output path (#204) * nit picks * nit picks * fix test * accept label from user input and provide in generated code (#205) * Rev handling of weight / label columns (#203) * migrate to private ML.NET nuget for latest bug fixes (#131) * fix multiclass with nonstandard label (#207) * Multiclass nondefault label test (#208) * printing escaped chars + bug (#212) * delete unused internal samples (#211) * fix SMAC bug that causes multiclass sample to infinite loop (#209) * Rev user input validation for new API (#210) * added console message for exit and nit picks (#215) * exit when exception encountered (#216) * Seal API classes (and make EnableCaching internal) (#217) * Suggested sample nits (feel free to ask for any of these to be reverted) (#219) * User input column type validation (#218) * upgrade commandline and renaming (#221) * upgrade commandline and renaming * renaming fields * Make build.sh, init-tools.sh, & run.sh executable on OSX/Linux (#225) * CLI argument descriptions updated (#224) * CLI argument descriptions updated * No version in .csproj * added flag to disable training code (#227) * Exit if perfect model produced (#220) * removed header (#228) * removed header * added auto generated header * removed console read key (#229) * Fix model path in generated file (#230) * removed console read key * fix model path * fix test * reorder samples (#231) * remove rule that infers column purpose as categorical if # of distinct values is < 100 (#233) * Null reference exception fix for finding best model when some runs have failed (#239) * samples fixes (#238) * fix for defaulting Averaged Perceptron # of iterations to 10 (#237) * Bug bash feedback Feb 27. API changes and sample changes (#240) * Bug bash feedback Feb 27. API changes Sample changes Exception fix * Samples / API rev from 2/27 bug bash feedback (#242) * changed the directory structure for generated project (#243) * changed the directory structure for generated project * changed test * upgraded commandline package * Fix test file locations on OSX (#235) * fix test file locations on OSX * changing to Path.Combine() * Additional Path.Combine() * Remove ConsoleCodeGeneratorTests.GeneratedTrainCodeTest.received.txt * Additional Path.Combine() * add back in double comparison fix * remove metrics agent NaN returns * test fix * test format fix * mock out path Thanks to @daholste for additional fixes! * upgrade to latest ML.NET public surface (#246) * Upgrade to ML.NET 0.11 (#247) * initial changes * fix lightgbm * changed normalize method * added tests * fix tests * fix test * Private preview final API changes (#250) * .NET framework design guidelines applied to public surface * WhitelistedTrainers -> Trainers * Add estimator to public API iteration result (#248) * LightGBM pipeline serialization fix (#251) * Change order that we search for TextLoader's parameters (#256) * CLI IFileInfo null exception fix (#254) * Averaged Perceptron pipeline serialization fix (#257) * Upgrade command-line-api and default folder name change (#258) * change in defautl folderName * upgrade command line * Update src/mlnet/Program.cs Co-Authored-By: srsaggam <41802116+srsaggam@users.noreply.github.com> * eliminate IFileInfo from CLI (#260) * Rev samples towards private preview; ignored columns fix (#259) * remove unused methods in consolehelper and nit picks in generated code (#261) * nit picks * change in console helper * fix tests * add space * fix tests * added nuget sources in generated csproj (#262) * added nuget sources in csproj * changed the structure in generated code * space * upgrade to mlnet 0.11 (#263) * Formatting CLI metrics (#264) Ensures space between printed metrics (also model counter). Right aligned metrics. Extended AUC to four digits. * Add implementation of non -ova multi class trainers code gen (#267) * added non ova multi class learners * added tests * test cases * Add caching (#249) * AdvancedExperimentSettings sample nits (#265) * Add sampling key column (#268) * Initial work for multi-class classification support for CLI (#226) * Initial work for multi-class classification support for CLI * String updates * more strings * Whitelist non-OVA multi-class learners * Refactor the orchestration of AutoML calls (#272) * Do not auto-group columns with suggested purpose = 'Ignore' (#273) * Fix: during type inferencing, parse whitespace strings as NaN (#271) * Printing additional metrics in CLI for binary classification (#274) * Printing additional metrics in CLI for binary classification * Update src/mlnet/Utilities/ConsolePrinter.cs * Add API option to store models on disk (instead of in memory); fix IEstimator memory leak (#269) * Print failed iterations in CLI (#275) * change the type to float from double (#277) * cache arg implementation in CLI (#280) * cache implementation * corrected the null case * added tests for all cases * Remove duplicate value-to-key mapping transform for multiclass string labels (#283) * Add post-trainer transform SDK infra; add KeyToValueMapping transform to CLI; fix: for generated multiclass models, convert predicted label from key to original label column type (#286) * Implement ignore columns command line arg (#290) * normalize line endings * added --ignore-columns * null checks * unit tests * Print winning iteration and runtime in CLI (#288) * Print best metric and runtime * Print best metric and runtime * Line endings in AutoMLEngine.cs * Rename time column to duration to match Python SDK * Revert to MicroAccuracy and MacroAccuracy spellings * Revert spelling of BinaryClassificationMetricsAgent to BinaryMetricsAgent to reduce merge conflicts * Revert spelling of MulticlassMetricsAgent to MultiMetricsAgent to reduce merge conflicts * missed some files * Fix merge conflict * Update AutoMLEngine.cs * Add MacOS & Linux to CI; MacOS & Linux test fixes (#293) * MicroAccuracy as default for multi-class (#295) Change default optimization metric for multi-class classification to MicroAccuracy (accuracy). Previously it was set to MacroAccuracy. * Null exception for ignorecolumns in CLI (#294) * Null exception for ignorecolumns in CLI * Check if ignore-columns array has values (as the default is now a empty array) * Emit caching flag in pipeline object model. (Includes SuggestedPipelineBuilder refactor & debug string fixes / refactor) (#296) * removed sln (#297) * Caching enabling in code gen part -2 (#298) * add * added caching codegen * support comma separated values for --ignore-columns (#300) * default initialization for ignore columns (#302) * default initialization * adde null check * Codegen for multiclass non-ova (#303) * changes to template * multicalss codegen * test cases * fix test cases * Generated Project new structure. (#305) * added new templates * writing files to disck * change path * added new templates * misisng braces * fix bugs * format code * added util methods for solution file creation and addition of projects to it * added extra packages to project files * new tests * added correct path for sln * build fix * fix build * include using system in prediction class (#307) * added using * fix test * Random number generator is not thread safe (#310) * Random number generator is not thread safe * Another local random generator * Missed a few references * Referncing AutoMlUtils.random instead of a local RNG * More refs to mail RNG; remove Float as per https://github.com/dotnet/machinelearning/issues/1669 * Missed Random.cs * Fix multiclass code gen (#314) * compile error in codegen * removes scores printing * fix bugs * fix test * Fix compile error in codegen project (#319) * removed redundant code * fix test case * Rev OVA pipeline node SDK output: wrap binary trainers as children inside parent OVA node (#317) * Ova Multi class codegen support (#321) * dummy * multiova implementation * fix tests * remove inclusion list * fix tests and console helper * Rev run result trainer name for OVA: output different trainer name for each OVA + binary learner combination (#322) * Rev run result trainer name for Ova: output different trainer name for each Ova + binary learner combination * test fixes * Console helper bug in generated code for multiclass (#323) * fix * fix test * looping perlogclass * fix test * Initial version of Progress bar impl and CLI UI experience (#325) * progressbar * added progressbar and refactoring * reverted * revert sign assembly * added headers and removed exception rethrow * Setting model directory to temp directory (#327) * Suggested changes to progress bar (#335) * progressbar * added progressbar and refactoring * reverted * revert sign assembly * added headers and removed exception rethrow * bug fixes and updates to UI * added friendly name printing for metric * formatting * Rev Samples (#334) * Telemetry2 (#333) * Create test.txt * Create test.txt * changes needed for benchmarking * forgot one file * merge conflict fix * fix build break * back out my version of the fix for Label column issue and fix the original fix * bogus file removal * undo SuggestedPipeline change * remove labelCol from pipeline suggester * fix build break * rename AutoML to Microsoft.ML.Auto everywhere and a shot at publishing nuget package (will probably need tweaks once I try to use the pipleline) * tweak queue in vsts-ci.yml * CLI telemetry implementation * Telemetry implementation * delete unnecessary file and change file size bucket to actually log log2 instead of nearest ceil value * add headers, remove comments * one more header missing * Fix progress bar in linux/osx (#336) * progressbar * added progressbar and refactoring * reverted * revert sign assembly * added headers and removed exception rethrow * bug fixes and updates to UI * added friendly name printing for metric * formatting * change from task to thread * Update src/mlnet/CodeGenerator/CodeGenerationHelper.cs Co-Authored-By: srsaggam <41802116+srsaggam@users.noreply.github.com> * Mem leak fix (#328) * Create test.txt * Create test.txt * changes needed for benchmarking * forgot one file * merge conflict fix * fix build break * back out my version of the fix for Label column issue and fix the original fix * bogus file removal * undo SuggestedPipeline change * remove labelCol from pipeline suggester * fix build break * rename AutoML to Microsoft.ML.Auto everywhere and a shot at publishing nuget package (will probably need tweaks once I try to use the pipleline) * tweak queue in vsts-ci.yml * there is still investigation to be done but this fix works and solves memory leak problems * minor refactor * Upgrade ML.NET package (#343) * Add cross-validation (CV), and auto-CV for small datasets; push common API experiment methods into base class (#287) * restore old yml for internal pipeline so we can publish nuget again to devdiv stream (#344) * Polishing the CLI UI part-1 (#338) * formatting of pbar message * Polishing the UI * optimization * rename variable * Update src/mlnet/AutoML/AutoMLEngine.cs Co-Authored-By: srsaggam <41802116+srsaggam@users.noreply.github.com> * Update src/mlnet/CodeGenerator/CodeGenerationHelper.cs Co-Authored-By: srsaggam <41802116+srsaggam@users.noreply.github.com> * new message * changed hhtp to https * added iteration num + 1 * change string name and add color to artifacts * change the message * build errors * added null checks * added exception messsages to log file * added exception messsages to log file * CLI ML.NET version upgrade (#345) * Sample revs; ColumnInformation property name revs; pre-featurizer fixes (#346) * CLI -- consume logs from AutoML SDK (#349) * Rename RunDetails --> RunDetail (#350) * command line api upgrade and progress bar rendering bug (#366) * added fix for all platforms progress bar * upgrade nuget * removed args from writeline * change in the version (#368) * fix few bugs in progressbar and verbosity (#374) * fix few bugs in progressbar and verbosity * removed unused name space * Fix for folders with space in it while generating project (#376) * support for folders with spaces * added support for paths with space * revert file * change name of var * remove spaces * SMAC fix for minimizing metrics (#363) * Formatting Regression metrics and progress bar display days. (#379) * added progress bar day display and fix regression metrics * fix formatting * added total time * formatted total time * change command name and add pbar message (#380) * change command name and add pbar message * fix tests * added aliases * duplicate alias * added another alias for task * UI missing features (#382) * added formatting changes * added accuracy specifically * downgrade the codepages (#384) * Change in project structure (#385) * initial changes * Change in project structure * correcting test * change variable name * fix tests * fix tests * fix more tests * fix codegen errors * adde log file message * changed name of args * change variable names * fix test * FileSizeBuckets in correct units (#387) * Minor telemetry change to log in correct units and make our life easier in the future * Use Ceiling instead of Round * changed order (#388) * prep work to transfer to ml.net (#389) * move test projects to top level test subdir * rename some projects to make naming consistent and make it build again * fix test project refs * Add AutoML components to build, fix issues related to that so it builds * fix test cases, remove AppInsights ref from AutoML (#3329) * [AutoML] disable netfx build leg for now (#3331) * disable netfx build leg for now * disable netfx build leg for now. * [AutoML] Add AutoML XML documentation to all public members; migrate AutoML projects & tests into ML.NET solution; AutoML test fixes (#3351) * [AutoML] Rev AutoML public API; add required native references to AutoML projects (#3364) * [AutoML] Minor changes to generated project in CLI based on feedback (#3371) * nitpicks for generated project * revert back the target framework * [AutoML] Migrate AutoML back to its own s…

* Fixed build errors resulting from upgrade to VS2019 compilers * Added additional message describing the previous fix * Syncing upstream fork (#10) * Throw error on incorrect Label name in InferColumns API (#47) * Added sequential grouping of columns * reverted the file * addded infer columns label name checking * added column detection error * removed unsed usings * added quotes * replace Where with Any clause * replace Where with Any clause * Set Nullable Auto params to null values (#50) * Added sequential grouping of columns * reverted the file * added auto params as null * change to the update fields method * First public api propsal (#52) * Includes following 1) Final proposal for 0.1 public API surface 2) Prefeaturization 3) Splitting train data into train and validate when validation data is null 4) Providing end to end samples one each for regression, binaryclassification and multiclass classification * Incorporating code review feedbacks * Revert "Set Nullable Auto params to null values" (#53) * Revert "First public api propsal (#52)" This reverts commit e4a64cf4aeab13ee9e5bf0efe242da3270241bd7. * Revert "Set Nullable Auto params to null values (#50)" This reverts commit 41c663cd14247d44022f40cf2dce5977dbab282d. * AutoFit return type is now an IEnumerable (#55) AutoFit returns is now an IEnumerable - this enables many good things Implementing variety of early stopping criteria (See sample) Early discard of models that are no good. This improves memory usage efficiency. (See sample) No need to implement a callback to get results back Getting best score is now outside of API implementation. It is a simple math function to compare scores (See sample). Also templatized the return type for better type safety through out the code. * misc fixes & test additions, towards 0.1 release (#56) * Enable UnitTests on build server (#57) * 1) Making trainer name public (#62) 2) Fixing up samples to reflect it * Initial version of CLI tool for mlnet (#61) * added global tool initial project * removed unneccesary files, renamed files * refactoring and added base abstract classes for trainer generator * removed unused class * Added classes for transforms * added transform generate dummy classes * more refactoring, added first transform * more refactoring and added classes * changed the project structure * restructing added options class * sln changes * refactored options to different class: * added more logic for code generation of class * misc changes * reverted file * added commandline api package * reverted sample * added new command line api parser * added normalization of column names * Added command defaults and error message * implementation of all trainers * changed auto to null * added all transform generators * added error handling when args is empty and minor changes due to change in AutoML api names * changed the name of param * added new command line options and restructuring code * renamed proj file and added solution * Added code to generate usings, Fixed few bugs in the code * added validation to the command line options * changed project name * Bug fixes due to API change in AutoML * changed directory structure * added test framework and basic tests * added more tests * added improvements to template and error handling * renamed the estimator name * fixed test case * added comments * added headers * changed namespace and removed unneccesary properties from project * Revert "changed namespace and removed unneccesary properties from project" This reverts commit 9edae033e9845e910f663f296e168f1182b84f5f. * fixed test cases and renamed namespaces * cleaned up proj file * added folder structure * added symbols/tokens for strings * added more tests * review comments * modified test cases * review comments * change in the exception message * normalized line endings * made method private static * simplified range building /optimization * minor fix * added header * added static methods in command where necessary * nit picks * made few methods static * review comments * nitpick * remove line pragmas * fix test case * Use better AutiFit overload and ignore Multiclass (#64) * Upgrading CLI to produce ML.NET V.10 APIs and bunch of Refactoring tasks (#65) * Added sequential grouping of columns * reverted the file * upgrade to v .10 and refactoring * added null check * fixed unit tests * review comments * removed the settings change * added regions * fixed unit tests * Upgrade ML.NET package to 0.10.0 (#70) * Change in template to accomodate new API of TextLoader (#72) * Added sequential grouping of columns * reverted the file * changed to new API of Text Loader * changed signature * added params for taking additional settings * changes to codegen params * refactoring of templates and fixing errors * Enable gated check for mlnet.tests (#79) * Added sequential grouping of columns * reverted the file * changed to new API of Text Loader * changed signature * added params for taking additional settings * changes to codegen params * refactoring of templates and fixing errors * added run-tests.proj and referred it in build.proj * CLI tool - make validation dataset optional and support for crossvalidation in generated code (#83) * Added sequential grouping of columns * reverted the file * bug fixes, more logic to templates to support cross-validate * formatting and fix type in consolehelper * Added logic in templates * revert settings * benchmarking related changes (#63) * Create test.txt * Create test.txt * changes needed for benchmarking * forgot one file * merge conflict fix * fix build break * back out my version of the fix for Label column issue and fix the original fix * bogus file removal * undo SuggestedPipeline change * remove labelCol from pipeline suggester * fix build break * fix fast forest learner (don't sweep over learning rate) (#88) * Made changes to Have non-calibrated scoring for binary classifiers (#86) * Added sequential grouping of columns * reverted the file * added calibration workaround * removed print probability * reverted settings * rev ColumnInference API: can take label index; rev output object types; add tests (#89) * rename AutoML to Microsoft.ML.Auto everywhere and a shot at publishing nuget package (#99) * Create test.txt * Create test.txt * changes needed for benchmarking * forgot one file * merge conflict fix * fix build break * back out my version of the fix for Label column issue and fix the original fix * bogus file removal * undo SuggestedPipeline change * remove labelCol from pipeline suggester * fix build break * rename AutoML to Microsoft.ML.Auto everywhere and a shot at publishing nuget package (will probably need tweaks once I try to use the pipleline) * publish nuget (#101) * use dotnet-internal-temp agent for internal build * use dotnet-internal feed * Fix Codegen for columnConvert and ValueToKeyMapping transform and add individual transform tests (#95) * Added sequential grouping of columns * reverted the file * fix usings for type convert * added transforms tests * review comments * When generating usings choose only distinct usings directives (#94) * Added sequential grouping of columns * reverted the file * Added code to have unique strings * refactoring * minor fix * minor fix * Autofit overloads + cancellation + progress callbacks 1) Introduce AutoFit overloads (basic and advanced) 2) AutoFit Cancellation 3) AutoFit progress callbacks * Default the kfolds to value 5 in CLI generated code (#115) * Added sequential grouping of columns * reverted the file * Set up CI with Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * remove file * added kfold param and defaulted to value * changed type * added for regression * Remove extra ; from generated code (#114) * Added sequential grouping of columns * reverted the file * Set up CI with Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * removed extra ; from generated code * removed file * fix unit tests * TimeoutInSeconds (#116) Specifying timeout in seconds instead of minutes * Added more command line args implementation to CLI tool and refactoring (#110) * Added sequential grouping of columns * reverted the file * Set up CI with Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * added git status * reverted change * added codegen options and refactoring * minor fixes' * renamed params, minor refactoring * added tests for commandline and refactoring * removed file * added back the test case * minor fixes * Update src/mlnet.Test/CommandLineTests.cs Co-Authored-By: srsaggam <41802116+srsaggam@users.noreply.github.com> * review comments * capitalize the first character * changed the name of test case * remove unused directives * Fail gracefully if unable to instantiate data view with swept parameters (#125) * gracefully fail if fail to parse a datai * rev * validate AutoFit 'Features' column must be of type R4 (#132) * Samples: exceptions / nits (#124) * Logging support in CLI + Implementation of cmd args [--name,--output,--verbosity] (#121) * addded logging and helper methods * fixing code after merge * added resx files, added logger framework, added logging messages * added new options * added spacing * minor fixes * change command description * rename option, add headers, include new param in test * formatted * build fix * changed option name * Added NlogConfig file * added back config package * fix tests * added correct validation check (#137) * Use CreateTextLoader<T>(..) instead of CreateTextLoader(..) (#138) * added support to loaddata by class in the generated code * fix tests * changed CreateTextLoader to ReadFromTextFile method. (#140) * changed textloader to readfromtextfile method * formatting * exception fixes (#136) * infer purpose of hidden columns as 'ignore' (#142) * Added approval tests and bunch of refactoring of code and normalizing namespaces (#148) * changed textloader to readfromtextfile method * formatting * added approval tests and refactoring of code * removed few comments * API 2.0 skeleton (#149) Incorporating API review feedback * The CV code should come before the training when there is no test dataset in generated code (#151) * reorder cv code * build fix * fixed structure * Format the generated code + bunch of misc tasks (#152) * added formatting and minor changes for reordering cv * fixing the template * minor changes * formatting changes * fixed approval test * removed unused nuget * added missing value replacing * added test for new transform * fix test * Update src/mlnet/Templates/Console/MLCodeGen.cs Co-Authored-By: srsaggam <41802116+srsaggam@users.noreply.github.com> * Sanitize the column names in CLI (#162) * added sanitization layer in CLI * fix test * changed exception.StackTrace to exception.ToString() * fix package name (#168) * Rev public API (#163) * Rename TransformGeneratorBase .cs to TransformGeneratorBase.cs (#153) * Fix minor version for the repository + remove Nlog config package (#171) * changed the minor version * removed the nlog config package * Added new test to columninfo and fixing up API (#178) * Make optimizing metric customizable and add trainer whitelist functionality (#172) * API rev (#181) * propagate root MLContext thru AutoML (instead of creating our own) (#182) * Enabling new command line args (#183) * fix package name * initial commit * added more commandline args * fixed tests * added headers * fix tests * fix test * rename 'AutoFitter' to 'Experiment' (#169) * added tests (#187) * rev InferColumns to accept ColumnInfo input param (#186) * Implement argument --has-header and change usage of dataset (#194) * added has header and fixed dataset and train dataset * fix tests * removed dummy command (#195) * Fix bug for regression and sanitize input label from user (#198) * removed dummy command * sanitize label and fix template * fix tests * Do not generate code concatenating columns when the dataset has a single feature column (#191) * Include some missed logging in the generated code. (#199) * added logging messages for generated code * added log messages * deleted file * cleaning up proj files (#185) * removed platform target * removed platform target * Some spaces and extra lines + bug in output path (#204) * nit picks * nit picks * fix test * accept label from user input and provide in generated code (#205) * Rev handling of weight / label columns (#203) * migrate to private ML.NET nuget for latest bug fixes (#131) * fix multiclass with nonstandard label (#207) * Multiclass nondefault label test (#208) * printing escaped chars + bug (#212) * delete unused internal samples (#211) * fix SMAC bug that causes multiclass sample to infinite loop (#209) * Rev user input validation for new API (#210) * added console message for exit and nit picks (#215) * exit when exception encountered (#216) * Seal API classes (and make EnableCaching internal) (#217) * Suggested sample nits (feel free to ask for any of these to be reverted) (#219) * User input column type validation (#218) * upgrade commandline and renaming (#221) * upgrade commandline and renaming * renaming fields * Make build.sh, init-tools.sh, & run.sh executable on OSX/Linux (#225) * CLI argument descriptions updated (#224) * CLI argument descriptions updated * No version in .csproj * added flag to disable training code (#227) * Exit if perfect model produced (#220) * removed header (#228) * removed header * added auto generated header * removed console read key (#229) * Fix model path in generated file (#230) * removed console read key * fix model path * fix test * reorder samples (#231) * remove rule that infers column purpose as categorical if # of distinct values is < 100 (#233) * Null reference exception fix for finding best model when some runs have failed (#239) * samples fixes (#238) * fix for defaulting Averaged Perceptron # of iterations to 10 (#237) * Bug bash feedback Feb 27. API changes and sample changes (#240) * Bug bash feedback Feb 27. API changes Sample changes Exception fix * Samples / API rev from 2/27 bug bash feedback (#242) * changed the directory structure for generated project (#243) * changed the directory structure for generated project * changed test * upgraded commandline package * Fix test file locations on OSX (#235) * fix test file locations on OSX * changing to Path.Combine() * Additional Path.Combine() * Remove ConsoleCodeGeneratorTests.GeneratedTrainCodeTest.received.txt * Additional Path.Combine() * add back in double comparison fix * remove metrics agent NaN returns * test fix * test format fix * mock out path Thanks to @daholste for additional fixes! * upgrade to latest ML.NET public surface (#246) * Upgrade to ML.NET 0.11 (#247) * initial changes * fix lightgbm * changed normalize method * added tests * fix tests * fix test * Private preview final API changes (#250) * .NET framework design guidelines applied to public surface * WhitelistedTrainers -> Trainers * Add estimator to public API iteration result (#248) * LightGBM pipeline serialization fix (#251) * Change order that we search for TextLoader's parameters (#256) * CLI IFileInfo null exception fix (#254) * Averaged Perceptron pipeline serialization fix (#257) * Upgrade command-line-api and default folder name change (#258) * change in defautl folderName * upgrade command line * Update src/mlnet/Program.cs Co-Authored-By: srsaggam <41802116+srsaggam@users.noreply.github.com> * eliminate IFileInfo from CLI (#260) * Rev samples towards private preview; ignored columns fix (#259) * remove unused methods in consolehelper and nit picks in generated code (#261) * nit picks * change in console helper * fix tests * add space * fix tests * added nuget sources in generated csproj (#262) * added nuget sources in csproj * changed the structure in generated code * space * upgrade to mlnet 0.11 (#263) * Formatting CLI metrics (#264) Ensures space between printed metrics (also model counter). Right aligned metrics. Extended AUC to four digits. * Add implementation of non -ova multi class trainers code gen (#267) * added non ova multi class learners * added tests * test cases * Add caching (#249) * AdvancedExperimentSettings sample nits (#265) * Add sampling key column (#268) * Initial work for multi-class classification support for CLI (#226) * Initial work for multi-class classification support for CLI * String updates * more strings * Whitelist non-OVA multi-class learners * Refactor the orchestration of AutoML calls (#272) * Do not auto-group columns with suggested purpose = 'Ignore' (#273) * Fix: during type inferencing, parse whitespace strings as NaN (#271) * Printing additional metrics in CLI for binary classification (#274) * Printing additional metrics in CLI for binary classification * Update src/mlnet/Utilities/ConsolePrinter.cs * Add API option to store models on disk (instead of in memory); fix IEstimator memory leak (#269) * Print failed iterations in CLI (#275) * change the type to float from double (#277) * cache arg implementation in CLI (#280) * cache implementation * corrected the null case * added tests for all cases * Remove duplicate value-to-key mapping transform for multiclass string labels (#283) * Add post-trainer transform SDK infra; add KeyToValueMapping transform to CLI; fix: for generated multiclass models, convert predicted label from key to original label column type (#286) * Implement ignore columns command line arg (#290) * normalize line endings * added --ignore-columns * null checks * unit tests * Print winning iteration and runtime in CLI (#288) * Print best metric and runtime * Print best metric and runtime * Line endings in AutoMLEngine.cs * Rename time column to duration to match Python SDK * Revert to MicroAccuracy and MacroAccuracy spellings * Revert spelling of BinaryClassificationMetricsAgent to BinaryMetricsAgent to reduce merge conflicts * Revert spelling of MulticlassMetricsAgent to MultiMetricsAgent to reduce merge conflicts * missed some files * Fix merge conflict * Update AutoMLEngine.cs * Add MacOS & Linux to CI; MacOS & Linux test fixes (#293) * MicroAccuracy as default for multi-class (#295) Change default optimization metric for multi-class classification to MicroAccuracy (accuracy). Previously it was set to MacroAccuracy. * Null exception for ignorecolumns in CLI (#294) * Null exception for ignorecolumns in CLI * Check if ignore-columns array has values (as the default is now a empty array) * Emit caching flag in pipeline object model. (Includes SuggestedPipelineBuilder refactor & debug string fixes / refactor) (#296) * removed sln (#297) * Caching enabling in code gen part -2 (#298) * add * added caching codegen * support comma separated values for --ignore-columns (#300) * default initialization for ignore columns (#302) * default initialization * adde null check * Codegen for multiclass non-ova (#303) * changes to template * multicalss codegen * test cases * fix test cases * Generated Project new structure. (#305) * added new templates * writing files to disck * change path * added new templates * misisng braces * fix bugs * format code * added util methods for solution file creation and addition of projects to it * added extra packages to project files * new tests * added correct path for sln * build fix * fix build * include using system in prediction class (#307) * added using * fix test * Random number generator is not thread safe (#310) * Random number generator is not thread safe * Another local random generator * Missed a few references * Referncing AutoMlUtils.random instead of a local RNG * More refs to mail RNG; remove Float as per https://github.com/dotnet/machinelearning/issues/1669 * Missed Random.cs * Fix multiclass code gen (#314) * compile error in codegen * removes scores printing * fix bugs * fix test * Fix compile error in codegen project (#319) * removed redundant code * fix test case * Rev OVA pipeline node SDK output: wrap binary trainers as children inside parent OVA node (#317) * Ova Multi class codegen support (#321) * dummy * multiova implementation * fix tests * remove inclusion list * fix tests and console helper * Rev run result trainer name for OVA: output different trainer name for each OVA + binary learner combination (#322) * Rev run result trainer name for Ova: output different trainer name for each Ova + binary learner combination * test fixes * Console helper bug in generated code for multiclass (#323) * fix * fix test * looping perlogclass * fix test * Initial version of Progress bar impl and CLI UI experience (#325) * progressbar * added progressbar and refactoring * reverted * revert sign assembly * added headers and removed exception rethrow * Setting model directory to temp directory (#327) * Suggested changes to progress bar (#335) * progressbar * added progressbar and refactoring * reverted * revert sign assembly * added headers and removed exception rethrow * bug fixes and updates to UI * added friendly name printing for metric * formatting * Rev Samples (#334) * Telemetry2 (#333) * Create test.txt * Create test.txt * changes needed for benchmarking * forgot one file * merge conflict fix * fix build break * back out my version of the fix for Label column issue and fix the original fix * bogus file removal * undo SuggestedPipeline change * remove labelCol from pipeline suggester * fix build break * rename AutoML to Microsoft.ML.Auto everywhere and a shot at publishing nuget package (will probably need tweaks once I try to use the pipleline) * tweak queue in vsts-ci.yml * CLI telemetry implementation * Telemetry implementation * delete unnecessary file and change file size bucket to actually log log2 instead of nearest ceil value * add headers, remove comments * one more header missing * Fix progress bar in linux/osx (#336) * progressbar * added progressbar and refactoring * reverted * revert sign assembly * added headers and removed exception rethrow * bug fixes and updates to UI * added friendly name printing for metric * formatting * change from task to thread * Update src/mlnet/CodeGenerator/CodeGenerationHelper.cs Co-Authored-By: srsaggam <41802116+srsaggam@users.noreply.github.com> * Mem leak fix (#328) * Create test.txt * Create test.txt * changes needed for benchmarking * forgot one file * merge conflict fix * fix build break * back out my version of the fix for Label column issue and fix the original fix * bogus file removal * undo SuggestedPipeline change * remove labelCol from pipeline suggester * fix build break * rename AutoML to Microsoft.ML.Auto everywhere and a shot at publishing nuget package (will probably need tweaks once I try to use the pipleline) * tweak queue in vsts-ci.yml * there is still investigation to be done but this fix works and solves memory leak problems * minor refactor * Upgrade ML.NET package (#343) * Add cross-validation (CV), and auto-CV for small datasets; push common API experiment methods into base class (#287) * restore old yml for internal pipeline so we can publish nuget again to devdiv stream (#344) * Polishing the CLI UI part-1 (#338) * formatting of pbar message * Polishing the UI * optimization * rename variable * Update src/mlnet/AutoML/AutoMLEngine.cs Co-Authored-By: srsaggam <41802116+srsaggam@users.noreply.github.com> * Update src/mlnet/CodeGenerator/CodeGenerationHelper.cs Co-Authored-By: srsaggam <41802116+srsaggam@users.noreply.github.com> * new message * changed hhtp to https * added iteration num + 1 * change string name and add color to artifacts * change the message * build errors * added null checks * added exception messsages to log file * added exception messsages to log file * CLI ML.NET version upgrade (#345) * Sample revs; ColumnInformation property name revs; pre-featurizer fixes (#346) * CLI -- consume logs from AutoML SDK (#349) * Rename RunDetails --> RunDetail (#350) * command line api upgrade and progress bar rendering bug (#366) * added fix for all platforms progress bar * upgrade nuget * removed args from writeline * change in the version (#368) * fix few bugs in progressbar and verbosity (#374) * fix few bugs in progressbar and verbosity * removed unused name space * Fix for folders with space in it while generating project (#376) * support for folders with spaces * added support for paths with space * revert file * change name of var * remove spaces * SMAC fix for minimizing metrics (#363) * Formatting Regression metrics and progress bar display days. (#379) * added progress bar day display and fix regression metrics * fix formatting * added total time * formatted total time * change command name and add pbar message (#380) * change command name and add pbar message * fix tests * added aliases * duplicate alias * added another alias for task * UI missing features (#382) * added formatting changes * added accuracy specifically * downgrade the codepages (#384) * Change in project structure (#385) * initial changes * Change in project structure * correcting test * change variable name * fix tests * fix tests * fix more tests * fix codegen errors * adde log file message * changed name of args * change variable names * fix test * FileSizeBuckets in correct units (#387) * Minor telemetry change to log in correct units and make our life easier in the future * Use Ceiling instead of Round * changed order (#388) * prep work to transfer to ml.net (#389) * move test projects to top level test subdir * rename some projects to make naming consistent and make it build again * fix test project refs * Add AutoML components to build, fix issues related to that so it builds * fix test cases, remove AppInsights ref from AutoML (#3329) * [AutoML] disable netfx build leg for now (#3331) * disable netfx build leg for now * disable netfx build leg for now. * [AutoML] Add AutoML XML documentation to all public members; migrate AutoML projects & tests into ML.NET solution; AutoML test fixes (#3351) * [AutoML] Rev AutoML public API; add required native references to AutoML projects (#3364) * [AutoML] Minor changes to generated project in CLI based on feedback (#3371) * nitpicks for generated project * revert back the target framework * [AutoML] Migrate AutoML back to its own solution, w/ NuGet dependencies (#3373) * Migrate AutoML back to its own solution, w/ NuGet dependencies * build project updates; parameter name revert * dummy change * Revert "dummy change" This reverts commit 3e8574266f556a4d5b6805eb55b4d8b8b84cf355. * [AutoML] publish AutoML package (#3383) * publish AutoML package * Only leave automl and mlnet tests to run * publish AutoML package * Only leave automl and mlnet tests to run * fix build issues when ml.net is not building * bump version to 0.3 since that's the one we're going to ship for build (#3416) * [AutoML] temporarily disable all but x64 platforms -- don't want to do native builds and can't find a way around that with the current VSTS pipeline (#3420) * disable steps but keep phases to keep vsts build pipeline happy (#3423) * API docs for experimentation (#3484) * fixed path bug and regression metrics correction (#3504) * changed the casing of option alias as it conflicts with --help (#3554) * [AutoML] Generated project - FastTree nuget package inclusion dynamically (#3567) * added support for fast tree nuget pack inclusion in generated project * fix testcase * changed the tool name in telemetry message * dummy commit * remove space * dummy commit to trigger build * [AutoML] Add AutoML example code (#3458) * AutoML PipelineSuggester: don't recommend pipelines from first-stage trainers that failed (#3593) * InferColumns API: Validate all columns specified in column info exist in inferred data view (#3599) * [AutoML] AutoML SDK API: validate schema types of input IDataView (#3597) * [AutoML] If first three iterations all fail, short-circuit AutoML experiment (#3591) * mlnet CLI nupkg creation/signing (#3606) * mlnet CLI nupkg creation/signing * relmove includeinpackage from mlnet csproj * address PR comments -- some minor reshuffling of stuff * publish symbols for mlnet CLI * fix case in NLog.config * [AutoML] rename Auto to AutoML in namespace and nuget (#3609) * mlnet CLI nupkg creation/signing * [AutoML] take dependency on a specific ml.net version (#3610) * take dependency on a specific ml.net version * catch up to spelling fix for OptimizationTolerance * force a specific ml.net nuget version, fix typo (#3616) * [AutoML] Fix error handling in CLI. (#3618) * fix error handling * renaming variables * [AutoML] turn off line pragmas in .tt files to play nice with signing (#3617) * turn off line pragmas in .tt files to play nice with signing * dedupe tags * change the param name (#3619) * [AutoML] return null instead of null ref crash on Model property accessor (#3620) * return null instead of null ref crash on Model property accessor * [AutoML] Handling label column names which have space and exception logging (#3624) * fix case of label with space and exception logging * final handler * revert file * use Name instead of FullName for telemetry filename hash (#3633) * renamed classes (#3634) * change ML.NET dependency to 1.0 (#3639) [AutoML] undo pinning ML.NET dependency * set exploration time default in CLI to half hour (#3640) * [AutoML] step 2 of removing pinned nupkg versions (#3642) * InferColumns API that consumes label column index -- Only rename label column to 'Label' for headerless files (#3643) * [AutoML] Upgrade ml.net package in generated code (#3644) * upgrade the mlnet package in gen code * Update src/mlnet/Templates/Console/ModelProject.cs Co-Authored-By: srsaggam <41802116+srsaggam@users.noreply.github.com> * Update src/mlnet/Templates/Console/ModelProject.tt Co-Authored-By: srsaggam <41802116+srsaggam@users.noreply.github.com> * added spaces * [AutoML] Early stopping in CLI based on the exploration time (#3641) * early stopping in CLI * remove unused variables * change back to thread * remove sleep * fix review comments * remove ununsed usings * format message * collapse declaration * remove unused param * added environment.exit and removal of error message * correction in message * secs-> seconds * exit code * change value to 1 * reverse the declaration * [AutoML] Change wording for CouldNotFinshOnTime message (#3655) * set exploration time default in CLI to half hour * [AutoML] Change wording for CouldNotFinshOnTime message * [AutoML] Change wording for CouldNotFinshOnTime message * even better wording for CouldNotFinshOnTime * temp change to get around vsts publish failure (#3656) * [AutoML] bump version to 0.4.0 (#3658) * implement culture invariant strings (#3725) * reset culture (#3730) * [AutoML] Cross validation fixes; validate empty training / validation input data (#3794) * [AutoML] Enable style cop rules & resolve errors (#3823) * add task agnostic wrappers for autofit calls (#3860) * [AutoML] CLI telemetry rev (#3789) * delete automl .sln * CLI -- regenerate templated CS files (#3954) * [AutoML] Bump ML.NET package version to 1.2.0 in AutoML API and CLI; and AutoML package versions to 0.14.0 (#3958) * Build AutoML NuGet package (#3961) * Increment AutoML build version to 0.15.0 for preview. (#3968) * added culture independent parsing (#3731) * - convert tests to xunit - take project level dependency on ML.NET components instead of nuget - set up bestfriends relationship to ML.Core and remove some of the copies of util classes from AutoML.NET (more work needed to fully remove them, work item 4064) - misc build script changes to address PR comments * address issues only showing up in a couple configurations during CI build * fix cut&paste error * [AutoML] Bump version to ML.NET 1.3.1 in AutoML API and CLI and AutoML package version to 0.15.1 (#4071) * bumped version * change versions in nupkg * revert version bump in branch props * [AutoML] Fix for Exception thrown in cross val when one of the score equals infinity. (#4073) * bumped version * change versions in nupkg * revert version bump in branch props * added infinity fix * changes signing (#4079) * Addressed PR comments and build issues - sync block on creating test data file (failed intermittently) - removed classes we copied over from ML.Core and fixed their uses to de-dupe and use original ML.Core versions since we now have InternalsVisible and BestFriends - Fixed nupkg creation to use projects insted of public nuget version for AutoML - Fixed a bunch of unit tests that didn't actually test what they were supposed to test, while removing cut&past code and dependencies. - Few more misc small changes * minor nit - removed unused folder ref * Fix the .sln file for the right configurations. * Fix mistake in .sln file * test fixes and disable one test * fix tests, re-add AutoML samples csproj * bumped VS version to 16 in .sln, removed InternalsVisible for a dead assembly, removed unused references from AutoML test project * Updated docs to include PredictedLabel member (#4107) * Fixed build errors resulting from upgrade to VS2019 compilers * Added additional message describing the previous fix * Updated docs to include PredictedLabel member * Added CODEOWNERS file in the .github/ folder. (#4140) * Added CODEOWNERS file in the .github/ folder. This allows reviewers to review any changes in the machine learning repository * Updated .github/CODEOWNERS with the team instead of individual reviewers * Added AutoML team reviewers (#4144) * Added CODEOWNERS file in the .github/ folder. This allows reviewers to review any changes in the machine learning repository * Updated .github/CODEOWNERS with the team instead of individual reviewers * Added AutoML team reviwers to files owned by AutoML team * Added AutoML team reviwers to files owned by AutoML team * Removed two files that don't exist for AutoML team in CODEOWNERS * Build extension method to reload changes without specifying model name (#4146) * Image classification preview 2. (#4151) * Image classification preview 2. * PR feedback. * Add unit-test. * Add unit-test. * Add unit-test. * Add unit-test. * Use Path.Combine instead of Join. * fix test dataset path. * fix test dataset path. * Improve test. * Improve test. * Increase epochs in tests. * Disable test on Ubuntu. * Move test to its own project. * Move test to its own project. * Move test to its own project. * Move test to its own file. * cleanup. * Disable parallel execution of tensorflow tests. * PR feedback. * PR feedback. * PR feedback. * PR feedback. * Prevent TF test to execute in parallel. * PR feedback. * Build error. * clean up. * Syncing upstream fork (#11) * Throw error on incorrect Label name in InferColumns API (#47) * Added sequential grouping of columns * reverted the file * addded infer columns label name checking * added column detection error * removed unsed usings * added quotes * replace Where with Any clause * replace Where with Any clause * Set Nullable Auto params to null values (#50) * Added sequential grouping of columns * reverted the file * added auto params as null * change to the update fields method * First public api propsal (#52) * Includes following 1) Final proposal for 0.1 public API surface 2) Prefeaturization 3) Splitting train data into train and validate when validation data is null 4) Providing end to end samples one each for regression, binaryclassification and multiclass classification * Incorporating code review feedbacks * Revert "Set Nullable Auto params to null values" (#53) * Revert "First public api propsal (#52)" This reverts commit e4a64cf4aeab13ee9e5bf0efe242da3270241bd7. * Revert "Set Nullable Auto params to null values (#50)" This reverts commit 41c663cd14247d44022f40cf2dce5977dbab282d. * AutoFit return type is now an IEnumerable (#55) AutoFit returns is now an IEnumerable - this enables many good things Implementing variety of early stopping criteria (See sample) Early discard of models that are no good. This improves memory usage efficiency. (See sample) No need to implement a callback to get results back Getting best score is now outside of API implementation. It is a simple math function to compare scores (See sample). Also templatized the return type for better type safety through out the code. * misc fixes & test additions, towards 0.1 release (#56) * Enable UnitTests on build server (#57) * 1) Making trainer name public (#62) 2) Fixing up samples to reflect it * Initial version of CLI tool for mlnet (#61) * added global tool initial project * removed unneccesary files, renamed files * refactoring and added base abstract classes for trainer generator * removed unused class * Added classes for transforms * added transform generate dummy classes * more refactoring, added first transform * more refactoring and added classes * changed the project structure * restructing added options class * sln changes * refactored options to different class: * added more logic for code generation of class * misc changes * reverted file * added commandline api package * reverted sample * added new command line api parser * added normalization of column names * Added command defaults and error message * implementation of all trainers * changed auto to null * added all transform generators * added error handling when args is empty and minor changes due to change in AutoML api names * changed the name of param * added new command line options and restructuring code * renamed proj file and added solution * Added code to generate usings, Fixed few bugs in the code * added validation to the command line options * changed project name * Bug fixes due to API change in AutoML * changed directory structure * added test framework and basic tests * added more tests * added improvements to template and error handling * renamed the estimator name * fixed test case * added comments * added headers * changed namespace and removed unneccesary properties from project * Revert "changed namespace and removed unneccesary properties from project" This reverts commit 9edae033e9845e910f663f296e168f1182b84f5f. * fixed test cases and renamed namespaces * cleaned up proj file * added folder structure * added symbols/tokens for strings * added more tests * review comments * modified test cases * review comments * change in the exception message * normalized line endings * made method private static * simplified range building /optimization * minor fix * added header * added static methods in command where necessary * nit picks * made few methods static * review comments * nitpick * remove line pragmas * fix test case * Use better AutiFit overload and ignore Multiclass (#64) * Upgrading CLI to produce ML.NET V.10 APIs and bunch of Refactoring tasks (#65) * Added sequential grouping of columns * reverted the file * upgrade to v .10 and refactoring * added null check * fixed unit tests * review comments * removed the settings change * added regions * fixed unit tests * Upgrade ML.NET package to 0.10.0 (#70) * Change in template to accomodate new API of TextLoader (#72) * Added sequential grouping of columns * reverted the file * changed to new API of Text Loader * changed signature * added params for taking additional settings * changes to codegen params * refactoring of templates and fixing errors * Enable gated check for mlnet.tests (#79) * Added sequential grouping of columns * reverted the file * changed to new API of Text Loader * changed signature * added params for taking additional settings * changes to codegen params * refactoring of templates and fixing errors * added run-tests.proj and referred it in build.proj * CLI tool - make validation dataset optional and support for crossvalidation in generated code (#83) * Added sequential grouping of columns * reverted the file * bug fixes, more logic to templates to support cross-validate * formatting and fix type in consolehelper * Added logic in templates * revert settings * benchmarking related changes (#63) * Create test.txt * Create test.txt * changes needed for benchmarking * forgot one file * merge conflict fix * fix build break * back out my version of the fix for Label column issue and fix the original fix * bogus file removal * undo SuggestedPipeline change * remove labelCol from pipeline suggester * fix build break * fix fast forest learner (don't sweep over learning rate) (#88) * Made changes to Have non-calibrated scoring for binary classifiers (#86) * Added sequential grouping of columns * reverted the file * added calibration workaround * removed print probability * reverted settings * rev ColumnInference API: can take label index; rev output object types; add tests (#89) * rename AutoML to Microsoft.ML.Auto everywhere and a shot at publishing nuget package (#99) * Create test.txt * Create test.txt * changes needed for benchmarking * forgot one file * merge conflict fix * fix build break * back out my version of the fix for Label column issue and fix the original fix * bogus file removal * undo SuggestedPipeline change * remove labelCol from pipeline suggester * fix build break * rename AutoML to Microsoft.ML.Auto everywhere and a shot at publishing nuget package (will probably need tweaks once I try to use the pipleline) * publish nuget (#101) * use dotnet-internal-temp agent for internal build * use dotnet-internal feed * Fix Codegen for columnConvert and ValueToKeyMapping transform and add individual transform tests (#95) * Added sequential grouping of columns * reverted the file * fix usings for type convert * added transforms tests * review comments * When generating usings choose only distinct usings directives (#94) * Added sequential grouping of columns * reverted the file * Added code to have unique strings * refactoring * minor fix * minor fix * Autofit overloads + cancellation + progress callbacks 1) Introduce AutoFit overloads (basic and advanced) 2) AutoFit Cancellation 3) AutoFit progress callbacks * Default the kfolds to value 5 in CLI generated code (#115) * Added sequential grouping of columns * reverted the file * Set up CI with Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * remove file * added kfold param and defaulted to value * changed type * added for regression * Remove extra ; from generated code (#114) * Added sequential grouping of columns * reverted the file * Set up CI with Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * removed extra ; from generated code * removed file * fix unit tests * TimeoutInSeconds (#116) Specifying timeout in seconds instead of minutes * Added more command line args implementation to CLI tool and refactoring (#110) * Added sequential grouping of columns * reverted the file * Set up CI with Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * added git status * reverted change * added codegen options and refactoring * minor fixes' * renamed params, minor refactoring * added tests for commandline and refactoring * removed file * added back the test case * minor fixes * Update src/mlnet.Test/CommandLineTests.cs Co-Authored-By: srsaggam <41802116+srsaggam@users.noreply.github.com> * review comments * capitalize the first character * changed the name of test case * remove unused directives * Fail gracefully if unable to instantiate data view with swept parameters (#125) * gracefully fail if fail to parse a datai * rev * validate AutoFit 'Features' column must be of type R4 (#132) * Samples: exceptions / nits (#124) * Logging support in CLI + Implementation of cmd args [--name,--output,--verbosity] (#121) * addded logging and helper methods * fixing code after merge * added resx files, added logger framework, added logging messages * added new options * added spacing * minor fixes * change command description * rename option, add headers, include new param in test * formatted * build fix * changed option name * Added NlogConfig file * added back config package * fix tests * added correct validation check (#137) * Use CreateTextLoader<T>(..) instead of CreateTextLoader(..) (#138) * added support to loaddata by class in the generated code * fix tests * changed CreateTextLoader to ReadFromTextFile method. (#140) * changed textloader to readfromtextfile method * formatting * exception fixes (#136) * infer purpose of hidden columns as 'ignore' (#142) * Added approval tests and bunch of refactoring of code and normalizing namespaces (#148) * changed textloader to readfromtextfile method * formatting * added approval tests and refactoring of code * removed few comments * API 2.0 skeleton (#149) Incorporating API review feedback * The CV code should come before the training when there is no test dataset in generated code (#151) * reorder cv code * build fix * fixed structure * Format the generated code + bunch of misc tasks (#152) * added formatting and minor changes for reordering cv * fixing the template * minor changes * formatting changes * fixed approval test * removed unused nuget * added missing value replacing * added test for new transform * fix test * Update src/mlnet/Templates/Console/MLCodeGen.cs Co-Authored-By: srsaggam <41802116+srsaggam@users.noreply.github.com> * Sanitize the column names in CLI (#162) * added sanitization layer in CLI * fix test * changed exception.StackTrace to exception.ToString() * fix package name (#168) * Rev public API (#163) * Rename TransformGeneratorBase .cs to TransformGeneratorBase.cs (#153) * Fix minor version for the repository + remove Nlog config package (#171) * changed the minor version * removed the nlog config package * Added new test to columninfo and fixing up API (#178) * Make optimizing metric customizable and add trainer whitelist functionality (#172) * API rev (#181) * propagate root MLContext thru AutoML (instead of creating our own) (#182) * Enabling new command line args (#183) * fix package name * initial commit * added more commandline args * fixed tests * added headers * fix tests * fix test * rename 'AutoFitter' to 'Experiment' (#169) * added tests (#187) * rev InferColumns to accept ColumnInfo input param (#186) * Implement argument --has-header and change usage of dataset (#194) * added has header and fixed dataset and train dataset * fix tests * removed dummy command (#195) * Fix bug for regression and sanitize input label from user (#198) * removed dummy command * sanitize label and fix template * fix tests * Do not generate code concatenating columns when the dataset has a single feature column (#191) * Include some missed logging in the generated code. (#199) * added logging messages for generated code * added log messages * deleted file * cleaning up proj files (#185) * removed platform target * removed platform target * Some spaces and extra lines + bug in output path (#204) * nit picks * nit picks * fix test * accept label from user input and provide in generated code (#205) * Rev handling of weight / label columns (#203) * migrate to private ML.NET nuget for latest bug fixes (#131) * fix multiclass with nonstandard label (#207) * Multiclass nondefault label test (#208) * printing escaped chars + bug (#212) * delete unused internal samples (#211) * fix SMAC bug that causes multiclass sample to infinite loop (#209) * Rev user input validation for new API (#210) * added console message for exit and nit picks (#215) * exit when exception encountered (#216) * Seal API classes (and make EnableCaching internal) (#217) * Suggested sample nits (feel free to ask for any of these to be reverted) (#219) * User input column type validation (#218) * upgrade commandline and renaming (#221) * upgrade commandline and renaming * renaming fields * Make build.sh, init-tools.sh, & run.sh executable on OSX/Linux (#225) * CLI argument descriptions updated (#224) * CLI argument descriptions updated * No version in .csproj * added flag to disable training code (#227) * Exit if perfect model produced (#220) * removed header (#228) * removed header * added auto generated header * removed console read key (#229) * Fix model path in generated file (#230) * removed console read key * fix model path * fix test * reorder samples (#231) * remove rule that infers column purpose as categorical if # of distinct values is < 100 (#233) * Null reference exception fix for finding best model when some runs have failed (#239) * samples fixes (#238) * fix for defaulting Averaged Perceptron # of iterations to 10 (#237) * Bug bash feedback Feb 27. API changes and sample changes (#240) * Bug bash feedback Feb 27. API changes Sample changes Exception fix * Samples / API rev from 2/27 bug bash feedback (#242) * changed the directory structure for generated project (#243) * changed the directory structure for generated project * changed test * upgraded commandline package * Fix test file locations on OSX (#235) * fix test file locations on OSX * changing to Path.Combine() * Additional Path.Combine() * Remove ConsoleCodeGeneratorTests.GeneratedTrainCodeTest.received.txt * Additional Path.Combine() * add back in double comparison fix * remove metrics agent NaN returns * test fix * test format fix * mock out path Thanks to @daholste for additional fixes! * upgrade to latest ML.NET public surface (#246) * Upgrade to ML.NET 0.11 (#247) * initial changes * fix lightgbm * changed normalize method * added tests * fix tests * fix test * Private preview final API changes (#250) * .NET framework design guidelines applied to public surface * WhitelistedTrainers -> Trainers * Add estimator to public API iteration result (#248) * LightGBM pipeline serialization fix (#251) * Change order that we search for TextLoader's parameters (#256) * CLI IFileInfo null exception fix (#254) * Averaged Perceptron pipeline serialization fix (#257) * Upgrade command-line-api and default folder name change (#258) * change in defautl folderName * upgrade command line * Update src/mlnet/Program.cs Co-Authored-By: srsaggam <41802116+srsaggam@users.noreply.github.com> * eliminate IFileInfo from CLI (#260) * Rev samples towards private preview; ignored columns fix (#259) * remove unused methods in consolehelper and nit picks in generated code (#261) * nit picks * change in console helper * fix tests * add space * fix tests * added nuget sources in generated csproj (#262) * added nuget sources in csproj * changed the structure in generated code * space * upgrade to mlnet 0.11 (#263) * Formatting CLI metrics (#264) Ensures space between printed metrics (also model counter). Right aligned metrics. Extended AUC to four digits. * Add implementation of non -ova multi class trainers code gen (#267) * added non ova multi class learners * added tests * test cases * Add caching (#249) * AdvancedExperimentSettings sample nits (#265) * Add sampling key column (#268) * Initial work for multi-class classification support for CLI (#226) * Initial work for multi-class classification support for CLI * String updates * more strings * Whitelist non-OVA multi-class learners * Refactor the orchestration of AutoML calls (#272) * Do not auto-group columns with suggested purpose = 'Ignore' (#273) * Fix: during type inferencing, parse whitespace strings as NaN (#271) * Printing additional metrics in CLI for binary classification (#274) * Printing additional metrics in CLI for binary classification * Update src/mlnet/Utilities/ConsolePrinter.cs * Add API option to store models on disk (instead of in memory); fix IEstimator memory leak (#269) * Print failed iterations in CLI (#275) * change the type to float from double (#277) * cache arg implementation in CLI (#280) * cache implementation * corrected the null case * added tests for all cases * Remove duplicate value-to-key mapping transform for multiclass string labels (#283) * Add post-trainer transform SDK infra; add KeyToValueMapping transform to CLI; fix: for generated multiclass models, convert predicted label from key to original label column type (#286) * Implement ignore columns command line arg (#290) * normalize line endings * added --ignore-columns * null checks * unit tests * Print winning iteration and runtime in CLI (#288) * Print best metric and runtime * Print best metric and runtime * Line endings in AutoMLEngine.cs * Rename time column to duration to match Python SDK * Revert to MicroAccuracy and MacroAccuracy spellings * Revert spelling of BinaryClassificationMetricsAgent to BinaryMetricsAgent to reduce merge conflicts * Revert spelling of MulticlassMetricsAgent to MultiMetricsAgent to reduce merge conflicts * missed some files * Fix merge conflict * Update AutoMLEngine.cs * Add MacOS & Linux to CI; MacOS & Linux test fixes (#293) * MicroAccuracy as default for multi-class (#295) Change default optimization metric for multi-class classification to MicroAccuracy (accuracy). Previously it was set to MacroAccuracy. * Null exception for ignorecolumns in CLI (#294) * Null exception for ignorecolumns in CLI * Check if ignore-columns array has values (as the default is now a empty array) * Emit caching flag in pipeline object model. (Includes SuggestedPipelineBuilder refactor & debug string fixes / refactor) (#296) * removed sln (#297) * Caching enabling in code gen part -2 (#298) * add * added caching codegen * support comma separated values for --ignore-columns (#300) * default initialization for ignore columns (#302) * default initialization * adde null check * Codegen for multiclass non-ova (#303) * changes to template * multicalss codegen * test cases * fix test cases * Generated Project new structure. (#305) * added new templates * writing files to disck * change path * added new templates * misisng braces * fix bugs * format code * added util methods for solution file creation and addition of projects to it * added extra packages to project files * new tests * added correct path for sln * build fix * fix build * include using system in prediction class (#307) * added using * fix test * Random number generator is not thread safe (#310) * Random number generator is not thread safe * Another local random generator * Missed a few references * Referncing AutoMlUtils.random instead of a local RNG * More refs to mail RNG; remove Float as per https://github.com/dotnet/machinelearning/issues/1669 * Missed Random.cs * Fix multiclass code gen (#314) * compile error in codegen * removes scores printing * fix bugs * fix test * Fix compile error in codegen project (#319) * removed redundant code * fix test case * Rev OVA pipeline node SDK output: wrap binary trainers as children inside parent OVA node (#317) * Ova Multi class codegen support (#321) * dummy * multiova implementation * fix tests * remove inclusion list * fix tests and console helper * Rev run result trainer name for OVA: output different trainer name for each OVA + binary learner combination (#322) * Rev run result trainer name for Ova: output different trainer name for each Ova + binary learner combination * test fixes * Console helper bug in generated code for multiclass (#323) * fix * fix test * looping perlogclass * fix test * Initial version of Progress bar impl and CLI UI experience (#325) * progressbar * added progressbar and refactoring * reverted * revert sign assembly * added headers and removed exception rethrow * Setting model directory to temp directory (#327) * Suggested changes to progress bar (#335) * progressbar * added progressbar and refactoring * reverted * revert sign assembly * added headers and removed exception rethrow * bug fixes and updates to UI * added friendly name printing for metric * formatting * Rev Samples (#334) * Telemetry2 (#333) * Create test.txt * Create test.txt * changes needed for benchmarking * forgot one file * merge conflict fix * fix build break * back out my version of the fix for Label column issue and fix the original fix * bogus file removal * undo SuggestedPipeline change * remove labelCol from pipeline suggester * fix build break * rename AutoML to Microsoft.ML.Auto everywhere and a shot at publishing nuget package (will probably need tweaks once I try to use the pipleline) * tweak queue in vsts-ci.yml * CLI telemetry implementation * Telemetry implementation * delete unnecessary file and change file size bucket to actually log log2 instead of nearest ceil value * add headers, remove comments * one more header missing * Fix progress bar in linux/osx (#336) * progressbar * added progressbar and refactoring * reverted * revert sign assembly * added headers and removed exception rethrow * bug fixes and updates to UI * added friendly name printing for metric * formatting * change from task to thread * Update src/mlnet/CodeGenerator/CodeGenerationHelper.cs Co-Authored-By: srsaggam <41802116+srsaggam@users.noreply.github.com> * Mem leak fix (#328) * Create test.txt * Create test.txt * changes needed for benchmarking * forgot one file * merge conflict fix * fix build break * back out my version of the fix for Label column issue and fix the original fix * bogus file removal * undo SuggestedPipeline change * remove labelCol from pipeline suggester * fix build break * rename AutoML to Microsoft.ML.Auto everywhere and a shot at publishing nuget package (will probably need tweaks once I try to use the pipleline) * tweak queue in vsts-ci.yml * there is still investigation to be done but this fix works and solves memory leak problems * minor refactor * Upgrade ML.NET package (#343) * Add cross-validation (CV), and auto-CV for small datasets; push common API experiment methods into base class (#287) * restore old yml for internal pipeline so we can publish nuget again to devdiv stream (#344) * Polishing the CLI UI part-1 (#338) * formatting of pbar message * Polishing the UI * optimization * rename variable * Update src/mlnet/AutoML/AutoMLEngine.cs Co-Authored-By: srsaggam <41802116+srsaggam@users.noreply.github.com> * Update src/mlnet/CodeGenerator/CodeGenerationHelper.cs Co-Authored-By: srsaggam <41802116+srsaggam@users.noreply.github.com> * new message * changed hhtp to https * added iteration num + 1 * change string name and add color to artifacts * change the message * build errors * added null checks * added exception messsages to log file * added exception messsages to log file * CLI ML.NET version upgrade (#345) * Sample revs; ColumnInformation property name revs; pre-featurizer fixes (#346) * CLI -- consume logs from AutoML SDK (#349) * Rename RunDetails --> RunDetail (#350) * command line api upgrade and progress bar rendering bug (#366) * added fix for all platforms progress bar * upgrade nuget * removed args from writeline * change in the version (#368) * fix few bugs in progressbar and verbosity (#374) * fix few bugs in progressbar and verbosity * removed unused name space * Fix for folders with space in it while generating project (#376) * support for folders with spaces * added support for paths with space * revert file * change name of var * remove spaces * SMAC fix for minimizing metrics (#363) * Formatting Regression metrics and progress bar display days. (#379) * added progress bar day display and fix regression metrics * fix formatting * added total time * formatted total time * change command name and add pbar message (#380) * change command name and add pbar message * fix tests * added aliases * duplicate alias * added another alias for task * UI missing features (#382) * added formatting changes * added accuracy specifically * downgrade the codepages (#384) * Change in project structure (#385) * initial changes * Change in project structure * correcting test * change variable name * fix tests * fix tests * fix more tests * fix codegen errors * adde log file message * changed name of args * change variable names * fix test * FileSizeBuckets in correct units (#387) * Minor telemetry change to log in correct units and make our life easier in the future * Use Ceiling instead of Round * changed order (#388) * prep work to transfer to ml.net (#389) * move test projects to top level test subdir * rename some projects to make naming consistent and make it build again * fix test project refs * Add AutoML components to build, fix issues related to that so it builds * fix test cases, remove AppInsights ref from AutoML (#3329) * [AutoML] disable netfx build leg for now (#3331) * disable netfx build leg for now * disable netfx build leg for now. * [AutoML] Add AutoML XML documentation to all public members; migrate AutoML projects & tests into ML.NET solution; AutoML test fixes (#3351) * [AutoML] Rev AutoML public API; add required native references to AutoML projects (#3364) * [AutoML] Minor changes to generated project in CLI based on feedback (#3371) * nitpicks for generated project * revert back the target framework * [AutoML] Migrate AutoML back to its own solution, w/ NuGet dependencies (#3373) * Migrate AutoML back to its own solut…

Ivan Matantsev added 2 commits June 14, 2018 12:33

add pipelineitem for Ova

fe9f740

make tests work

6cf3f06

Ivanidzo4ka requested review from codemzs, TomFinley, yaeldekel, sfilipi and zeahmed June 15, 2018 17:02

zeahmed reviewed Jun 20, 2018

View reviewed changes

zeahmed approved these changes Jun 20, 2018

View reviewed changes

sfilipi approved these changes Jun 20, 2018

View reviewed changes

Ivan Matantsev added 2 commits June 20, 2018 13:09

address zeeshan comment

40dcc26

tweek presicion

7a3ad19

Ivanidzo4ka merged commit e5de547 into dotnet:master Jun 20, 2018

TomFinley mentioned this pull request Jun 22, 2018

Bring ensembles into codebase #379

Merged

shauheen mentioned this pull request Jun 25, 2018

Enable using OneVersusAll (OVA) in LearningPipeline #34

Closed

eerhardt pushed a commit to eerhardt/machinelearning that referenced this pull request Jul 27, 2018

add pipelineitem for Ova (dotnet#363)

a651016

add pipelineitem for Ova

Dmitry-A pushed a commit to Dmitry-A/machinelearning that referenced this pull request Apr 12, 2019

SMAC fix for minimizing metrics (dotnet#363)

40496fd

Dmitry-A pushed a commit to Dmitry-A/machinelearning that referenced this pull request Aug 22, 2019

SMAC fix for minimizing metrics (dotnet#363)

3a8ea68

ghost locked as resolved and limited conversation to collaborators Mar 30, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add pipelineitem for Ova #363

add pipelineitem for Ova #363

Ivanidzo4ka commented Jun 14, 2018 •

edited

Loading

Ivanidzo4ka commented Jun 20, 2018

zeahmed Jun 20, 2018

zeahmed left a comment

sfilipi left a comment

add pipelineitem for Ova #363

add pipelineitem for Ova #363

Conversation

Ivanidzo4ka commented Jun 14, 2018 • edited Loading

Ivanidzo4ka commented Jun 20, 2018

zeahmed Jun 20, 2018

Choose a reason for hiding this comment

zeahmed left a comment

Choose a reason for hiding this comment

sfilipi left a comment

Choose a reason for hiding this comment

Ivanidzo4ka commented Jun 14, 2018 •

edited

Loading