Adding LDA Transform #377

ganik · 2018-06-19T23:09:55Z

issue #373

…n LinearPredictor class

Merge from master

ganik · 2018-06-19T23:12:27Z

Step 1: adding LDA Native code #Resolved

ganik · 2018-06-19T23:37:55Z

test OSX10.13 Debug #Resolved

Ivanidzo4ka · 2018-06-25T18:01:21Z

src/Native/LDANative/lda_engine.cpp

+                for (int t = 0; t < K_; ++t)
+                {
+                    // if (word_topic_table_[w * K_ + t])
+                    // if (word_topic_table_[(int64_t)w * K_ + t])


remove it #Resolved

Ivanidzo4ka · 2018-06-25T18:03:07Z

src/Native/LDANative/lda_engine.hpp

+        int FeedInDataDense(int* term_freq, int32_t term_num, int32_t vocab_size);
+
+        //IO, model 
+        // NOTE(jiyuan): assume pTopic and pProb are allocated outside the function


jiyuan [](start = 16, length = 6)

alias #Resolved

Ivanidzo4ka · 2018-06-25T18:03:15Z

src/Native/LDANative/lda_engine.hpp

+        void AllocateModelMemory(int num_vocabs, int num_topics, int64_t nonzero_num);
+        void AllocateModelMemory(int num_vocabs, int num_topics, int64_t mem_block_size, int64_t alias_mem_block_size);
+        //void SetNonzeroNum(int32_t word_id, int32_t nonzero_num);
+        //void FinallizeNonzero();


remove it #Resolved

Ivanidzo4ka · 2018-06-25T18:03:29Z

src/Native/LDANative/lda_engine_export.cpp

+
+#include "lda_engine.hpp"
+
+/// This file use to expose public API to be consumed by TLC.


TLC [](start = 57, length = 3)

ML.Net #Resolved

Ivanidzo4ka · 2018-06-25T18:03:45Z

src/Native/LDANative/light_doc_sampler.cpp

+        doc_topic_counter_.clear();
+        doc->GetDocTopicCounter(doc_topic_counter_);
+
+        // note(taifengw), do we have to assume this?


taifengw [](start = 16, length = 8)

alias #Resolved

Ivanidzo4ka · 2018-06-25T18:04:46Z

src/Native/LDANative/light_doc_sampler.cpp

+
+        for (int w = lower; w < upper; ++w)
+        {
+            //taifeng: why not just a serialization of current hybrid_map? do we need to do a search?


taifeng [](start = 14, length = 7)

alias #Resolved

Ivanidzo4ka · 2018-06-25T18:05:20Z

src/Native/LDANative/light_hash_map.h

+
+        // must call set_memory after construction before use
+        light_hash_map();
+        // NOTE(jiyuan): the size of mem_block_ = 2 * capacity_


jiyuan [](start = 16, length = 6)

alias
#Resolved

Ivanidzo4ka · 2018-06-25T18:38:34Z

src/Native/LDANative/model_block.cpp

+        mem_block_size_ = 2 * upper_bound(load_factor_ * nonzero_num);
+        alias_mem_block_size_ = nonzero_num * 3; 
+
+        mem_block_ = new int32_t[mem_block_size_]();                // NOTE(jiyuan) : force to initialize the values to be zero


jiyuan [](start = 76, length = 6)

all this aliases #Resolved

#Resolved

In reply to: 197899749 [](ancestors = 197899749)

Ivanidzo4ka · 2018-06-25T18:39:22Z

src/Native/LDANative/model_block.h

+
+        // void InitFromModelFile(const )
+        //void SetNonzeroNum(int32_t word_id, int32_t nonzero_num);
+        //void FinallizeNonzero();


remove them? #Resolved

Ivanidzo4ka · 2018-06-25T18:40:46Z

src/Native/LDANative/rand_int_rng.h

+#include <time.h>
+#include <stdint.h>
+
+namespace wood


wood [](start = 10, length = 4)

what a nice namespace name :)

Ivanidzo4ka · 2018-06-25T18:42:42Z

src/Microsoft.ML.Transforms/Text/LdaTransform.cs

+                _exes[i] = ex;
+                _types[i] = new VectorType(NumberType.Float, ex.NumTopic);
+                // C.f. taifengw's shelveset. The following line should be defered in order to pass in numVocab.
+                // _ldas[i] = new LdaState(ex);


should we remove this? #Resolved

Ivanidzo4ka · 2018-06-25T18:43:30Z

src/Microsoft.ML.Transforms/Text/LdaTransform.cs

+            {
+                Contracts.AssertValue(ectx);
+
+                // REVIEW taifengw(tfinley): Input the counts to your trainer here. This


taifengw(tfinley) [](start = 26, length = 17)

alias #Resolved

Ivanidzo4ka · 2018-06-25T18:44:07Z

src/Microsoft.ML.Transforms/Text/LdaTransform.cs

+                    {
+                        // REVIEW: Should this log a warning message? And what should it produce?
+                        // It currently produces a vbuffer of all NA values.
+                        // REVIEW shonk: Need a utility method to do this...


shonk [](start = 34, length = 5)

alias #Resolved

ganik · 2018-06-25T19:19:59Z

test OSX10.13 Release #ByDesign

ganik · 2018-06-25T21:47:27Z

added unit tests that do that

In reply to: 399518917 [](ancestors = 399518917)

ganik · 2018-06-25T22:24:33Z

test OSX10.13 Release

Ivanidzo4ka

TomFinley · 2018-06-25T23:40:04Z

Ah OK. What I meant was their own code, rather than checking it in ourselves... but maybe this is harder, since I think they've probably diverged. Something we should think about though.

In reply to: 400106624 [](ancestors = 400106624,399518917)

TomFinley

Thank you @ganik !

* Bump master to v0.3 (#269) * RocketEngine fix for selecting top learners (#270) * Changes to RocketEngine to fix take top k logic. * Add namespace information to allow file to reference correct version of Formatting object. * small code cleanup (#271) * Preparation for syncing sources with internal repo (#275) * make class partial so I can add constuctor in separate file. add constructros for testing * formatting * Changes to use evaluator metrics names in PipelineSweeperSupportedMetrics. Made the private const strings in two classes public. (#276) * add missing subcomponents to sweepers (#278) * add missing subcomponents * right one * more cleanup * remove lotus references. (#252) * Random seed and concurrency for tests (#277) * first attempt * add comments * specify seed for random. make constructor internal. * Fix SupportedMetric.ByName() method (#280) * Fix for SupportedMetric.ByName() method. Include new unit test for function. * Fix for SupportedMetric.ByName() method. Include new unit test for function. * Fix for SupportedMetric.ByName() method. Include new unit test for function. * Removed unnecessary field filter, per review comment. * ML.NET-242: FastTreeRanking per-iteration loss metrics are empty (#289) When training a FastTreeRanker using the `testFrequency` parameter, it is expected that NDCG is prented every testFrequency iterations. However, instead of NDCG, only empty strings are printed. The root cause was that the MaxDCG property of the dataset was never calculated, so the NDCG calculation is aborted, leaving an empty string as a result. This PR fixes the problem by computing the MaxDCG for the dataset when the Tests are defined (so that if the tests are not defined, the MaxDCG will never be calculated). Closes #242 * Fixed typo in the method summary (#296) * Remove stale line of code from test. (#297) * Update release notes link to use aka.ms. (#294) Our release notes link is broken because the `Documentation` was renamed to `docs`. Fix this for the future to use a redirection link. * Add release notes for ML.NET 0.2 (#301) * Add release notes for ML.NET 0.2 * Adding release note about TextLoader changes and additional issue/PR references * Addressing comments: fixing typos, changing formatting, and adding references * Get the cross validation macro to work with non-default column names (#291) * Add label/grou/weight column name arguments to CV and train-test macros * Fix unit test. * Merge. * Update CSharp API. * Fix EntryPointCatalog test. * Address PR comments. * update sample in README.MD with 0.2 features. (#304) * update sample with new text loader API. * update with 0.2 stuff. * OVA should respect normalization in underlying learner (#310) * Respect normalization in OVA. * some cleanup * fix copypaste issues * Export to ONNX and cross-platform command-line tool to script ML.NET training and inference (#248) * Export to ONNX and Maml cross-platform executable. * Add Cluster evaluator (#316) * Add Cluster evaluator * fix copypaste * address comments * formatting * Fixes locale dependent test output comparisons (#109) The tests do not pass on systems with locale other than en-US. The error happens since the results are written to files and the contents of the files are compared to set of correct results produced under en-US locale. The fix is to imbue en-US culture to the test thread so that results will be output in format that is comparable with the test format. This patch fixes only tests, but do not guarantee calculation will be correct in production systems using a locale different than en-US. In particular, there can be problems in reading data and then conversing data from characters to numeric format. Fixes #74 * Add PartitionedFileLoader (#61) * Remove unexisting project from solution (#335) * GetSummaryDataView/Row implementation for Pca and Linear Predictors (#185) * Implement `ICanGetSummaryAsIDataView` on `PcaPredictor` class * Implement `ICanGetSummaryAsIRow` on `LinearPredictor` class * Disable ordinary least squares by removing the entry point (#286) * Disable ols by temporarily removing the entry point. It may be added again once we figure out how to ship MKL as part of this project. * add append function to pipeline (#284) Add `Append` function to pipeline for more fluent API than that allowed by `Add` * Removed field/column name checking of input type in TextLoader. (#327) * fix namespace issue in CSharpGenerator and some refactoring (#339) fix namespace issue and refactoring * Using named-tuple in OneToOneTransforms' constructor to make API more readable. (#324) * Minor formatting in CollectionDataSourceTests.cs (#348) * Create CalibratedPredictor instead of SchemaBindableCalibratedPredictor (#338) `CalibratorUtils.TrainCalibrator` and `TrainCalibratorIfNeeded` now creates `CalibratedPredictor` instead of `SchemaBindableCalibratedPredictor` whenever the predictor implements `IValueMapper`. * Remove reference and dependency on System.ValueTuple (#351) * Add link to samples (#355) * Use HideEnumValueAttribute for both manifest and C# API generation. (#356) * Use HideEnumValueAttribute for both manifest and C# API generation. * Unhide NAReplaceTransform.ReplacementKind.SpecifiedValue. This may require some other PR to resolve the corresponding issues. * Move the NuGet package build files into a TFM specific directory. (#370) When installing Microsoft.ML on an unsupported framework (like net452), it is currently getting installed successfully. However, users should be getting an error stating that net452 is not supported by this package. The cause is the build files exist for any TFM, which NuGet interprets as this package supports any TFM. Moving the build files to be consistent with the 'lib' folder support. Fix #357 * `Stream` subclasses now have `Close` call `base.Close` to ensure disposal. (#369) * Subclasses of `Stream` now have `Close` call `base.Close` to ensure disposal. * Add DeleteOnClose to File opening. * Remove explicit delete of file. * Remove explicit close of substream. * Since no longer deleting explicitly, no longer need `_overflowPath` member. * Return distinct array of ParameterSet when ProposeSweep is called (#368) * Changed List to HashSet to ensure that there are no duplicates * Update fast tree argument help text (#372) * Update fast tree argument help text * Update wording * Update API to fix test * Update core manifest JSON to update help text * Combine multiple tree ensemble models into a single tree ensemble (#364) * Add a way to create a single tree ensemble model from multiple tree ensemble models. * Address PR comments, and fix bugs in serializing/deserializing RegressionTrees. * Address PR comments. * add pipelineitem for Ova (#363) add pipelineitem for Ova * Fix CV macro to output the warnings data view properly. (#385) * Link to an example on using converting ML.NET model to ONNX. (#386) * Adding documentation about entry points, and entry points graphs: EntryPoints.md and GraphRunner.md (#295) * Adding EntryPoints.md and GraphRunner.md * addressing PR feedback * Updating the title of the GraphRunner.md file * adressing Tom's feedback * adressing feedback * code formatting for class names * Addressing Gal's comments * Adding an example of an entry point. Fixing casing on ML.NET * fixing link * Adding LDA Transform (#377) * Revert to using the native code (#413) Corrects an unintentional "typo" in FastTreeRanking.cs where there was mistakenly a USE_FASTTREENATIVE2 instead of USE_FASTTREENATIVE. This resulted in some obscure hidden ranking options (distance weighting, normalize query lambdas, and a few others) being unavailable. These are important for some applications. * LightGBM (#392) * LightGBM and test. * add test baselines and nuget source for lightGBM binaries. * Add entrypoint for lightGBM. * add unsafe flag for release build. * update nuget version. * make lightgbm test single threaded. * install gcc on OS machines to resolve dependencies on openmp thatis needed by lightgbm native code. * PR comments. Leave BREW and GCC in bash script to verify macOS tests work. * remove brew and gcc from build script. * PR feedback. * disable test on macOS. * disable test on macOS. * PR feedback. * Adding Factorization Machines (#383) * Adding Factorization Machines * ONNX API documentation. (#419) * ONNX API documentation. * Bring ensembles into codebase (#379) Introduce Ensemble codebase * enable macOS tests for LightGBM. (#422) * Create a shorter temp file name for model loading. (#397) Create a shorter temp file name for model loading, as well as remove the potential for a race condition among multiple openings by using the creation of a lock file. * removing extraneous character that broke the linux build, and with it unecessary cmake version requirement (#425) * EvaluatorUtils to handle label column of type key without text key values (#394) * Fix EvaluatorUtils to handle label column of type key without text key values. * Removing non source files from solution (#362)

ganik and others added 22 commits May 17, 2018 17:32

GetSummaryDataView() implementation for Pca and Linear Predictors

ec890d6

typo

9394cdd

Implement ICanGetSummaryAsIRow instead of ICanGetSummaryAsIDataView o…

b508e68

…n LinearPredictor class

Merge branch 'master' of https://github.com/ganik/machinelearning

e072546

Add vector column for top rank eigenvectors

ec8200a

remove unnecessary code

cf37f62

remove space

9943586

fix build

45cd5f2

no need for slot names

083645f

Merge pull request #1 from dotnet/master

58d0f31

Merge from master

Enable back PCA anomaly tests

c5c0173

fix PCA Anomaly tests baseline

695abc5

fix unit tests for Release

eea9c69

Add PCA azure test data

637b325

Added entrypoint summpary test for LinearPredictor

bdec903

added PCA summary test

b942537

use using() { } on IDisposable

678633c

remove slot names

3031a4c

remove not needed datasets

b0c2e49

Merge pull request #2 from dotnet/master

eb3df32

Merge from master

Added LDANative project

7cb7bc8

Merge pull request #3 from dotnet/master

0690012

Merge from master

ganik closed this Jun 19, 2018

ganik reopened this Jun 19, 2018

ganik changed the title ~~Adding LDA Transform~~ [WIP] Adding LDA Transform Jun 19, 2018

ganik added 3 commits June 19, 2018 16:54

remove malloc.h

9ee8703

Merge branch 'master' of https://github.com/ganik/machinelearning

1c4310f

no affinity for MAC for now

1574123

Ivanidzo4ka reviewed Jun 25, 2018

View reviewed changes

address comments

336ddb0

Ivanidzo4ka reviewed Jun 25, 2018

View reviewed changes

remove aliases

98c15c6

Add unit tests for "direct" API

34eedf1

add LdaNative reference to test proj

9b4dcab

ganik added 3 commits June 25, 2018 15:41

thread affinity on OSX

6205510

include <mach/mach_types.h>

b72d194

include <mach/thread_act.h>

1ce124b

Ivanidzo4ka approved these changes Jun 25, 2018

View reviewed changes

TomFinley approved these changes Jun 25, 2018

View reviewed changes

shauheen merged commit 0d5e317 into dotnet:master Jun 26, 2018

eerhardt pushed a commit to eerhardt/machinelearning that referenced this pull request Jul 27, 2018

Adding LDA Transform (dotnet#377)

94fba63

ghost locked as resolved and limited conversation to collaborators Mar 30, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding LDA Transform #377

Adding LDA Transform #377

ganik commented Jun 19, 2018 •

edited

Loading

ganik commented Jun 19, 2018 •

edited

Loading

ganik commented Jun 19, 2018 •

edited

Loading

Ivanidzo4ka Jun 25, 2018 •

edited

Loading

Ivanidzo4ka Jun 25, 2018 •

edited

Loading

Ivanidzo4ka Jun 25, 2018 •

edited

Loading

Ivanidzo4ka Jun 25, 2018 •

edited

Loading

Ivanidzo4ka Jun 25, 2018 •

edited

Loading

Ivanidzo4ka Jun 25, 2018 •

edited

Loading

Ivanidzo4ka Jun 25, 2018 •

edited

Loading

Ivanidzo4ka Jun 25, 2018 •

edited

Loading

ganik Jun 25, 2018

Ivanidzo4ka Jun 25, 2018 •

edited

Loading

ganik Jun 25, 2018

Ivanidzo4ka Jun 25, 2018

Ivanidzo4ka Jun 25, 2018 •

edited

Loading

Ivanidzo4ka Jun 25, 2018 •

edited

Loading

Ivanidzo4ka Jun 25, 2018 •

edited

Loading

ganik commented Jun 25, 2018 •

edited by Ivanidzo4ka

Loading

ganik commented Jun 25, 2018

ganik commented Jun 25, 2018

Ivanidzo4ka left a comment

TomFinley commented Jun 25, 2018

TomFinley left a comment


		#include "lda_engine.hpp"

		/// This file use to expose public API to be consumed by TLC.

Adding LDA Transform #377

Adding LDA Transform #377

Conversation

ganik commented Jun 19, 2018 • edited Loading

ganik commented Jun 19, 2018 • edited Loading

ganik commented Jun 19, 2018 • edited Loading

Ivanidzo4ka Jun 25, 2018 • edited Loading

Choose a reason for hiding this comment

Ivanidzo4ka Jun 25, 2018 • edited Loading

Choose a reason for hiding this comment

Ivanidzo4ka Jun 25, 2018 • edited Loading

Choose a reason for hiding this comment

Ivanidzo4ka Jun 25, 2018 • edited Loading

Choose a reason for hiding this comment

Ivanidzo4ka Jun 25, 2018 • edited Loading

Choose a reason for hiding this comment

Ivanidzo4ka Jun 25, 2018 • edited Loading

Choose a reason for hiding this comment

Ivanidzo4ka Jun 25, 2018 • edited Loading

Choose a reason for hiding this comment

Ivanidzo4ka Jun 25, 2018 • edited Loading

Choose a reason for hiding this comment

ganik Jun 25, 2018

Choose a reason for hiding this comment

Ivanidzo4ka Jun 25, 2018 • edited Loading

Choose a reason for hiding this comment

ganik Jun 25, 2018

Choose a reason for hiding this comment

Ivanidzo4ka Jun 25, 2018

Choose a reason for hiding this comment

Ivanidzo4ka Jun 25, 2018 • edited Loading

Choose a reason for hiding this comment

Ivanidzo4ka Jun 25, 2018 • edited Loading

Choose a reason for hiding this comment

Ivanidzo4ka Jun 25, 2018 • edited Loading

Choose a reason for hiding this comment

ganik commented Jun 25, 2018 • edited by Ivanidzo4ka Loading

ganik commented Jun 25, 2018

ganik commented Jun 25, 2018

Ivanidzo4ka left a comment

Choose a reason for hiding this comment

TomFinley commented Jun 25, 2018

TomFinley left a comment

Choose a reason for hiding this comment

ganik commented Jun 19, 2018 •

edited

Loading

ganik commented Jun 19, 2018 •

edited

Loading

ganik commented Jun 19, 2018 •

edited

Loading

Ivanidzo4ka Jun 25, 2018 •

edited

Loading

Ivanidzo4ka Jun 25, 2018 •

edited

Loading

Ivanidzo4ka Jun 25, 2018 •

edited

Loading

Ivanidzo4ka Jun 25, 2018 •

edited

Loading

Ivanidzo4ka Jun 25, 2018 •

edited

Loading

Ivanidzo4ka Jun 25, 2018 •

edited

Loading

Ivanidzo4ka Jun 25, 2018 •

edited

Loading

Ivanidzo4ka Jun 25, 2018 •

edited

Loading

Ivanidzo4ka Jun 25, 2018 •

edited

Loading

Ivanidzo4ka Jun 25, 2018 •

edited

Loading

Ivanidzo4ka Jun 25, 2018 •

edited

Loading

Ivanidzo4ka Jun 25, 2018 •

edited

Loading

ganik commented Jun 25, 2018 •

edited by Ivanidzo4ka

Loading