Merge pull request #112 from Microsoft/master

merge master

README.md

@@ -83,11 +83,12 @@ You can use these commands to get more information about the experiment
          commands                       description
 1. nnictl experiment show        show the information of experiments
 2. nnictl trial ls               list all of trial jobs
-3. nnictl log stderr             show stderr log content
-4. nnictl log stdout             show stdout log content
-5. nnictl stop                   stop an experiment
-6. nnictl trial kill             kill a trial job by id
-7. nnictl --help                 get help information about nnictl
+3. nnictl top                    monitor the status of running experiments
+4. nnictl log stderr             show stderr log content
+5. nnictl log stdout             show stdout log content
+6. nnictl stop                   stop an experiment
+7. nnictl trial kill             kill a trial job by id
+8. nnictl --help                 get help information about nnictl
 -----------------------------------------------------------------------
 ```
 

docs/HowToChooseTuner.md

@@ -11,6 +11,8 @@ For now, NNI has supported the following tuner algorithms. Note that NNI install
  - [Grid Search](#Grid)
  - [Hyperband](#Hyperband)
  - [Network Morphism](#NetworkMorphism) (require pyTorch)
+ - [Metis Tuner](#MetisTuner) (require sklearn)
+
 
  ## Supported tuner algorithms
 
@@ -178,7 +180,7 @@ _Usage_:
 <a name="NetworkMorphism"></a>
 **Network Morphism**
 
-[Network Morphism](7) provides functions to automatically search for architecture of deep learning models. Every child network inherits the knowledge from its parent network and morphs into diverse types of networks, including changes of depth, width and skip-connection. Next, it estimates the value of child network using the history architecture and metric pairs. Then it selects the most promising one to train. More detail can be referred to [here](../src/sdk/pynni/nni/networkmorphism_tuner/README.md). 
+[Network Morphism][7] provides functions to automatically search for architecture of deep learning models. Every child network inherits the knowledge from its parent network and morphs into diverse types of networks, including changes of depth, width and skip-connection. Next, it estimates the value of child network using the history architecture and metric pairs. Then it selects the most promising one to train. More detail can be referred to [here](../src/sdk/pynni/nni/networkmorphism_tuner/README.md). 
 
 _Installation_: 
 NetworkMorphism requires [pyTorch](https://pytorch.org/get-started/locally), so users should install it first.
@@ -205,6 +207,43 @@ _Usage_:
 ```
 
 
+<a name="MetisTuner"></a>
+**Metis Tuner**
+
+[Metis][10] offers the following benefits when it comes to tuning parameters:
+While most tools only predicts the optimal configuration, Metis gives you two outputs: (a) current prediction of optimal configuration, and (b) suggestion for the next trial. No more guess work!
+
+While most tools assume training datasets do not have noisy data, Metis actually tells you if you need to re-sample a particular hyper-parameter.
+
+While most tools have problems of being exploitation-heavy, Metis' search strategy balances exploration, exploitation, and (optional) re-sampling.
+
+Metis belongs to the class of sequential model-based optimization (SMBO), and it is based on the Bayesian Optimization framework. To model the parameter-vs-performance space, Metis uses both Gaussian Process and GMM. Since each trial can impose a high time cost, Metis heavily trades inference computations with naive trial. At each iteration, Metis does two tasks:
+*  It finds the global optimal point in the Gaussian Process space. This point represents the optimal configuration.
+* It identifies the next hyper-parameter candidate. This is achieved by inferring the potential information gain of exploration, exploitation, and re-sampling.
+
+Note that the only acceptable types of search space are `choice`, `quniform`, `uniform` and `randint`.
+
+More details can be found in our paper: https://www.microsoft.com/en-us/research/publication/metis-robustly-tuning-tail-latencies-cloud-systems/
+
+
+_Installation_: 
+Metis Tuner requires [sklearn](https://scikit-learn.org/), so users should install it first. User could use `pip3 install sklearn` to install it.
+
+
+_Suggested scenario_:
+Similar to TPE and SMAC, Metis is a black-box tuner. If your system takes a long time to finish each trial, Metis is more favorable than other approaches such as random search. Furthermore, Metis provides guidance on the subsequent trial. Here is an [example](../examples/trials/auto-gbdt/search_space_metis.json) about the use of Metis. User only need to send the final result like `accuracy` to tuner, by calling the nni SDK.
+
+_Usage_:
+```yaml
+  # config.yaml
+  tuner:
+    builtinTunerName: MetisTuner
+    classArgs:
+      #choice: maximize, minimize
+      optimize_mode: maximize
+```
+
+
 # How to use Assessor that NNI supports?
 
 For now, NNI has supported the following assessor algorithms.
@@ -273,3 +312,4 @@ _Usage_:
 [7]: https://arxiv.org/abs/1806.10282
 [8]: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46180.pdf
 [9]: http://aad.informatik.uni-freiburg.de/papers/15-IJCAI-Extrapolation_of_Learning_Curves.pdf
+[10]:https://www.microsoft.com/en-us/research/publication/metis-robustly-tuning-tail-latencies-cloud-systems/ 

docs/NNICTLDOC.md

@@ -17,6 +17,7 @@ nnictl log
 nnictl webui
 nnictl tensorboard
 nnictl top
+nnictl --version
 ```
 ### Manage an experiment
 * __nnictl create__ 
@@ -362,4 +363,15 @@ nnictl top
     	
        | Name, shorthand | Required|Default | Description |
        | ------ | ------ | ------ |------ |
-     | id|  False| |ID of the experiment you want to set|
+     | id|  False| |ID of the experiment you want to set|
+
+### Check nni version
+
+* __nnictl --version__
+  * Description
+
+    Describe the current version of nni installed.
+
+  * Usage
+
+        nnictl --version

examples/trials/auto-gbdt/config_metis.yml

@@ -0,0 +1,21 @@
+authorName: default
+experimentName: example_auto-gbdt-metis
+trialConcurrency: 1
+maxExecDuration: 10h
+maxTrialNum: 10
+#choice: local, remote, pai
+trainingServicePlatform: local
+searchSpacePath: search_space_metis.json
+#choice: true, false
+useAnnotation: false
+tuner:
+  #choice: TPE, Random, Anneal, Evolution, BatchTuner
+  #SMAC (SMAC should be installed through nnictl)
+  builtinTunerName: MetisTuner
+  classArgs:
+    #choice: maximize, minimize
+    optimize_mode: minimize
+trial:
+  command: python3 main.py
+  codeDir: .
+  gpuNum: 0

examples/trials/auto-gbdt/requirments.txt

@@ -1 +1,2 @@
-pip install lightgbm
+lightgbm
+pandas

examples/trials/auto-gbdt/search_space_metis.json

@@ -0,0 +1,5 @@
+{
+    "num_leaves":{"_type":"choice","_value":[31, 28, 24, 20]},
+    "learning_rate":{"_type":"choice","_value":[0.01, 0.05, 0.1, 0.2]},
+    "bagging_freq":{"_type":"choice","_value":[1, 2, 4, 8, 10]}
+}

examples/trials/mnist-batch-tune-keras/mnist-keras.py

@@ -84,7 +84,7 @@ def on_epoch_end(self, epoch, logs={}):
         Run on end of each epoch
         '''
         LOG.debug(logs)
-        nni.report_intermediate_result(logs['acc'])
+        nni.report_intermediate_result(logs["val_acc"])
 
 def train(args, params):
     '''

examples/trials/mnist-keras/mnist-keras.py

@@ -84,7 +84,7 @@ def on_epoch_end(self, epoch, logs={}):
         Run on end of each epoch
         '''
         LOG.debug(logs)
-        nni.report_intermediate_result(logs['acc'])
+        nni.report_intermediate_result(logs["val_acc"])
 
 def train(args, params):
     '''

examples/trials/network_morphism/FashionMNIST/FashionMNIST_keras.py

@@ -40,7 +40,7 @@
     datefmt="%m/%d %I:%M:%S %p",
 )
 # set the logger format
-logger = logging.getLogger("fashion_mnist-network-morphism-keras")
+logger = logging.getLogger("FashionMNIST-network-morphism-keras")
 
 
 # restrict gpu usage background
@@ -152,7 +152,7 @@ def on_epoch_end(self, epoch, logs=None):
         if logs is None:
             logs = dict()
         logger.debug(logs)
-        nni.report_intermediate_result(logs["acc"])
+        nni.report_intermediate_result(logs["val_acc"])
 
 
 # Training
@@ -184,7 +184,7 @@ def train_eval():
 
     # trial report final acc to tuner
     _, acc = net.evaluate(x_test, y_test)
-    logger.debug("Final result is: %d", acc)
+    logger.debug("Final result is: %.3f", acc)
     nni.report_final_result(acc)
 
 

examples/trials/network_morphism/FashionMNIST/FashionMNIST_pytorch.py

@@ -42,7 +42,7 @@
 )
 # pylint: disable=W0603
 # set the logger format
-logger = logging.getLogger("FashionMNIST-network-morphism")
+logger = logging.getLogger("FashionMNIST-network-morphism-pytorch")
 
 
 def get_args():

examples/trials/network_morphism/cifar10/cifar10_keras.py

@@ -152,7 +152,7 @@ def on_epoch_end(self, epoch, logs=None):
         if logs is None:
             logs = dict()
         logger.debug(logs)
-        nni.report_intermediate_result(logs["acc"])
+        nni.report_intermediate_result(logs["val_acc"])
 
 
 # Training
@@ -184,7 +184,7 @@ def train_eval():
 
     # trial report final acc to tuner
     _, acc = net.evaluate(x_test, y_test)
-    logger.debug("Final result is: %d", acc)
+    logger.debug("Final result is: %.3f", acc)
     nni.report_final_result(acc)
 
 

examples/trials/network_morphism/requirements.txt

@@ -2,5 +2,5 @@ numpy==1.14.2
 tensorflow==1.12.0
 torchvision==0.2.1
 Keras==2.2.2
-nni==0.3.0
+nni==0.4.1
 torch==0.4.1

examples/trials/network_morphism/utils.py

@@ -1,8 +1,20 @@
-"""Some helper functions for PyTorch, including:
-    - get_mean_and_std: calculate the mean and std value of dataset.
-    - msr_init: net parameter initialization.
-    - progress_bar: progress bar mimic xlua.progress.
-"""
+# Copyright (c) Microsoft Corporation
+# All rights reserved.
+#
+# MIT License
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
+# documentation files (the "Software"), to deal in the Software without restriction, including without limitation
+# the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and
+# to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+# The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
+# BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
 import numpy as np
 import torch
 import torch.nn as nn

src/nni_manager/rest_server/restValidationSchemas.ts

@@ -148,7 +148,7 @@ export namespace ValidationSchemas {
                 checkpointDir: joi.string().allow('')
             }),
             tuner: joi.object({
-                builtinTunerName: joi.string().valid('TPE', 'Random', 'Anneal', 'Evolution', 'SMAC', 'BatchTuner', 'GridSearch', 'NetworkMorphism'),
+                builtinTunerName: joi.string().valid('TPE', 'Random', 'Anneal', 'Evolution', 'SMAC', 'BatchTuner', 'GridSearch', 'NetworkMorphism', 'MetisTuner'),
                 codeDir: joi.string(),
                 classFileName: joi.string(),
                 className: joi.string(),

src/sdk/pynni/nni/constants.py

@@ -28,7 +28,8 @@
     'Medianstop': 'nni.medianstop_assessor.medianstop_assessor',
     'GridSearch': 'nni.gridsearch_tuner.gridsearch_tuner',
     'NetworkMorphism': 'nni.networkmorphism_tuner.networkmorphism_tuner',
-    'Curvefitting': 'nni.curvefitting_assessor.curvefitting_assessor'
+    'Curvefitting': 'nni.curvefitting_assessor.curvefitting_assessor',
+    'MetisTuner': 'nni.metis_tuner.metis_tuner'
 }
 
 ClassName = {
@@ -40,6 +41,7 @@
     'BatchTuner': 'BatchTuner',
     'GridSearch': 'GridSearchTuner',
     'NetworkMorphism':'NetworkMorphismTuner',
+    'MetisTuner':'MetisTuner',
 
     'Medianstop': 'MedianstopAssessor',
     'Curvefitting': 'CurvefittingAssessor'

src/sdk/pynni/nni/metis_tuner/Regression_GMM/CreateModel.py

@@ -0,0 +1,58 @@
+# Copyright (c) Microsoft Corporation
+# All rights reserved.
+#
+# MIT License
+#
+# Permission is hereby granted, free of charge,
+# to any person obtaining a copy of this software and associated
+# documentation files (the "Software"), to deal in the Software without restriction,
+# including without limitation the rights to use, copy, modify, merge, publish,
+# distribute, sublicense, and/or sell copies of the Software, and
+# to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+# The above copyright notice and this permission notice shall be included
+# in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
+# BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+import os
+import sys
+from operator import itemgetter
+
+import sklearn.mixture as mm
+
+sys.path.insert(1, os.path.join(sys.path[0], '..'))
+
+
+def create_model(samples_x, samples_y_aggregation, percentage_goodbatch=0.34):
+    '''
+    Create the Gaussian Mixture Model
+    '''
+    samples = [samples_x[i] + [samples_y_aggregation[i]] for i in range(0, len(samples_x))]
+
+    # Sorts so that we can get the top samples
+    samples = sorted(samples, key=itemgetter(-1))
+    samples_goodbatch_size = int(len(samples) * percentage_goodbatch)
+    samples_goodbatch = samples[0:samples_goodbatch_size]
+    samples_badbatch = samples[samples_goodbatch_size:]
+
+    samples_x_goodbatch = [sample_goodbatch[0:-1] for sample_goodbatch in samples_goodbatch]
+    #samples_y_goodbatch = [sample_goodbatch[-1] for sample_goodbatch in samples_goodbatch]
+    samples_x_badbatch = [sample_badbatch[0:-1] for sample_badbatch in samples_badbatch]
+
+    # === Trains GMM clustering models === #
+    #sys.stderr.write("[%s] Train GMM's GMM model\n" % (os.path.basename(__file__)))
+    bgmm_goodbatch = mm.BayesianGaussianMixture(n_components=max(1, samples_goodbatch_size - 1))
+    bad_n_components = max(1, len(samples_x) - samples_goodbatch_size - 1)
+    bgmm_badbatch = mm.BayesianGaussianMixture(n_components=bad_n_components)
+    bgmm_goodbatch.fit(samples_x_goodbatch)
+    bgmm_badbatch.fit(samples_x_badbatch)
+
+    model = {}
+    model['clusteringmodel_good'] = bgmm_goodbatch
+    model['clusteringmodel_bad'] = bgmm_badbatch
+    return model
+