Merge pull request #189 from microsoft/master

merge master

README_zh_CN.md

@@ -10,7 +10,7 @@
 
 NNI (Neural Network Intelligence) 是自动机器学习（AutoML）的工具包。 它通过多种调优的算法来搜索最好的神经网络结构和（或）超参，并支持单机、本地多机、云等不同的运行环境。
 
-### **NNI [v0.8](https://github.com/Microsoft/nni/releases) 已发布！**
+### **NNI [v0.9](https://github.com/Microsoft/nni/releases) 已发布！**
 
 <p align="center">
   <a href="#nni-v05-has-been-released"><img src="docs/img/overview.svg" /></a>
@@ -61,11 +61,12 @@ NNI (Neural Network Intelligence) 是自动机器学习（AutoML）的工具包
           <li><a href="examples/tuners/enas_nni/README_zh_CN.md">ENAS</a></li>
           <li><a href="docs/zh_CN/BuiltinTuner.md#MetisTuner">Metis Tuner</a></li>
           <li><a href="docs/zh_CN/BuiltinTuner.md#BOHB">BOHB</a></li>
+          <li><a href="docs/zh_CN/BuiltinTuner.md#GPTuner">GP Tuner</a></li>
         </ul>
-          <a href="docs/zh_CN/BuiltinAssessors.md">Assessor（评估器）</a> 
+          <a href="docs/zh_CN/BuiltinAssessor.md">Assessor（评估器）</a>
         <ul>
-          <li><a href="docs/zh_CN/BuiltinAssessors.md#Medianstop">Median Stop</a></li>
-          <li><a href="docs/zh_CN/BuiltinAssessors.md#Curvefitting">Curve Fitting</a></li>
+          <li><a href="docs/zh_CN/BuiltinAssessor.md#Medianstop">Median Stop</a></li>
+          <li><a href="docs/zh_CN/BuiltinAssessor.md#Curvefitting">Curve Fitting</a></li>
         </ul>
       </td>
       <td>
@@ -101,12 +102,6 @@ NNI (Neural Network Intelligence) 是自动机器学习（AutoML）的工具包
 
 ## **安装和验证**
 
-在 Windows 本机模式下，并且是第一次使用 PowerShell 来运行脚本，需要**使用管理员权限**运行一次下列命令：
-
-```bash
-    Set-ExecutionPolicy -ExecutionPolicy Unrestricted
-```
-
 **通过 pip 命令安装**
 
 * 当前支持 Linux，MacOS 和 Windows（本机，远程，OpenPAI 模式），在 Ubuntu 16.04 或更高版本，MacOS 10.14.1 以及 Windows 10.1809 上进行了测试。 在 `python >= 3.5` 的环境中，只需要运行 `pip install` 即可完成安装。
@@ -131,14 +126,14 @@ python -m pip install --upgrade nni
 
 **通过源代码安装**
 
-* 当前支持 Linux（Ubuntu 16.04 或更高版本），MacOS（10.14.1）以及 Windows 10（1809 版）。 
+* 当前支持 Linux（Ubuntu 16.04 或更高版本），MacOS（10.14.1）以及 Windows 10（1809 版）。
 
 Linux 和 macOS
 
 * 在 `python >= 3.5` 的环境中运行命令： `git` 和 `wget`，确保安装了这两个组件。
 
 ```bash
-    git clone -b v0.7 https://github.com/Microsoft/nni.git
+    git clone -b v0.8 https://github.com/Microsoft/nni.git
     cd nni
     source install.sh
 ```
@@ -148,9 +143,9 @@ Windows
 * 在 `python >=3.5` 的环境中运行命令： `git` 和 `PowerShell`，确保安装了这两个组件。
 
 ```bash
-  git clone -b v0.7 https://github.com/Microsoft/nni.git
+  git clone -b v0.8 https://github.com/Microsoft/nni.git
   cd nni
-  powershell .\install.ps1
+  powershell -ExecutionPolicy Bypass -file install.ps1
 ```
 
 参考[安装 NNI](docs/zh_CN/Installation.md) 了解系统需求。
@@ -164,7 +159,7 @@ Windows 上参考 [Windows 上使用 NNI](docs/zh_CN/NniOnWindows.md)。
 * 通过克隆源代码下载示例。
 
 ```bash
-    git clone -b v0.7 https://github.com/Microsoft/nni.git
+    git clone -b v0.8 https://github.com/Microsoft/nni.git
 ```
 
 Linux 和 macOS

azure-pipelines.yml

@@ -13,6 +13,11 @@ jobs:
   - script: |
       source install.sh
     displayName: 'Install nni toolkit via source code'
+  - script: |
+      python3 -m pip install flake8 --user
+      IGNORE=./tools/nni_annotation/testcase/*:F821,./examples/trials/mnist-nas/mnist.py:F821
+      python3 -m flake8 . --count --per-file-ignores=$IGNORE --select=E9,F63,F72,F82 --show-source --statistics
+    displayName: 'Run flake8 tests to find Python syntax errors and undefined names'
   - script: |
       cd test
       source unittest.sh

docs/en_US/GeneralNasInterfaces.md → docs/en_US/AdvancedFeature/GeneralNasInterfaces.md

@@ -10,13 +10,13 @@ To facilitate NAS innovations (e.g., design/implement new NAS models, compare di
 
  A new programming interface for designing and searching for a model is often demanded in two scenarios. 1) When designing a neural network, the designer may have multiple choices for a layer, sub-model, or connection, and not sure which one or a combination performs the best. It would be appealing to have an easy way to express the candidate layers/sub-models they want to try. 2) For the researchers who are working on automatic NAS, they want to have an unified way to express the search space of neural architectures. And making unchanged trial code adapted to different searching algorithms.
 
- We designed a simple and flexible programming interface based on [NNI annotation](./AnnotationSpec.md). It is elaborated through examples below.
+ We designed a simple and flexible programming interface based on [NNI annotation](../Tutorial/AnnotationSpec.md). It is elaborated through examples below.
 
 ### Example: choose an operator for a layer
 
-When designing the following model there might be several choices in the fourth layer that may make this model perform good. In the script of this model, we can use annotation for the fourth layer as shown in the figure. In this annotation, there are five fields in total:
+When designing the following model there might be several choices in the fourth layer that may make this model perform well. In the script of this model, we can use annotation for the fourth layer as shown in the figure. In this annotation, there are five fields in total:
 
-![](../img/example_layerchoice.png)
+![](../../img/example_layerchoice.png)
 
 * __layer_choice__: It is a list of function calls, each function should have defined in user's script or imported libraries. The input arguments of the function should follow the format: `def XXX(inputs, arg2, arg3, ...)`, where inputs is a list with two elements. One is the list of `fixed_inputs`, and the other is a list of the chosen inputs from `optional_inputs`. `conv` and `pool` in the figure are examples of function definition. For the function calls in this list, no need to write the first argument (i.e., input). Note that only one of the function calls are chosen for this layer.
 * __fixed_inputs__: It is a list of variables, the variable could be an output tensor from a previous layer. The variable could be `layer_output` of another `nni.mutable_layer` before this layer, or other python variables before this layer. All the variables in this list will be fed into the chosen function in `layer_choice` (as the first element of the input list).
@@ -32,25 +32,25 @@ __Debugging__: We provided an `nnictl trial codegen` command to help debugging y
 
 Designing connections of layers is critical for making a high performance model. With our provided interface, users could annotate which connections a layer takes (as inputs). They could choose several ones from a set of connections. Below is an example which chooses two inputs from three candidate inputs for `concat`. Here `concat` always takes the output of its previous layer using `fixed_inputs`.
 
-![](../img/example_connectchoice.png)
+![](../../img/example_connectchoice.png)
 
 ### Example: choose both operators and connections
 
 In this example, we choose one from the three operators and choose two connections for it. As there are multiple variables in inputs, we call `concat` at the beginning of the functions.
 
-![](../img/example_combined.png)
+![](../../img/example_combined.png)
 
 ### Example: [ENAS][1] macro search space
 
 To illustrate the convenience of the programming interface, we use the interface to implement the trial code of "ENAS + macro search space". The left figure is the macro search space in ENAS paper.
 
-![](../img/example_enas.png)
+![](../../img/example_enas.png)
 
 ## Unified NAS search space specification
 
 After finishing the trial code through the annotation above, users have implicitly specified the search space of neural architectures in the code. Based on the code, NNI will automatically generate a search space file which could be fed into tuning algorithms. This search space file follows the following JSON format.
 
-```json
+```javascript
 {
     "mutable_1": {
         "layer_1": {
@@ -67,7 +67,7 @@ After finishing the trial code through the annotation above, users have implicit
 
 Accordingly, a specified neural architecture (generated by tuning algorithm) is expressed as follows:
 
-```json
+```javascript
 {
     "mutable_1": {
         "layer_1": {
@@ -91,7 +91,7 @@ With the specification of the format of search space and architecture (choice) e
 
 NNI's annotation compiler transforms the annotated trial code to the code that could receive architecture choice and build the corresponding model (i.e., graph). The NAS search space can be seen as a full graph (here, full graph means enabling all the provided operators and connections to build a graph), the architecture chosen by the tuning algorithm is a subgraph in it. By default, the compiled trial code only builds and executes the subgraph.
 
-![](../img/nas_on_nni.png)
+![](../../img/nas_on_nni.png)
 
 The above figure shows how the trial code runs on NNI. `nnictl` processes user trial code to generate a search space file and compiled trial code. The former is fed to tuner, and the latter is used to run trials. 
 
@@ -101,7 +101,7 @@ The above figure shows how the trial code runs on NNI. `nnictl` processes user t
 
 Sharing weights among chosen architectures (i.e., trials) could speedup model search. For example, properly inheriting weights of completed trials could speedup the converge of new trials. One-Shot NAS (e.g., ENAS, Darts) is more aggressive, the training of different architectures (i.e., subgraphs) shares the same copy of the weights in full graph.
 
-![](../img/nas_weight_share.png)
+![](../../img/nas_weight_share.png)
 
 We believe weight sharing (transferring) plays a key role on speeding up NAS, while finding efficient ways of sharing weights is still a hot research topic. We provide a key-value store for users to store and load weights. Tuners and Trials use a provided KV client lib to access the storage.
 
@@ -111,9 +111,9 @@ Example of weight sharing on NNI.
 
 One-Shot NAS is a popular approach to find good neural architecture within a limited time and resource budget. Basically, it builds a full graph based on the search space, and uses gradient descent to at last find the best subgraph. There are different training approaches, such as [training subgraphs (per mini-batch)][1], [training full graph through dropout][6], [training with architecture weights (regularization)][3]. Here we focus on the first approach, i.e., training subgraphs (ENAS).
 
-With the same annotated trial code, users could choose One-Shot NAS as execution mode on NNI. Specifically, the compiled trial code builds the full graph (rather than subgraph demonstrated above), it receives a chosen architecture and training this architecture on the full graph for a mini-batch, then request another chosen architecture. It is supported by [NNI multi-phase](./multiPhase.md). We support this training approach because training a subgraph is very fast, building the graph every time training a subgraph induces too much overhead.
+With the same annotated trial code, users could choose One-Shot NAS as execution mode on NNI. Specifically, the compiled trial code builds the full graph (rather than subgraph demonstrated above), it receives a chosen architecture and training this architecture on the full graph for a mini-batch, then request another chosen architecture. It is supported by [NNI multi-phase](MultiPhase.md). We support this training approach because training a subgraph is very fast, building the graph every time training a subgraph induces too much overhead.
 
-![](../img/one-shot_training.png)
+![](../../img/one-shot_training.png)
 
 The design of One-Shot NAS on NNI is shown in the above figure. One-Shot NAS usually only has one trial job with full graph. NNI supports running multiple such trial jobs each of which runs independently. As One-Shot NAS is not stable, running multiple instances helps find better model. Moreover, trial jobs are also able to synchronize weights during running (i.e., there is only one copy of weights, like asynchronous parameter-server mode). This may speedup converge.
 

docs/en_US/CurvefittingAssessor.md → docs/en_US/Assessor/CurvefittingAssessor.md

@@ -6,15 +6,15 @@ Curve Fitting Assessor is a LPA(learning, predicting, assessing) algorithm. It s
 
 In this algorithm, we use 12 curves to fit the learning curve, the large set of parametric curve models are chosen from [reference paper][1]. The learning curves' shape coincides with our prior knowlwdge about the form of learning curves: They are typically increasing, saturating functions.
 
-![](../img/curvefitting_learning_curve.PNG)
+![](../../img/curvefitting_learning_curve.PNG)
 
 We combine all learning curve models into a single, more powerful model. This combined model is given by a weighted linear combination:
 
-![](../img/curvefitting_f_comb.gif)
+![](../../img/curvefitting_f_comb.gif)
 
 where the new combined parameter vector
 
-![](../img/curvefitting_expression_xi.gif)
+![](../../img/curvefitting_expression_xi.gif)
 
 Assuming additive a Gaussian noise and the noise parameter is initialized to its maximum likelihood estimate.
 
@@ -30,7 +30,7 @@ Concretely,this algorithm goes through three stages of learning, predicting and
 
 The figure below is the result of our algorithm on MNIST trial history data, where the green point represents the data obtained by Assessor, the blue point represents the future but unknown data, and the red line is the Curve predicted by the Curve fitting assessor.
 
-![](../img/curvefitting_example.PNG)
+![](../../img/curvefitting_example.PNG)
 
 ## 2. Usage
 To use Curve Fitting Assessor, you should add the following spec in your experiment's YAML config file:

docs/en_US/CommunitySharings/HpoComparision.md

@@ -5,15 +5,15 @@ Comparison of Hyperparameter Optimization algorithms on several problems.
 
 Hyperparameter Optimization algorithms are list below:
 
-- [Random Search](../BuiltinTuner.md)
-- [Grid Search](../BuiltinTuner.md)
-- [Evolution](../BuiltinTuner.md)
-- [Anneal](../BuiltinTuner.md)
-- [Metis](../BuiltinTuner.md)
-- [TPE](../BuiltinTuner.md)
-- [SMAC](../BuiltinTuner.md)
-- [HyperBand](../BuiltinTuner.md)
-- [BOHB](../BuiltinTuner.md)
+- [Random Search](../Tuner/BuiltinTuner.md)
+- [Grid Search](../Tuner/BuiltinTuner.md)
+- [Evolution](../Tuner/BuiltinTuner.md)
+- [Anneal](../Tuner/BuiltinTuner.md)
+- [Metis](../Tuner/BuiltinTuner.md)
+- [TPE](../Tuner/BuiltinTuner.md)
+- [SMAC](../Tuner/BuiltinTuner.md)
+- [HyperBand](../Tuner/BuiltinTuner.md)
+- [BOHB](../Tuner/BuiltinTuner.md)
 
 All algorithms run in NNI local environment.
 
@@ -34,7 +34,7 @@ is running in docker?: no
 
 ### Problem Description
 
-Nonconvex problem on the hyper-parameter search of [AutoGBDT](../gbdt_example.md) example.
+Nonconvex problem on the hyper-parameter search of [AutoGBDT](../TrialExample/GbdtExample.md) example.
 
 ### Search Space
 
@@ -98,8 +98,11 @@ The total search space is 1,204,224, we set the number of maximum trial to 1000.
 | HyperBand     |0.414065|0.415222|0.417628|
 | HyperBand     |0.416807|0.417549|0.418828|
 | HyperBand     |0.415550|0.415977|0.417186|
+| GP            |0.414353|0.418563|0.420263|
+| GP            |0.414395|0.418006|0.420431|
+| GP            |0.412943|0.416566|0.418443|
 
-For Metis, there are about 300 trials because it runs slowly due to its high time complexity O(n^3) in Gaussian Process.
+In this example, all the algorithms are used with default parameters. For Metis, there are about 300 trials because it runs slowly due to its high time complexity O(n^3) in Gaussian Process.
 
 ## RocksDB Benchmark 'fillrandom' and 'readrandom'
 

docs/en_US/community_sharings.rst → docs/en_US/CommunitySharings/community_sharings.rst

@@ -7,6 +7,6 @@ In addtion to the official tutorilas and examples, we encourage community contri
 ..  toctree::
     :maxdepth: 2
 
-    NNI Practice Sharing<nni_practice_sharing>
-    Neural Architecture Search Comparison<CommunitySharings/NasComparison>
-    Hyper-parameter Tuning Algorithm Comparsion<CommunitySharings/HpoComparison>
+    NNI in Recommenders <RecommendersSvd>
+    Neural Architecture Search Comparison <NasComparision>
+    Hyper-parameter Tuning Algorithm Comparsion <HpoComparision>

docs/en_US/Overview.md

@@ -33,27 +33,27 @@ Basically, an experiment runs as follows: Tuner receives search space and genera
 
 For each experiment, user only needs to define a search space and update a few lines of code, and then leverage NNI built-in Tuner/Assessor and training platforms to search the best hyperparameters and/or neural architecture. There are basically 3 steps:
 
->Step 1: [Define search space](SearchSpaceSpec.md)
+>Step 1: [Define search space](Tutorial/SearchSpaceSpec.md)
 
->Step 2: [Update model codes](Trials.md)
+>Step 2: [Update model codes](TrialExample/Trials.md)
 
->Step 3: [Define Experiment](ExperimentConfig.md)
+>Step 3: [Define Experiment](Tutorial/ExperimentConfig.md)
 
 
 <p align="center">
 <img src="https://user-images.githubusercontent.com/23273522/51816627-5d13db80-2302-11e9-8f3e-627e260203d5.jpg" alt="drawing"/>
 </p>
 
-More details about how to run an experiment, please refer to [Get Started](QuickStart.md).
+More details about how to run an experiment, please refer to [Get Started](Tutorial/QuickStart.md).
 
 ## Learn More
-* [Get started](QuickStart.md)
-* [How to adapt your trial code on NNI?](Trials.md)
-* [What are tuners supported by NNI?](BuiltinTuner.md)
-* [How to customize your own tuner?](CustomizeTuner.md)
-* [What are assessors supported by NNI?](BuiltinAssessors.md)
-* [How to customize your own assessor?](CustomizeAssessor.md)
-* [How to run an experiment on local?](LocalMode.md)
-* [How to run an experiment on multiple machines?](RemoteMachineMode.md)
-* [How to run an experiment on OpenPAI?](PaiMode.md)
-* [Examples](MnistExamples.md)
+* [Get started](Tutorial/QuickStart.md)
+* [How to adapt your trial code on NNI?](TrialExample/Trials.md)
+* [What are tuners supported by NNI?](Tuner/BuiltinTuner.md)
+* [How to customize your own tuner?](Tuner/CustomizeTuner.md)
+* [What are assessors supported by NNI?](Assessor/BuiltinAssessor.md)
+* [How to customize your own assessor?](Assessor/CustomizeAssessor.md)
+* [How to run an experiment on local?](TrainingService/LocalMode.md)
+* [How to run an experiment on multiple machines?](TrainingService/RemoteMachineMode.md)
+* [How to run an experiment on OpenPAI?](TrainingService/PaiMode.md)
+* [Examples](TrialExample/MnistExamples.md)