NNI 2021 Oct~Nov Iteration Plan

[scarlett2018]

This is the plan for the iteration in 2021 Nov~Dec Iteration, it's a 6 weeks iteration.

Release Plan for upcoming release

• Release manager: @liuzhe-lz
• Feature freeze date: 11-19 11-26 (last delay)
• Code freeze and demo date: 12-3 12-8 12-10 (last delay)
• Branch cut and next release planning date: 12-7 12-24
• Release date: 12-10 12-29

Documentation Update

• @ultmaster Check statics
  • @QuanluZhang Get Ni Yan's account
• Decide which ones to keep Chinese doc
• Update docs for current release
• Deal with Chinese doc @ultmaster (12.1) Adopt symbolic links in Chinese documentation #4345
• HPO doc hierarchy @ zebin (11.23)

NAS

• end2end example for NAS with Retiarii (target starting date 11/17)
  • P0 - @JiahangXu evaluate current SPOS support https://nni.readthedocs.io/en/stable/NAS/SPOS.html Evaluate current SPOS support #4322
  • P0 - @JiahangXu evaluate current ProxylessNAS support https://nni.readthedocs.io/en/stable/NAS/Proxylessnas.html (11.19) (Jiahang: the example works, refine its doc in PR [Docs] Refine proxylessNAS doc #4321 )
• P0 - @ultmaster Serialization (11.5 11.12 11.19 11.24) Serialization infrastructure V2 #4337
• P1 - @QuanluZhang Fix webui chart
  • survey (11.3)
  • fix (11.10) [retiarii] support visualize model space with the hpo chart on webui #4304
  • further design
• P1 - @Lijiaoa Trial parameters (11.23) [webui] Improve trial parameter for retiarii experiment #4314
• P1 - @QuanluZhang option of not exit experiment after done [retiarii] support the option of keeping experiment/webui accessible when experiment is done or failed #4340

Model Compression

• P0 - @J-shang Add doc for v2 config list
• Pruner Improvement
  • @J-shang Support pruning by step (movement pruning, taylor, activation...). (11.12) [Compression V2] Movement pruning #4308
• Migrate Pruner from v1
  • @J-shang AutoCompress Pruner. (11.5) [Compression v2] Auto-Compress Pruner #4280 review: @QuanluZhang @Fiascolsy
  • P0 - @Fiascolsy AMC Pruner. (11.5 11.12 11.19) [Compression V2] AMC Pruner #4320
• P2 - @Fiascolsy Fix AutoCompressPruner GPU memory leak.
• Doc
  • @J-shang Guide on how to customize pruner in v2.
  • P1 - @Fiascolsy Report performance of v2 compare with paper / v1. V2 Model Compression experiment result #4365
  • Need add: fine-grained with speed up in iteration will case issue. how to write config_list.
  • review doc & update the outdated doc.
    • P1 - @Fiascolsy pruning
    • @linbinskn quantize
  • P1 - @J-shang review overview and tutorial hirachey
• P2 - @Fiascolsy More v2 Example [Model Compression] add v2 pruning examples #4305
• @J-shang pass optimizer to pruner redesign (11.26) [Compression v2] Add optimizer & lr scheduler construct helper #4332

HPO

• @QuanluZhang Survey optuna, onedb, sagemaker
• @QuanluZhang fix dispatcher error, experiment status
• P1 - (zebin) improve tuner doc
• @ultmaster Survey ConfigSpace
• P1 - @liuzhe-lz TPE (PR Refactor Hyperopt Tuners (Stage 3) - TPE tuner #4239)
  • doc Update random & tpe & grid search tuner doc #4339
• Nested search space
  • P1 - @liuzhe-lz Grid search (11.5 11.10 11.19) Grid search tuner #4324

nnictl

• P0 - @liuzhe-lz Deprecate config v1 (11.1)
  • ConfigBase class config refactor stage 0 - new base class #4303
  • local training service config refactor stage 1 - ExperimentConfig and LocalConfig #4313
  • All training services config refactor stage 2 - training service configs #4318
  • launcher config refactor stage 3 - nnictl & Experiment #4330
  • update nni manager config refactor stage 4 - nni manager #4355
  • fix annotation (11.26)
  • run ITs
• P1 - @liuzhe-lz Raise error for setting invalid field on ConfigBase
• P0 - @liuzhe-lz add validation for trialGpuNumber/maxTrialNumberPerGpu combination

NNI manager & training service

• P1 - @SparkSnail Pipeline for reuse training services (11.5 11.8 11.10) Support reuse mode for pipeline #4310
• P1 - @SparkSnail Hybrid for all reuse training services (11.26) Add kubeflow and frameworkcontroller hybrid example #4343

WebUI

• Overview page: blank at the bottom (11.16)
• Update readthedoc index.html (11.4 11.10)
• About trial duration question [webui] about trial duration question #3012

Other

• readthedocs dependency
• doc translation

Deferred

NAS

• P1 - @JiahangXu a TBD end2end example for NAS with Retiarii
• P1 - @QuanluZhang Support tuner_no_more_trial
• P1 - @JiahangXu review graph engine
• P? - @ultmaster Update mnasnet and example refactor (plan: NNI NAS example refactor plan #4249)
• P2 - (zebin) More choice type design
• P2 - (zebin) Survey naslib
• P2 - (zebin) Port Houwen's algorithm
• P2 - @liuzhe-lz Merge RetiariiExperiment back to Experiment
• P3 - Visualization nasbench
• P3 - Support HPO in Retiarii
• P3 - Review one-shot, refactor with lightning

Compression

• Pruning Speed Up (check with ningxin)
• P1 - Fine-grained iterative speedup logic.
• P1 - NetAdapt Pruner
• @J-shang Transformer Pruner. (11.5 11.8) [Model Compression] transformer pruner #4180
• @zheng-ningxin Function replacement
• P1 - Sensitivity Pruner.
• @J-shang ADMM: support more metrics.
• P2 - Support min_sparsity_per_layer in global mode.
• P2 - Mixed sparse pattern.
• P2 - Support mask output/input.
• P2 - Pruning with constraints (remain 4x channels).
• P2 - @Fiascolsy Iterative pruning starts from any sparsity ratio.
• P2 - KD in pruning scheduler (maybe use original model as teacher).
• @zheng-ningxin Memory usage optimization
• Refactor of model graph generation (as an independent component)
• Speed up with special/customized layer.
• Quantization V2

HPO

• P1 - fix len(arr) > 0 problem
• P1 - (zebin) Improve hyperband & PBT with shared storage
• P3 - Multi-objective
• P2 - (zebin) transferable HPO
• P2 - Evolution nested search space

nnictl

• P2 - @J-shang Improve nnictl experiment export
• P2 - @liuzhe-lz auto-completion
• P2 - @liuzhe-lz early validation

NNI Manager

• P2 - Distributed trial
• P1 - @acured Project back-end

Web UI

• Align issue in overview page
• Support searching customized string in the search box
• Experiment list could not open experiment with prefix Experiment list could not open experiment with prefix #3746 (need backend)

[chenbohua3]

Hi @scarlett2018 I want to add these items for this iteration:

• About quantization:

  • Support dtype & qscheme customization for lsq quantizer and observer quantizer (P0)
  • upgrade lsq quantizer to lsq+ quantizer, which can used for asymmetric quantization (P0)
  • add model converter for mnn backend (P1)

• About architecture for quantization V2:

  • Analyze the architecture of the PyTorch quantization tool in the open source community, and summarize the advantages and disadvantages of each tool.(P0), with @linbinskn

• A demo case for quantizing YOLOX and deploy it on PyTorch backend (P1)

[liuzhe-lz]

Bug Bash

NAS

• @ultmaster SPOS
• @ultmaster ProxylessNAS
• @QuanluZhang Serialization
• @liuzhe-lz NAS visualization

Model Compression

• @JiahangXu Movement pruning
• @JiahangXu AutoCompress Pruner
• @ultmaster AMC Pruner
• @QuanluZhang optimizer & lr scheduler construct helper
• @J-shang v2 pruning example

HPO

• @J-shang TPE
• @J-shang Grid Search

nnictl

• Create experiment
  • @QuanluZhang v2 correct config
  • @QuanluZhang v2 wrong config
  • @QuanluZhang v1 correct config
  • @QuanluZhang v1 wrong config (will suggest user to manually convert)
  • @QuanluZhang annotation correct config
  • @QuanluZhang annotation wrong config
  • @J-shang From Python correct
  • @J-shang From Python wrong
  • @J-shang remote
  • @J-shang hybrid
  • @SparkSnail AML
• @QuanluZhang Using invalid field
• @QuanluZhang Report error when set trialGpuNumber to zero/undefined and maxTrialNumberPerGpu > 1

Training Service

• @liuzhe-lz Hybrid for all reuse training services

Web UI

• @liuzhe-lz Style update

New Items

• @ultmaster Algorithm config file upgrading

[liuzhe-lz]

Release Note

NOTE: NNI v2.6 is the last version that supports Python 3.6. From next release NNI will require Python 3.7+.

HPO

Experiment

• The legacy experiment config format is now deprecated. (doc of new config)
  • If you are still using legacy format, nnictl will show equivalent new config on start. Please save it to replace the old one.
• nnictl now uses nni.experiment.Experiment APIs as backend. The output message of create, resume, and view commands have changed.
• Added Kubeflow and Frameworkcontroller support to hybrid mode. (doc)
• The hidden tuner manifest file has been updated. This should be transparent to users, but if you encounter issues like failed to find tuner, please try to remove ~/.config/nni.

Algorithms

• Random tuner now supports classArgs seed. (doc)
• TPE tuner is refactored: (doc)
  • Support classArgs seed.
  • Support classArgs tpe_args for expert users to customize algorithm behavior.
  • Parallel optimization has been turned on by default. To turn it off set tpe_args.constant_liar_type to null (or None in Python).
  • constant_liar_type has been moved into tpe_args. If you are using it please update your config.
• Grid search tuner now supports all search space types, including uniform, normal, and nested choice. (doc)

NAS

• Enhancement to serialization utilities (doc) and changes to recommended practice of customizing evaluators. (doc)
• Support latency constraint on edge device for ProxylessNAS based on nn-Meter. (doc)
• Trial parameters are showed more friendly in Retiarii experiments.
• Refactor NAS examples of ProxylessNAS and SPOS.

Model Compression

• New Pruner Supported in Pruning V2
  • Auto-Compress Pruner (doc)
  • AMC Pruner (doc)
  • Movement Pruning Pruner (doc)
• Support nni.trace wrapped Optimizer in Pruning V2. In the case of not affecting the user experience as much as possible, trace the input parameters of the optimizer. (doc)
• Optimize Taylor Pruner, APoZ Activation Pruner, Mean Activation Pruner in V2 memory usage.
• Add more examples for Pruning V2.
• Add document for pruning config list. (doc)
• Parameter masks_file of ModelSpeedup now accepts pathlib.Path object. (Thanks to dosemeion) (doc)
• Bug Fix
  • Fix Slim Pruner in V2 not sparsify the BN weight.
  • Fix Simulator Annealing Task Generator generates config ignoring 0 sparsity.

Documentation

• Supported GitHub feature "Cite this repository".
• Updated index page of readthedocs.
• Updated Chinese documentation.
  • From now on NNI only maintains translation for most import docs and ensures they are up to date.
• Reorganized HPO tuners' doc.

Bugfixes

• Fixed a bug where numpy array is used as a truth value. (Thanks to khituras)
• Fixed a bug in updating search space.
• Fixed a bug that HPO search space file does not support scientific notation and tab indent.
  • For now NNI does not support mixing scientific notation and YAML features. We are waiting for PyYAML to update.
• Fixed a bug that causes DARTS 2nd order to crash.
• Fixed a bug that causes deep copy of mutation primitives (e.g., LayerChoice) to crash.
• Removed blank at bottom in Web UI overview page.

[liuzhe-lz]

• @JiahangXu
  • Feature: support nn-Meter in Proxyless NAS #4206
  • Dev: add python name as Node attribute of graph_gen #4243
  • [Docs] Refine proxylessNAS doc #4321
  • Evaluate current SPOS support #4322
• @QuanluZhang
  • [retiarii] support visualize model space with the hpo chart on webui #4304
  • [bug fix] fix the bug in update search space #4438
• @ultmaster
  • Serialization infrastructure V2 #4337
  • Fix model reference for pylint #4338
  • Fix DARTS 2nd order #4385
  • Serialization improvements (docs, examples, bug fixes) #4390
  • Fix deepcopy of mutables #4400
  • Minor fixes to SPOS and ProxylessNAS examples #4420
  • Retiarii serializer user experience improvements #4437
  • Fix model type in lightning #4451

[ultmaster]

NAS

• Enhancement to serialization utilities (doc) and changes to recommended practice of customizing evaluators (doc)

Bug fixes

• Fix a bug that causes DARTS 2nd order to crash.
• Fix a bug that causes deep copy of mutation primitives (e.g., LayerChoice) to crash.

[JiahangXu]

NAS

• Refactor NAS examples of ProxylessNAS and SPOS.
• Support latency constraint on edge device for ProxylessNAS based on nn-Meter.