3482 - update bundle doc (Project-MONAI#3982)

* update bundle doc Signed-off-by: Wenqi Li <wenqil@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update meta Signed-off-by: Wenqi Li <wenqil@nvidia.com> * update intro Signed-off-by: Wenqi Li <wenqil@nvidia.com> * update examples Signed-off-by: Wenqi Li <wenqil@nvidia.com> * adds yaml demo Signed-off-by: Wenqi Li <wenqil@nvidia.com> * update schema Signed-off-by: Wenqi Li <wenqil@nvidia.com> * update toc Signed-off-by: Wenqi Li <wenqil@nvidia.com> * update doc page Signed-off-by: Wenqi Li <wenqil@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * $import Signed-off-by: Wenqi Li <wenqil@nvidia.com> * adds recommendations Signed-off-by: Wenqi Li <wenqil@nvidia.com> * update config Signed-off-by: Wenqi Li <wenqil@nvidia.com> * update based on comments Signed-off-by: Wenqi Li <wenqil@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * adds a link on CLI parsing Signed-off-by: Wenqi Li <wenqil@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Nic-Ma · Mar 30, 2022 · 3947d4e · 3947d4e
1 parent df19315
commit 3947d4e
Show file tree

Hide file tree

Showing 9 changed files with 212 additions and 14 deletions.
diff --git a/docs/source/bundle_intro.rst b/docs/source/bundle_intro.rst
@@ -0,0 +1,10 @@
+:github_url: https://github.com/Project-MONAI/MONAI
+
+Bundle
+======
+
+.. toctree::
+   :maxdepth: 1
+
+   mb_specification
+   config_syntax.md
diff --git a/docs/source/config_syntax.md b/docs/source/config_syntax.md
@@ -0,0 +1,184 @@
+# MONAI Bundle Configuration
+
+The `monai.bundle` module supports building Python-based workflows via structured configurations.
+
+The main benefits are threefold:
+
+- it provides good readability and usability by separating system parameter settings from the Python code.
+- it describes workflow at a relatively high level and allows for different low-level implementations.
+- learning paradigms at a higher level such as federated learning and AutoML can be decoupled from the component details.
+
+Content:
+
+- [A basic example](#a-basic-example)
+- [Syntax examples explained](#syntax-examples-explained)
+  - [`@` to interpolate with Python objects](#1--to-interpolate-with-python-objects)
+  - [`$` to evaluate as Python expressions](#2--to-evaluate-as-python-expressions)
+  - [`%` to textually replace configuration elements](#3--to-textually-replace-configuration-elements)
+  - [`_target_` (`_disabled_` and `_requires_`) to instantiate a Python object](#4-instantiate-a-python-object)
+- [The command line interface](#the-command-line-interface)
+- [Recommendations](#recommendations)
+
+## A basic example
+
+Components as part of a workflow can be specified using `JSON` or `YAML` syntax, for example, a network architecture
+definition could be stored in a `demo_config.json` file with the following content:
+
+```json
+{
+  "demo_net": {
+    "_target_": "monai.networks.nets.BasicUNet",
+    "spatial_dims": 3,
+    "in_channels": 1,
+    "out_channels": 2,
+    "features": [16, 16, 32, 32, 64, 64]
+  }
+}
+```
+
+or alternatively, in `YAML` format (`demo_config.yaml`):
+
+```yaml
+demo_net:
+  _target_: monai.networks.nets.BasicUNet
+  spatial_dims: 3
+  in_channels: 1
+  out_channels: 2
+  features: [16, 16, 32, 32, 64, 64]
+```
+
+The configuration parser can instantiate the component as a Python object:
+
+```py
+>>> from monai.bundle import ConfigParser
+>>> config = ConfigParser()
+>>> config.read_config("demo_config.json")
+>>> net = config.get_parsed_content("demo_net", instantiate=True)
+BasicUNet features: (16, 16, 32, 32, 64, 64).
+>>> print(type(net))
+<class 'monai.networks.nets.basic_unet.BasicUNet'>
+```
+
+or additionally, tune the input parameters then instantiate the component:
+
+```py
+>>> config["demo_net"]["features"] = [32, 32, 32, 64, 64, 64]
+>>> net = config.get_parsed_content("demo_net", instantiate=True)
+BasicUNet features: (32, 32, 32, 64, 64, 64).
+```
+
+For more details on the `ConfigParser` API, please see https://docs.monai.io/en/latest/bundle.html#config-parser.
+
+## Syntax examples explained
+
+A few characters and keywords are interpreted beyond the plain texts, here are examples of the syntax:
+
+### 1. `@` to interpolate with Python objects
+
+```json
+"@preprocessing#transforms#keys"
+```
+
+_Description:_ A reference to another configuration value defined at `preprocessing#transforms#keys`.
+where `#` indicates a sub-structure of this configuration file.
+
+```json
+"@preprocessing#1"
+```
+
+_Description:_ `1` is interpreted as an integer, which is used to index (zero-based indexing) the `preprocessing` sub-structure.
+
+### 2. `$` to evaluate as Python expressions
+
+```json
+"$print(42)"
+```
+
+_Description:_ `$` is a special character to indicate evaluating `print(42)` at runtime.
+
+```json
+"$[i for i in @datalist]"
+```
+
+_Description:_ Create a list at runtime using the values in `datalist` as input.
+
+```json
+"$from torchvision.models import resnet18"
+```
+
+_Description:_ `$` followed by an import statement is handled slightly differently from the
+Python expressions. The imported module `resnet18` will be available as a global variable
+to the other configuration sections. This is to simplify the use of external modules in the configuration.
+
+### 3. `%` to textually replace configuration elements
+
+```json
+"%demo_config.json#demo_net#in_channels"
+```
+
+_Description:_ A macro to replace the current configuration element with the texts at `demo_net#in_channels` in the
+`demo_config.json` file. The replacement is done before instantiating or evaluating the components.
+
+### 4. instantiate a Python object
+
+```json
+{
+  "demo_name":{
+    "_target_": "my.python.module.Class",
+    "args1": "string",
+    "args2": 42}
+}
+```
+
+_Description:_ This dictionary defines an object with a reference name `demo_name`, with an instantiable type
+specified at `_target_` and with input arguments `args1` and `args2`.
+This dictionary will be instantiated as a Pytorch object at runtime.
+
+`_target_` is a required key by monai bundle syntax for the Python object name.
+`args1` and `args2` should be compatible with the Python object to instantiate.
+
+```json
+{
+  "component_name": {
+    "_target_": "my.module.Class",
+    "_requires_": "@cudnn_opt",
+    "_disabled_": "true"}
+}
+```
+
+_Description:_ `_requires_` and `_disabled_` are optional keys.
+`_requires_` specifies references (string starts with `@`) or
+Python expression that will be evaluated/instantiated before `_target_` object is instantiated.
+It is useful when the component does not explicitly depend on the other ConfigItems via
+its arguments, but requires the dependencies to be instantiated/evaluated beforehand.
+`_disabled_` specifies a flag to indicate whether to skip the instantiation.
+
+## The command line interface
+
+In addition to the Pythonic APIs, a few command line interfaces (CLI) are provided to interact with the bundle.
+The primary usage is:
+```bash
+python -m monai.bundle COMMANDS
+```
+
+where `COMMANDS` is one of the following: `run`, `verify_metadata`, `ckpt_export`, ...
+(please see `python -m monai.bundle --help` for a list of available options).
+
+To display a usage page for a command, for example `run`:
+```bash
+python -m monai.bundle run -- --help
+```
+
+The support is provided by [Python Fire](https://github.com/google/python-fire), please
+make sure the optional dependency is installed, for example,
+using `pip install monai[fire]` or `pip install fire`.
+Details on the CLI argument parsing is provided in the
+[Python Fire Guide](https://github.com/google/python-fire/blob/master/docs/guide.md#argument-parsing).
+
+## Recommendations
+- Both `YAML` and `JSON` are supported, but the advanced features of these formats are not supported.
+- Using meaningful names for the configuration elements can improve the readability.
+- While it is possible to build complex configurations with the bundle syntax,
+  simple structures with sparse uses of expressions or references are preferred.
+- For `$import <module>` in the configuration, please make sure there are instructions for the users to install
+  the `<module>` if it is not a (optional) dependency of MONAI.
diff --git a/docs/source/index.rst b/docs/source/index.rst
@@ -70,7 +70,7 @@ Technical documentation is available at `docs.monai.io <https://docs.monai.io>`_
   :maxdepth: 1
   :caption: Specifications
 
-  mb_specification
+  bundle_intro
 
 Links
 -----

diff --git a/docs/source/mb_specification.rst b/docs/source/mb_specification.rst
@@ -13,7 +13,7 @@ This specification defines the directory structure a bundle must have and the ne
 Directory Structure
 ===================
 
-A MONAI Bundle is defined primarily as a directory with a set of specifically named subdirectories containing the model and metadata files. The root directory should be named for the model, given as "ModelName" in this exmaple, and should contain the following structure:
+A MONAI Bundle is defined primarily as a directory with a set of specifically named subdirectories containing the model and metadata files. The root directory should be named for the model, given as "ModelName" in this example, and should contain the following structure:
 
 ::
 
@@ -55,7 +55,7 @@ This file contains the metadata information relating to the model, including wha
 * **optional_packages_version**: dictionary relating optional package names to their versions, these packages are not needed but are recommended to be installed with this stated minimum version.
 * **task**: plain-language description of what the model is meant to do.
 * **description**: longer form plain-language description of what the model is, what it does, etc.
-* **authorship**: state author(s) of the model.
+* **authors**: state author(s) of the model.
 * **copyright**: state model copyright.
 * **network_data_format**: defines the format, shape, and meaning of inputs and outputs to the model, contains keys "inputs" and "outputs" relating named inputs/outputs to their format specifiers (defined below).
 
@@ -98,7 +98,7 @@ An example JSON metadata file:
       "optional_packages_version": {"nibabel": "3.2.1"},
       "task": "Decathlon spleen segmentation",
       "description": "A pre-trained model for volumetric (3D) segmentation of the spleen from CT image",
-      "authorship": "MONAI team",
+      "authors": "MONAI team",
       "copyright": "Copyright (c) MONAI Consortium",
       "data_source": "Task09_Spleen.tar from http://medicaldecathlon.com/",
       "data_type": "dicom",

diff --git a/monai/bundle/config_item.py b/monai/bundle/config_item.py
@@ -21,7 +21,7 @@
 from monai.bundle.utils import EXPR_KEY
 from monai.utils import ensure_tuple, first, instantiate, optional_import
 
-__all__ = ["ComponentLocator", "ConfigItem", "ConfigExpression", "ConfigComponent"]
+__all__ = ["ComponentLocator", "ConfigItem", "ConfigExpression", "ConfigComponent", "Instantiable"]
 
 
 class Instantiable(ABC):
@@ -173,7 +173,7 @@ class ConfigComponent(ConfigItem, Instantiable):
         - ``"_requires_"`` (optional): specifies reference IDs (string starts with ``"@"``) or ``ConfigExpression``
           of the dependencies for this ``ConfigComponent`` object. These dependencies will be
           evaluated/instantiated before this object is instantiated.  It is useful when the
-          component doesn't explicitly depends on the other `ConfigItems` via its arguments,
+          component doesn't explicitly depend on the other `ConfigItems` via its arguments,
           but requires the dependencies to be instantiated/evaluated beforehand.
         - ``"_disabled_"`` (optional): a flag to indicate whether to skip the instantiation.
 
@@ -302,7 +302,7 @@ class ConfigExpression(ConfigItem):
 
         config = "$monai.__version__"
         expression = ConfigExpression(config, id="test", globals={"monai": monai})
-        print(expression.execute())
+        print(expression.evaluate())
 
     Args:
         config: content of a config item.

diff --git a/monai/bundle/config_parser.py b/monai/bundle/config_parser.py
@@ -211,12 +211,16 @@ def get_parsed_content(self, id: str = "", **kwargs):
                 Use digits indexing from "0" for list or other strings for dict.
                 For example: ``"xform#5"``, ``"net#channels"``. ``""`` indicates the entire ``self.config``.
             kwargs: additional keyword arguments to be passed to ``_resolve_one_item``.
-                Currently support ``reset`` (for parse), ``instantiate`` and ``eval_expr``. All defaulting to True.
+                Currently support ``lazy`` (whether to retain the current config cache, default to `False`),
+                ``instantiate`` (whether to instantiate the `ConfigComponent`, default to `True`) and
+                ``eval_expr`` (whether to evaluate the `ConfigExpression`, default to `True`).
 
         """
         if not self.ref_resolver.is_resolved():
             # not parsed the config source yet, parse it
-            self.parse(kwargs.get("reset", True))
+            self.parse(reset=True)
+        elif not kwargs.get("lazy", False):
+            self.parse(reset=not kwargs.get("lazy", False))
         return self.ref_resolver.get_resolved_content(id=id, **kwargs)
 
     def read_meta(self, f: Union[PathLike, Sequence[PathLike], Dict], **kwargs):

diff --git a/monai/bundle/scripts.py b/monai/bundle/scripts.py
@@ -207,7 +207,7 @@ def verify_metadata(
 ):
     """
     Verify the provided `metadata` file based on the predefined `schema`.
-    `metadata` content must contain the `schema` field for the URL of shcema file to download.
+    `metadata` content must contain the `schema` field for the URL of schema file to download.
     The schema standard follows: http://json-schema.org/.
 
     Args:

diff --git a/tests/testing_data/data_config.json b/tests/testing_data/data_config.json
@@ -70,9 +70,9 @@
     },
     "configs": {
         "test_meta_file": {
-            "url": "https://github.com/Project-MONAI/MONAI-extra-test-data/releases/download/0.8.1/meta_schema_202203171008.json",
+            "url": "https://github.com/Project-MONAI/MONAI-extra-test-data/releases/download/0.8.1/meta_schema_20220324.json",
             "hash_type": "md5",
-            "hash_val": "e3a7e23d1113a1f3e6c69f09b6f9ce2c"
+            "hash_val": "e12813de2c15672a8c8fa8466b3dfc95"
         }
     }
 }
diff --git a/tests/testing_data/metadata.json b/tests/testing_data/metadata.json
@@ -1,5 +1,5 @@
 {
-    "schema": "https://github.com/Project-MONAI/MONAI-extra-test-data/releases/download/0.8.1/meta_schema_202203171008.json",
+    "schema": "https://github.com/Project-MONAI/MONAI-extra-test-data/releases/download/0.8.1/meta_schema_20220324.json",
     "version": "0.1.0",
     "changelog": {
         "0.1.0": "complete the model package",
@@ -13,7 +13,7 @@
     },
     "task": "Decathlon spleen segmentation",
     "description": "A pre-trained model for volumetric (3D) segmentation of the spleen from CT image",
-    "authorship": "MONAI team",
+    "authors": "MONAI team",
     "copyright": "Copyright (c) MONAI Consortium",
     "data_source": "Task09_Spleen.tar from http://medicaldecathlon.com/",
     "data_type": "dicom",