Develop MVP of model bundle

[Nic-Ma]

Is your feature request related to a problem? Please describe.
Thanks for the interesting technical discussion with @ericspod @wyli @atbenmurray @rijobro , as we still have many unclear requirements and unknown use cases, we plan to develop the model package feature step by step, May adjust the design based on feedback during the development.

For the initial step, the core team aligned to develop a small but typical example for inference first, it will use JSON config files to define environments, components and workflow, save the config and model into TorchScript model. then other projects can easily reconstruct the exact same python program and parameters to reproduce the inference. When the small MVP is ready, will share and discuss within the team for the next steps.

I will try to implement the MVP referring to some existing solutions, like NVIDIA Clara MMAR, ignite online package, etc. Basic task steps:

• Include metadata (env / sys info, changelog, version, input / output data format, etc), configs, model weights, etc. in a model package example for review. PR: [WIP] 486 Add example model package tutorials#487
• Search specified python packages and build python instances from dictionary config with name / path & args. PR: 3482 Add ConfigComponent for config parsing #3720
• Recursively parse configs in a dictionary with dependencies and executable code, for example:
  {"dataset": {"<name>": "Dataset", "<args>": {"data": "$load_datalist()"}}, "dataloader": {"<name>": "DataLoader", "<args>": {"data": "@dataset"}}}. PR: 3482 Add ReferenceResolver to manage config items #3818, 3482 3829 Add ConfigParser to recursively parse config content #3822
• Add support to save the raw config dictionaries into TorchScript model. PR: Adding Torchscript utility functions #3138 .
• Add schema mechanism to verify the content of folder structure, metadata.json content, etc. Refer to: https://json-schema.org/, https://github.com/Julian/jsonschema. PR: 3482 Add schema verification for metadata #3865
• Add support to verify network input / output with fake data (data input is from metadata).
• Add mechanism to easily load JSON config files with override (can add YAML support in future), similar to example: https://hydra.cc/docs/tutorials/basic/your_first_app/config_groups/. PR: 3482 add run API for common training, evaluation and inference #3832
• Add support to refer to other config item in the same config file or other confg files, referring to Hydra ideas: https://hydra.cc/docs/advanced/overriding_packages/.
• Complete the inference example MMAR for spleen segmentation task. PR: Add inference example for bundle tutorials#604
• Write user manual and detailed documentation. PR: MONAI Archive Specification #3834 3482 - update bundle doc #3982
• [*Optional] Upload the spleen MMAR example to huggingface(Pretrained Models #3451).
• [*Optional] Support relative config level in the reference ID, for example: "test": "@###data#1", 1 # means current level, 2 ## means upper level, etc. PR: 3482 Add support for relative IDs in the config content #3974
• [*Optional] Support customized ConfigItem and ReferenceResolver in the ConfigParser. PR: 3482 Add support for customized ConfigItem and resolver #3980
• _requires_ keyword for config component (adds a _requires_ key for ConfigComponent #3942).
• Import statement in bundle config (support of 'import' statement in bundle config #3966).
• Config python logging properties in a file. PR: 3482 Add support to set config file for logging module #3994
• [*Optional] Specify rank ID for component to run only on some ranks, for example, saving checkpoint in rank 0.

[dbericat]

@CPBridge this is the current PR to expand model metadata.

[Nic-Ma]

Hi @wyli @ericspod @rijobro @atbenmurray ,

I updated the ticket description to make it more clear.

Thanks.

[wyli]

We've discussed that the MVP should focus on the minimal structure, including model weights, basic environment/system info, maintaining changelog/versioning, how to ensure accessibility for both human and machine.

[Nic-Ma]

Hi @wyli ,

Sure, I added these items to the MVP task list.

Thanks.

[dbericat]

@MMelQin @gigony @GreySeaWolf @CPBridge @mocsharp @joshliberty @vikashg thoughts on proposed MVP?

[ericspod]

We should discuss a clear definition of requirements and objectives. We want to define a format of a single file or multiples files which contains the model weights at least with secondary information describing how to use it for various use cases. This will allow a human or a program to determine what sort of model it is, how to use the model, and what tasks to use it for. For our MVP we want to consider the starting position of what the model weight storage and metadata storage would look like and if it would achieve that objective to some degree.

The base level set of requirements I would suggest are:

• Ability to save model weights
• Ability to save runnable model in Torchscript and/or ONNX
• Include information on model name, version, description, authorship, intended use
• Include package and technical information needed to use model, eg. MONAI version, Pytorch and other package versions
• Description of input and output, eg. how to interpret predictions returned
• Description in some form of how to transform input data into the form expected by the model (ie. preprocess transform sequence) and transform for output

One use case for this information is a human user looking into how the model is used in a particular task. They would want a clear idea of what inputs are expected and what the outputs mean. Whatever format this information is can be either easily read by a human or easily converted into a convenient format using included tools.

A second use case is a deployment environment which automatically constructs whatever infrastructure is needed around a model to present it through a standard interface. This would require generating transform sequences automatically to pre- and post-process data, and load the model through some script defining the workflow. This would used by MONAI Deploy to automatically generate a MAP from the package, or another script automatically create a Docker image to serve the model as a command line tool or through Flask, or another script to interface with existing hosting services to upload the model and whatever other information is needed.

[Nic-Ma]

Hi @ericspod ,

Thanks very much for your detailed description.
I will try to prepare a draft model package for discussion according to your summary and our basic ideas, and then develop the necessary features to support it.

Thanks.

[wyli]

I'd strongly recommend that the MVP should support the essential features for fast prototyping of model packages. The following uses the popular detectron2 package as an example:

• Hybrid configs with python + yaml/json/xml
  Here is a workflow config of mixing .py and .yaml, and the essential mechanism is a lazy instantiation of objects.

• Compositional Configuration
  An extension of RCNN-FPN could be constructed by referring to the base config and overriding some of the options.

These are mainly to ensure a good flexibility and extensibility of the packaging.

[Nic-Ma]

Hi @wyli ,

Thanks for your great suggestions.
These are very good references for config parsing logic in steps 3 and 4 of this ticket, we definitely need similar features for flexibility and extensibility.

Thanks.

[Nic-Ma]

Hi @wyli @ericspod @atbenmurray @rijobro ,

I tried to prepare a draft model package structure for the first step of this ticket in the tutorial PR.
Project-MONAI/tutorials#487
Could you please help take a look and share some feedback for the model package structure?
(1) I didn't implement any config-parsing related logic so far, I think that's later steps.
(2) I am not very sure to put some description strings in metadata.json or README.md, so I put in both sides, would like to see your ideas.

Thanks in advance.

[ericspod]

To recap where we are with existing issues/PRs:

• Adding Torchscript utility functions #3138 is merged
• [WIP] 486 Add example model package tutorials#487 package MVP is here to review

Related issues:

• [FEA] Need capability for defining runtime requirements on MIS, and MIS should be able to serve an algorithm in "always on", "when requested" or "batch modes", as specified by the developer monai-deploy-app-sdk#227
• Need app running definition and requirements standardization capabilities and a common way to pass this onto platforms during deployment monai-deploy-app-sdk#226
• [FEA] Integrating Model Zoo and MMAR monai-deploy-app-sdk#213
• host/load pretrained weights for 3D resnet model-zoo#48

[Nic-Ma]

Hi @ericspod ,

Thanks for your great summary!
There is another related PR: #3518.

Thanks.

[wyli]

for the record, we could consider this cache dataset config vs content Integrity issue #999.

[Nic-Ma]

Hi @ericspod @wyli @dbericat ,

Another thing I want to mention is that: we can also consider to add the package format verification with predefined schemes, for example: check the JSON syntax, check necessary fields in the meta data, check essential folders, etc.

Thanks.

[ericspod]

Hi @ericspod @wyli @dbericat ,

Another thing I want to mention is that: we can also consider to add the package format verification with predefined schemes, for example: check the JSON syntax, check necessary fields in the meta data, check essential folders, etc.

Thanks.

We'll certainly want to check the format of specific fields in the json data so that they correctly adhere to a standard way of describing transforms and inference so that code can be generated from them. With the other data that can be added it would be preferrable that this can be as free-form as people like, so I would expect standardized elements in the metadat adhering to our format mixed in with ones that are application-specific in any format.

[Nic-Ma]

@ericspod sounds good to me! We can do that when we finalized the first MVP MMAR example.

Thanks.

[MMelQin]

As for the machine consumer of the model package, specifically TrochScript, we do have a specific one, Triton Inference Server. Triton can automatically generate runtime config for all the backends it supports, except PyTorch. Triton team had logged an issue for PyTorch, though based on the discussions we've already had here, the requested metadata are covered in the MONAI model package. Here is the PyTorch issue 38273,

[MMelQin]

For deployment of AI apps/models in clinical environment, there are a number of persona and user stories.

• Reporting of the AI result needs certain metadata of the model (some of them are already covered in proposal) per IHE AIR R1.1. Model information needs to be identified, per 6.5.3.1 General Result Encoding Requirements, more specifically,

1. Purpose of Reference Code Sequence (0040,A170) shall be (Newcode1, 99IHE,1630 "Processing Algorithm")
2. Manufacturer (model creator/trainer)
3. Manufacturer’s Model Name (model name)
4. Software Versions (version)
5. Device UID (UID for the model. Each time an AI Model is modified, for example by training, it would be appropriate to update the Device UID.)

• For integration with DICOM network and selection of relevant series to feed the model, an application needs some metadata description about the DCIOM study/series,

1. Modality Type (DICOM standards has a defined set
2. Minimum resolution of images, expressed in max pixel spacings (nice to have; it has been used in practice)
3. Demographic info, e.g. minimal age of patient the model can be applied to (nice to have)

• For models with multi-channel, e.g. Brain tumor segmentation model taking in images with T1, T2, FLARE, etc. When deployed, the application needs to request/receive these DICOM series, converts them into volumetric images, registers them, and then generates a multi-channel single image to feed the model

1. Expected order and meaning of the channel, e.g. corresponding to T1, or T2 series

• For flexibility in orchestrating the inference pipeline/Compose, there should not be implicit dependency in the inference config, e.g. MONAI Handlers depending on Ignite.

[Nic-Ma]

Another related PR relating to the configuration parsing: #3822.

Thanks.

[Nic-Ma]

Related PR for common training, evaluation and inference: #3832 .

Thanks.

[Nic-Ma]

As we are still developing the ConfigParser in PR #3822, I compared it with the Hydra OmegaConf: https://omegaconf.readthedocs.io/en/latest/index.html
There are several interesting features of it we still not support:

1. Create or access config by dot list instead of dict keys:
   https://omegaconf.readthedocs.io/en/latest/usage.html#from-a-dot-list
2. Create or override config by command line:
   https://omegaconf.readthedocs.io/en/latest/usage.html#from-command-line-arguments
   Added draft support in PR: 3482 add run API for common training, evaluation and inference #3832
3. Support default value if the target node is not existing:
   https://omegaconf.readthedocs.io/en/latest/usage.html#default-values
4. Use the value "???" to indicate mandatory values:
   https://omegaconf.readthedocs.io/en/latest/usage.html#mandatory-values
5. Serialization: https://omegaconf.readthedocs.io/en/latest/usage.html#serialization
   We tried to support it with TorchScript saving.
6. Relative interpolation for variables: https://omegaconf.readthedocs.io/en/latest/usage.html#variable-interpolation
   Similiar to our "@" mark with ID, but also support relative path.
7. Merge configs: https://omegaconf.readthedocs.io/en/latest/usage.html#merging-configurations
   Draft support in PR: 3482 add run API for common training, evaluation and inference #3832
8. Read-Only flag to protect config content: https://omegaconf.readthedocs.io/en/latest/usage.html#read-only-flag
9. Debugger integration: https://omegaconf.readthedocs.io/en/latest/usage.html#debugger-integration

I think we can try to support 3 with enhanced idea: use the "@" mark to define a placeholder, then parser gives the value.
And we can try to support 4 to make the base config similar to "base class with abstractmethod" that must be overridden.

For other features we may consider later.

Thanks.

[ericspod]

I have a PR now on the specification document #3834 which should have been a draft PR but oh well. It's very minimal compared to what @Nic-Ma prototyped here and lacks details about intent, use cases, design expectations, etc.

[wyli]

I think #3822 is now intuitive and flexible after a few rounds of refactoring. We may want to provide a system-wide flag to optionally disable eval() because it is too powerful and unsafe..

[Nic-Ma]

I put all the links of related PRs to the task steps in the above ticket description.

Thanks.

[Nic-Ma]

Related PR for inference example bundle: Project-MONAI/tutorials#604.

Thanks.

[wyli]

follow-up from the dev meeting:

• to generate a simple 'template' for monai bundle, including metadata and the required folder structure
• explore further the hugging face API
• look into integration with monai label
• meeting/demo with monai deploy

[ericspod]

Revising this tutorial to use our bundle would be a good exemplar: https://github.com/Project-MONAI/tutorials/blob/e3eea8704f5d002002f79cffec112c5c280476b4/modules/transfer_mmar.ipynb

[Nic-Ma]

After sharing with the internal NVIDIA Clara team, I put 2 missing features in the task feature.

 1. Config python logging properties in a file.
 2. Specify rank ID for component to run only on some ranks, for example, saving checkpoint in rank 0.

Thanks.

[Nic-Ma]

Feedback from MONAI deploy team:

1. How to detect / avoid dead-lock in macro replacement.
2. How to support real string starts with syntax, may add "" support.

Welcome more testing and feedback later.

Thanks.

[Nic-Ma]

Most of the tasks are completed, I created several new tickets to track the left items, and let's close this big first ticket now.

Thanks.