docs: dynamically generate command outputs (deepmodeling#4071)

## Summary by CodeRabbit - **New Features** - Introduced a new Sphinx extension to enhance documentation by including program output. - Updated the documentation structure for command-line help outputs across several commands to improve readability. - **Documentation** - Streamlined help documentation for the `dp compress`, `dp test`, `dp model-devi`, and `dp train` commands, focusing on concise output formats. - **Chores** - Added a new dependency for enhanced documentation capabilities.  --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
mtaillefumier · Sep 18, 2024 · eb3ef73 · eb3ef73
1 parent 54b48f3
commit eb3ef73
Show file tree

Hide file tree

Showing 7 changed files with 9 additions and 129 deletions.
diff --git a/doc/conf.py b/doc/conf.py
@@ -68,6 +68,7 @@
     "sphinxcontrib.bibtex",
     "sphinx_design",
     "autoapi.extension",
+    "sphinxcontrib.programoutput",
 ]
 
 # breathe_domain_by_extension = {

diff --git a/doc/freeze/compress.md b/doc/freeze/compress.md
@@ -76,61 +76,8 @@ $ dp compress --help
 
 An explanation will be provided
 
-```
-usage: dp compress [-h] [-v {DEBUG,3,INFO,2,WARNING,1,ERROR,0}] [-l LOG_PATH]
-                   [-m {master,collect,workers}] [-i INPUT] [-o OUTPUT]
-                   [-s STEP] [-e EXTRAPOLATE] [-f FREQUENCY]
-                   [-c CHECKPOINT_FOLDER]
-
-optional arguments:
-  -h, --help            show this help message and exit
-  -v {DEBUG,3,INFO,2,WARNING,1,ERROR,0}, --log-level {DEBUG,3,INFO,2,WARNING,1,ERROR,0}
-                        set verbosity level by string or number, 0=ERROR,
-                        1=WARNING, 2=INFO and 3=DEBUG (default: INFO)
-  -l LOG_PATH, --log-path LOG_PATH
-                        set log file to log messages to disk, if not
-                        specified, the logs will only be output to console
-                        (default: None)
-  -m {master,collect,workers}, --mpi-log {master,collect,workers}
-                        Set the manner of logging when running with MPI.
-                        'master' logs only on main process, 'collect'
-                        broadcasts logs from workers to master and 'workers'
-                        means each process will output its own log (default:
-                        master)
-  -i INPUT, --input INPUT
-                        The original frozen model, which will be compressed by
-                        the code (default: frozen_model.pb)
-  -o OUTPUT, --output OUTPUT
-                        The compressed model (default:
-                        frozen_model_compressed.pb)
-  -s STEP, --step STEP  Model compression uses fifth-order polynomials to
-                        interpolate the embedding-net. It introduces two
-                        tables with different step size to store the
-                        parameters of the polynomials. The first table covers
-                        the range of the training data, while the second table
-                        is an extrapolation of the training data. The domain
-                        of each table is uniformly divided by a given step
-                        size. And the step(parameter) denotes the step size of
-                        the first table and the second table will use 10 *
-                        step as it's step size to save the memory. Usually the
-                        value ranges from 0.1 to 0.001. Smaller step means
-                        higher accuracy and bigger model size (default: 0.01)
-  -e EXTRAPOLATE, --extrapolate EXTRAPOLATE
-                        The domain range of the first table is automatically
-                        detected by the code: [d_low, d_up]. While the second
-                        table ranges from the first table's upper
-                        boundary(d_up) to the extrapolate(parameter) * d_up:
-                        [d_up, extrapolate * d_up] (default: 5)
-  -f FREQUENCY, --frequency FREQUENCY
-                        The frequency of tabulation overflow check(Whether the
-                        input environment matrix overflow the first or second
-                        table range). By default do not check the overflow
-                        (default: -1)
-  -c CHECKPOINT_FOLDER, --checkpoint-folder CHECKPOINT_FOLDER
-                        path to checkpoint folder (default: .)
-  -t TRAINING_SCRIPT, --training-script TRAINING_SCRIPT
-                        The training script of the input frozen model
-                        (default: None)
+```{program-output} dp compress --help
+
 ```
 
 **Parameter explanation**

diff --git a/doc/install/install-from-source.md b/doc/install/install-from-source.md
@@ -225,20 +225,8 @@ dp -h
 
 It will print the help information like
 
-```text
-usage: dp [-h] {train,freeze,test} ...
+```{program-output} dp -h
 
-DeePMD-kit: A deep learning package for many-body potential energy
-representation and molecular dynamics
-
-optional arguments:
-  -h, --help           show this help message and exit
-
-Valid subcommands:
-  {train,freeze,test}
-    train              train a model
-    freeze             freeze the model
-    test               test the model
 ```
 
 ### Install horovod and mpi4py {{ tensorflow_icon }}

diff --git a/doc/test/model-deviation.md b/doc/test/model-deviation.md
@@ -61,34 +61,8 @@ dp model-devi -m graph.000.pb graph.001.pb graph.002.pb graph.003.pb -s ./data -
 
 where `-m` specifies model files to be calculated, `-s` gives the data to be evaluated, `-o` the file to which model deviation results are dumped. Here is more information on this sub-command:
 
-```bash
-usage: dp model-devi [-h] [-v {DEBUG,3,INFO,2,WARNING,1,ERROR,0}]
-                     [-l LOG_PATH] [-m MODELS [MODELS ...]] [-s SYSTEM]
-                     [-S SET_PREFIX] [-o OUTPUT] [-f FREQUENCY] [-i ITEMS]
-
-optional arguments:
-  -h, --help            show this help message and exit
-  -v {DEBUG,3,INFO,2,WARNING,1,ERROR,0}, --log-level {DEBUG,3,INFO,2,WARNING,1,ERROR,0}
-                        set verbosity level by string or number, 0=ERROR,
-                        1=WARNING, 2=INFO and 3=DEBUG (default: INFO)
-  -l LOG_PATH, --log-path LOG_PATH
-                        set log file to log messages to disk, if not
-                        specified, the logs will only be output to console
-                        (default: None)
-  -m MODELS [MODELS ...], --models MODELS [MODELS ...]
-                        Frozen models file to import (default:
-                        ['graph.000.pb', 'graph.001.pb', 'graph.002.pb',
-                        'graph.003.pb'])
-  -s SYSTEM, --system SYSTEM
-                        The system directory, not support recursive detection.
-                        (default: .)
-  -S SET_PREFIX, --set-prefix SET_PREFIX
-                        The set prefix (default: set)
-  -o OUTPUT, --output OUTPUT
-                        The output file for results of model deviation
-                        (default: model_devi.out)
-  -f FREQUENCY, --frequency FREQUENCY
-                        The trajectory frequency of the system (default: 1)
+```{program-output} dp model-devi -h
+
 ```
 
 For more details concerning the definition of model deviation and its application, please refer to [Yuzhi Zhang, Haidi Wang, Weijie Chen, Jinzhe Zeng, Linfeng Zhang, Han Wang, and Weinan E, DP-GEN: A concurrent learning platform for the generation of reliable deep learning based potential energy models, Computer Physics Communications, 2020, 253, 107206.](https://doi.org/10.1016/j.cpc.2020.107206)

diff --git a/doc/test/test.md b/doc/test/test.md
@@ -14,24 +14,6 @@ $ dp test --help
 
 An explanation will be provided
 
-```
-usage: dp test [-h] [-m MODEL] [-s SYSTEM] [-S SET_PREFIX] [-n NUMB_TEST]
-               [-r RAND_SEED] [--shuffle-test] [-d DETAIL_FILE]
+```{program-output} dp test -h
 
-optional arguments:
-  -h, --help            show this help message and exit
-  -m MODEL, --model MODEL
-                        Frozen model file to import
-  -s SYSTEM, --system SYSTEM
-                        The system dir
-  -S SET_PREFIX, --set-prefix SET_PREFIX
-                        The set prefix
-  -n NUMB_TEST, --numb-test NUMB_TEST
-                        The number of data for test
-  -r RAND_SEED, --rand-seed RAND_SEED
-                        The random seed
-  --shuffle-test        Shuffle test data
-  -d DETAIL_FILE, --detail-file DETAIL_FILE
-                        The prefix to files where details of energy, force and virial accuracy/accuracy per atom will be written
-  -a, --atomic          Test the accuracy of atomic label, i.e. energy / tensor (dipole, polar)
 ```
diff --git a/doc/train/training-advanced.md b/doc/train/training-advanced.md
@@ -135,21 +135,8 @@ $ dp train --help
 
 An explanation will be provided
 
-```
-positional arguments:
-  INPUT                 the input json database
-
-optional arguments:
-  -h, --help            show this help message and exit
-
-  --init-model INIT_MODEL
-                        Initialize a model by the provided checkpoint
-
-  --restart RESTART     Restart the training from the provided checkpoint
+```{program-output} dp train -h
 
-  --init-frz-model INIT_FRZ_MODEL
-                        Initialize the training from the frozen model.
-  --skip-neighbor-stat  Skip calculating neighbor statistics. Sel checking, automatic sel, and model compression will be disabled. (default: False)
 ```
 
 **`--init-model model.ckpt`**, initializes the model training with an existing model that is stored in the path prefix of checkpoint files `model.ckpt`, the network architectures should match.

diff --git a/pyproject.toml b/pyproject.toml
@@ -100,6 +100,7 @@ docs = [
     "pygments-lammps",
     "sphinxcontrib-bibtex",
     "sphinx-autoapi>=3.0.0",
+    "sphinxcontrib-programoutput",
 ]
 lmp = [
     "lammps~=2023.8.2.3.0",