From 7b9240d2d7a8f2f542f0f29fa1485271b4f10ea7 Mon Sep 17 00:00:00 2001
From: Stephen Roller <roller@fb.com>
Date: Wed, 12 Aug 2020 09:59:45 -0400
Subject: [PATCH 01/22] Add a custom metrics tutorial.

---
 docs/source/index.rst           |   1 +
 docs/source/tutorial_metrics.md | 370 ++++++++++++++++++++++++++++++++
 2 files changed, 371 insertions(+)
 create mode 100644 docs/source/tutorial_metrics.md

diff --git a/docs/source/index.rst b/docs/source/index.rst
index 78dd167ca92..eeb327d5feb 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -20,6 +20,7 @@ ParlAI is a one-stop-shop for dialog research.
    tutorial_worlds
    tutorial_torch_generator_agent
    tutorial_torch_ranker_agent
+   tutorial_metrics
    tutorial_fast
    tutorial_tipsntricks
    tutorial_mturk
diff --git a/docs/source/tutorial_metrics.md b/docs/source/tutorial_metrics.md
new file mode 100644
index 00000000000..ca2fcc2e8e7
--- /dev/null
+++ b/docs/source/tutorial_metrics.md
@@ -0,0 +1,370 @@
+# Understanding and adding new metrics
+
+Author: Stephen Roller
+
+## Introduction and Standard Metrics
+
+ParlAI contains a number of built-in metrics that are automatically computed when
+we train and evaluate models. Some of these metrics are _text generation_ metrics,
+which happen any time we generate a text: this includes F1, BLEU and Accuracy.
+
+For example, let's try a Fixed Response model, which always returns a given fixed
+response, and evaluate on the DailyDialog dataset:
+
+```
+$ parlai eval_model -m fixed_response -t dailydialog --fixed-response "how may i help you ?"
+... after a while ...
+14:41:40 | Evaluating task dailydialog using datatype valid.
+14:41:40 | creating task(s): dailydialog
+14:41:41 | Finished evaluating tasks ['dailydialog'] using datatype valid
+    accuracy  bleu-4  exs    f1
+    .0001239 .002617 8069 .1163
+```
+
+We see that we got 0.01239% accuracy, 0.26% BLEU-4 score, and 11.63% F1 across
+8069 examples. What do those metrics means?
+
+- Accuracy: this is perfect, exact, matching of the response, averaged across
+  all examples in the dataset
+- BLEU-4: this is the [BLEU score](https://en.wikipedia.org/wiki/BLEU) between
+  the predicted response and the reference response. It is measured on
+  tokenized text, and uses NLTK to compute it.
+- F1: This is the [Unigram](https://en.wikipedia.org/wiki/N-gram) F1 overlap
+  between your text and the reference response.
+- exs: the number of examples we have evaluated
+
+If you don't see the BLEU-4 score, you may need to install NLTK with
+`pip install nltk`.
+
+We can also measure ROUGE. Note that we need to `pip install py-rouge` for this
+functionality:
+
+```
+$ parlai eval_model -m fixed_response -t dailydialog --fixed-response "how may i help you ?" --metrics rouge
+14:47:24 | creating task(s): dailydialog
+14:47:31 | Finished evaluating tasks ['dailydialog'] using datatype valid
+    accuracy  exs    f1  rouge_1  rouge_2  rouge_L
+    .0001239 8069 .1163   .09887  .007285   .09525
+```
+
+### Agent-specific metrics
+
+Some agents include their own metrics that are computed for them. For example,
+generative models automatically compute `ppl`
+([perplexity](https://en.wikipedia.org/wiki/Perplexity)) and `token_acc`, both
+which measure the generative model's ability to predict indivdual tokens.  As
+an example, let's evaluate the [BlenderBot](https://parl.ai/projects/recipes/)
+90M model on DailyDialog:
+
+```
+$ parlai eval_model --task dailydialog -mf zoo:blender/blender_90M/model -bs 32
+...
+14:54:14 | Evaluating task dailydialog using datatype valid.
+14:54:14 | creating task(s): dailydialog
+...
+15:26:19 | Finished evaluating tasks ['dailydialog'] using datatype valid
+    accuracy  bleu-4  ctpb  ctps  exps  exs    f1  gpu_mem  loss      lr  ltpb  ltps   ppl  token_acc  total_train_updates   tpb   tps
+           0 .002097 14202 442.5 6.446 8069 .1345    .0384 2.979 7.5e-06  3242   101 19.67      .4133               339012 17445 543.5
+```
+
+Here we see a number of extra metrics, each of which we explain below:
+- `tpb`, `ctpb`, `ltpb`: stand for tokens per batch, context-tokens per batch,
+  and label-tokens per batch. These are useful for measuring how dense the
+  batches are, and are helpful when experimenting with [dynamic
+  batching](tutorial_fast).  tpb is always the sum of ctpb and lptb.
+- `tps`, `ctps`, `ltps`: are similar, but stand for "tokens per second". They
+  measure how fast we are training. Similarly, `exps` measures examples per
+  second.
+- `gpu_mem`: measures _roughly_ how much GPU memory your model is using, but it
+  is only approximate. This is useful for determining if you can possibly increase
+  the model size or the batch size.
+- `loss`: the loss metric
+- `ppl` and `token_acc`: the perplexity and per-token accuracy. these are generative
+  performance metrics.
+- `total_train_updates`: the number of SGD updates this model was trained for.
+  You will see this increase during training, but not during evaluation.
+
+## Adding custom metrics
+
+Of course, you may wish to add your own custom metrics: whether this is because
+you are developing a special model, special dataset, or otherwise want other
+information accessible to you. Metrics can be computed by either _the teacher_ OR
+_the model_. Within the model, they may be computed either _locally_ or _globally_.
+There are different reasons for why and where you would want to choose each
+location:
+
+- __Teacher metrics__: This is the best spot for computing metrics that depend
+  on a specific dataset. These metrics will only be available when evaluating
+  on this dataset. They have the advantage of being easy to compute and
+  understand.
+- __Global metrics__: Global metrics are computed by the model, and are globally
+  tracked. These metrics are easy to understand and track, but work poorly
+  when doing multitasking.
+- __Local metrics__: Local metrics are the model-analogue of teacher metrics.
+  They are computed and recorded on a per-example basis, and so they work well
+  when multitasking. They can be extremely complicated for some models, however.
+
+We will take you through writing each of these methods in turn, and demonstrate
+examples of how to add these metrics in your setup.
+
+## Teacher metrics
+
+Teacher metrics are useful for items that depend on a specific dataset.
+For example, in some of our task oriented datasets, like
+[`google_sgd`](https://github.com/facebookresearch/ParlAI/blob/master/parlai/tasks/google_sgd/agents.py),
+we want to additionally compute metrics around slots.
+
+Teacher metrics can be added by adding the following method to your teacher:
+
+```python
+    def custom_evaluation(
+        self,
+        teacher_action: Message,
+        labels: Optional[Tuple[str]],
+        model_response: Message,
+    ) -> None:
+        pass
+```
+
+The signature for this method is as follows:
+- `teacher_action`: this is the last message the teacher sent to the model. This likely
+   contains a "text" and "labels" field, as well as any custom fields you might
+   have.
+- `labels`: The gold label(s). This can also be found as information in the
+  `teacher_action`, but it is conveniently extracted for you.
+- `model_response`: The full model response, including any extra fields the model
+   may have sent.
+
+Let's take an actual example. We will add a custom metric which calculates
+how often the model says the word "hello", and call it `hello_avg`.
+
+We will add a [custom teacher](tutorial_task). For this example, we will use
+the `@register` syntax you may have seen in our [quickstart
+tutorial](tutorial_quick).
+
+```python
+from parlai.core.loader import register_teacher
+from parlai.core.metrics import AverageMetric
+from parlai.tasks.dailydialog.agents import DefaultTeacher as DailyDialogTeacher
+
+@register_teacher("hello_daily")
+class CustomDailyDialogTeacher(DailyDialogTeacher):
+    def custom_evaluation(
+        self, teacher_action, labels, model_response
+    ) -> None:
+        if 'text' not in model_response:
+            # model didn't speak, skip this example
+            return
+        model_text = model_response['text']
+        if 'hello' in model_text:
+            # count 1 / 1 messages having "hello"
+            self.metrics.add('hello_avg', AverageMetric(1, 1))
+        else:
+            # count 0 / 1 messages having "hello"
+            self.metrics.add('hello_avg', AverageMetric(0, 1))
+
+if __name__ == '__main__':
+    from parlai.scripts.eval_model import EvalModel
+
+    EvalModel.main(
+       task='hello_daily',
+       model_file='zoo:blender/blender_90M/model',
+       batchsize=32,
+    )
+```
+
+If we run the script, we will have a new metric in our output:
+
+```
+18:07:30 | Finished evaluating tasks ['hello_daily'] using datatype valid
+    accuracy  bleu-4  ctpb  ctps  exps  exs    f1  gpu_mem  hello_avg  loss  ltpb  ltps   ppl  token_acc  tpb   tps
+           0 .002035  2172   230 3.351 8069 .1346   .05211      .1228 2.979 495.9 52.52 19.67      .4133 2668 282.6
+```
+
+__What is AverageMetric?__
+
+Wait, what is this [`AverageMetric`](parlai.core.metrics.AverageMetric)? All
+metrics you want to create in ParlAI should be a
+[`Metric`](parlai.core.metrics.Metric) object. Metric objects define a way of
+instantiating the metric, a way of combining it with a like-metric, and a way
+of rendering it as a single float value. For an AverageMetric, this means we
+need to define a numerator and a denominator; the combination of AverageMetrics
+adds their numerators and denominators separately. As we do this across all
+examples, the numerator will be the number of examples with "hello" in it,
+and the denominator will be the total number of examples. When we go to print
+the metric, the division will be computed at the last second.
+
+If you're used to writing machine learning code in one-off scripts, you may ask
+why do I need to use this metric? Can't I just count and divide myself? While
+you can do this, your code could not be run in [_distributed
+mode_](tutorial_fast). If we only returned a single float, we would not be able
+to know if some distributed workers received more or fewer examples than
+others. However, when we explicitly store the numerator and denominator, we can
+combine and reduce the across multiple nodes, enabling us to train on hundreds
+of GPUs, while still ensuring correctness in all our metrics.
+
+In addition to AverageMetric, there is also
+[SumMetric](parlai.core.metrics.SumMetric), which keeps a running sum. SumMetric
+and AverageMetric are the most common ways to construct custom metrics, but others
+exist as well. For a full list (and views into advanced cases), please
+see the [metrics API documentation](parlai.core.metrics).
+
+## Agent (model) level metrics
+
+In the above example, we worked on a metric defined by a Teacher. However,
+sometimes our models will have special metrics that only they want to compute,
+which we call an Agent-level metric. Perplexity is one example.
+
+To compute model-level metrics, we can define either a global metric, or a
+local metric. Global metrics can be computed anywhere, and are easy to use,
+but cannot distinguish between different teachers when multitasking. We'll
+look at another example, counting the number of times the teacher says "hello".
+
+### Global metrics
+
+A global metric is computed anywhere in the model, and has an
+interface similar to that of the teacher:
+
+```python
+agent.global_metrics.add('my_metric', AverageMetric(1, 2))
+```
+
+Global metrics are called as such because they can be called anywhere in agent
+code. For example, we can add a metric that counts the number of times the
+model sees the word "hello" in `observe`. We'll do this while extending
+the `TransformerGeneratorAgent`, so that we can combined it with the BlenderBot
+model we used earlier.
+
+```python
+from parlai.core.metrics import AverageMetric
+from parlai.core.loader import register_agent
+from parlai.agents.transformer.transformer import TransformerGeneratorAgent
+
+
+@register_agent('GlobalHelloCounter')
+class GlobalHelloCounterAgent(TransformerGeneratorAgent):
+    def observe(self, observation):
+        retval = super().observe(observation)
+        if 'text' in observation:
+            text = observation['text']
+            self.global_metrics.add(
+                'global_hello', AverageMetric(int('hello' in text), 1)
+            )
+        return retval
+
+
+if __name__ == '__main__':
+    from parlai.scripts.eval_model import EvalModel
+
+    EvalModel.main(
+        task='dailydialog',
+        model='GlobalHelloCounter',
+        model_file='zoo:blender/blender_90M/model',
+        batchsize=32,
+    )
+```
+
+Running the script, we see that our new metric appears. Note that it is
+different than the metric in the first half of the tutorial: that is because
+previously we were counting the number of times the model said hello (a lot),
+but now we are counting how often the dataset says hello.
+
+```
+21:57:50 | Finished evaluating tasks ['dailydialog'] using datatype valid
+    accuracy  bleu-4  ctpb  ctps  exps  exs    f1  global_hello  gpu_mem  loss  ltpb  ltps   ppl  token_acc   tpb   tps
+           0 .002097 14202 435.1 6.338 8069 .1345      .0009914   .02795 2.979  3242 99.32 19.67      .4133 17445 534.4
+```
+
+The global metric works well, but have some drawbacks: if we were to start
+training on a multitask datasets, we would not be able to distinguish the
+`global_hello` of the two datasets, and we could only compute the micro-average
+of the combination of the two. Below is an excerpt from a training log with
+the above agents:
+
+```
+09:14:52 | time:112s total_exs:90180 epochs:0.41
+                clip  ctpb  ctps  exps  exs  global_hello  gnorm  gpu_mem  loss  lr  ltpb  ltps   ppl  token_acc  total_train_updates   tpb   tps   ups
+   all             1  9831 66874 841.9 8416        .01081  2.018    .3474 5.078   1  1746 11878 163.9      .2370                  729 11577 78752 6.803
+   convai2                             3434        .01081                 5.288                 197.9      .2120
+   dailydialog                         4982        .01081                 4.868                   130      .2620
+```
+
+Notice how `global_hello` is the same in both, because the model is unable to
+distinguish between the two settings. In the next section we'll show how to fix
+this with local metrics.
+
+__On placement__: In the example above, we recorded the global metric inside
+the `observe` function. However, global metrics can be recorded from anywhere.
+
+### Local metrics
+
+Having observed the limitation of global metrics being unable to distinguish
+settings in multitasking, we would like to improve upon this. Let's add a local
+metric, which is recorded _per example_. By recording this metric per example,
+we can unambiguously identify which metrics came from which dataset, and report
+averages correctly.
+
+Local metrics have a limitation: they can only be computed inside the scope of
+`batch_act`. This includes common places like `compute_loss` or `generate`,
+where we often want to instrument specific behavior.
+
+Let's look at an example. We'll add a metric inside the `batchify` function,
+which is called from within `batch_act`, and is used to convert from a list of
+[Messages](messages) objects to a
+[Batch](parlai.core.torch_agent.Batch) object. It is where we do things like
+padding, etc. We'll do something slightly different than our previous runs.
+In this case, we'll count the number of _tokens_ which are the word "hello".
+
+```python
+from parlai.core.metrics import AverageMetric
+from parlai.core.loader import register_agent
+from parlai.agents.transformer.transformer import TransformerGeneratorAgent
+
+
+@register_agent('LocalHelloCounter')
+class LocalHelloCounterAgent(TransformerGeneratorAgent):
+    def batchify(self, observations):
+        batch = super().batchify(observations)
+        if hasattr(batch, 'text_vec'):
+            num_hello = ["hello" in o['text'] for o in observations]
+            self.record_local_metric(
+                'local_hello',
+                # AverageMetric.many(seq1) is shorthand for
+                # [AverageMetric(item) for item in seq)
+                AverageMetric.many(num_hello),
+            )
+        return batch
+
+
+if __name__ == '__main__':
+    from parlai.scripts.train_model import TrainModel
+
+    TrainModel.main(
+        task='dailydialog,convai2',
+        model='LocalHelloCounter',
+        dict_file='zoo:blender/blender_90M/model.dict',
+        batchsize=32,
+    )
+```
+
+When we run this training script, we get one such output:
+```
+09:49:00 | time:101s total_exs:56160 epochs:0.26
+                clip  ctpb  ctps  exps  exs  gnorm  gpu_mem  local_hello  loss  lr  ltpb  ltps   ppl  token_acc  total_train_updates  tpb   tps  ups
+   all             1  3676 63204 550.2 5504  2.146    .1512       .01423 4.623   1 436.2  7500 101.8      .2757                 1755 4112 70704 17.2
+   convai2                             3652                       .02793 4.659                 105.5      .2651
+   dailydialog                         1852                       .00054 4.587                 98.17      .2863
+```
+
+Notice how the `local_hello` metric can now distinguish between hellos coming from
+convai2 and those coming from daily dialog? The average hides the fact that one
+dataset has many hellos, and the other does not.
+
+Local metrics are primarily worth the implementation when you care about the
+fidelity of _train time metrics_. During evaluation time, we evaluate each
+dataset individually, so we can ensure global metrics are not mixed up.
+
+__Under the hood__: Local metrics work by including a "metrics" field in the
+return message. This is a dictionary which maps field name to a metric value.
+When the teacher receives the response from the model, it utilizes the metrics
+field to update counters on its side.

From c6446cb1b91a5654aec37586833774ac52896190 Mon Sep 17 00:00:00 2001
From: Stephen Roller <roller@fb.com>
Date: Wed, 12 Aug 2020 16:45:25 -0400
Subject: [PATCH 02/22] Switch from recommonmark to myst for better parsing.

---
 docs/source/conf.py                          | 11 +------
 docs/source/index.rst                        |  2 +-
 docs/source/{metrics.rst => metrics_api.rst} |  5 ++-
 docs/source/tutorial_metrics.md              | 32 +++++++++++---------
 parlai/agents/bart/README.md                 |  4 +--
 requirements.txt                             |  2 +-
 6 files changed, 26 insertions(+), 30 deletions(-)
 rename docs/source/{metrics.rst => metrics_api.rst} (53%)

diff --git a/docs/source/conf.py b/docs/source/conf.py
index 653ddf14ca7..e122bf74ad4 100644
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -39,12 +39,11 @@
 import git
 import sphinx_rtd_theme
 import parlai
-from recommonmark.transform import AutoStructify
 
 extensions = [
     'sphinx.ext.autodoc',
     'sphinx.ext.githubpages',
-    'recommonmark',
+    'myst_parser',
     'sphinx.ext.linkcode',
 ]
 
@@ -207,11 +206,3 @@ def find_source():
         return "https://github.com/facebookresearch/ParlAI/blob/%s/%s" % (tag, filename)
     except Exception:
         return None
-
-
-# At the bottom of conf.py
-def setup(app):
-    app.add_config_value(
-        'recommonmark_config', {'auto_toc_tree_section': 'Contents'}, True
-    )
-    app.add_transform(AutoStructify)
diff --git a/docs/source/index.rst b/docs/source/index.rst
index eeb327d5feb..01f36e2165b 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -49,7 +49,7 @@ ParlAI is a one-stop-shop for dialog research.
    torch_agent
    build_data
    dict
-   metrics
+   metrics_api
    params
    teachers
    worlds
diff --git a/docs/source/metrics.rst b/docs/source/metrics_api.rst
similarity index 53%
rename from docs/source/metrics.rst
rename to docs/source/metrics_api.rst
index 994af38ebaa..7e864701cc6 100644
--- a/docs/source/metrics.rst
+++ b/docs/source/metrics_api.rst
@@ -1,4 +1,7 @@
-core.metrics
+Top level :any:`parlai.core.metrics.AverageMetric`
+
+
+parlai.core.metrics
 ===================================
 .. automodule:: parlai.core.metrics
   :members:
diff --git a/docs/source/tutorial_metrics.md b/docs/source/tutorial_metrics.md
index ca2fcc2e8e7..650ba4ceb7e 100644
--- a/docs/source/tutorial_metrics.md
+++ b/docs/source/tutorial_metrics.md
@@ -183,16 +183,18 @@ If we run the script, we will have a new metric in our output:
 
 __What is AverageMetric?__
 
-Wait, what is this [`AverageMetric`](parlai.core.metrics.AverageMetric)? All
-metrics you want to create in ParlAI should be a
-[`Metric`](parlai.core.metrics.Metric) object. Metric objects define a way of
-instantiating the metric, a way of combining it with a like-metric, and a way
-of rendering it as a single float value. For an AverageMetric, this means we
-need to define a numerator and a denominator; the combination of AverageMetrics
-adds their numerators and denominators separately. As we do this across all
-examples, the numerator will be the number of examples with "hello" in it,
-and the denominator will be the total number of examples. When we go to print
-the metric, the division will be computed at the last second.
+Wait, what is this
+[AverageMetric](parlai.core.metrics.AverageMetric)? All metrics
+you want to create in ParlAI should be a
+[Metric](Metric) object. Metric objects
+define a way of instantiating the metric, a way of combining it with a
+like-metric, and a way of rendering it as a single float value. For an
+AverageMetric, this means we need to define a numerator and a denominator; the
+combination of AverageMetrics adds their numerators and denominators
+separately. As we do this across all examples, the numerator will be the number
+of examples with "hello" in it, and the denominator will be the total number of
+examples. When we go to print the metric, the division will be computed at the
+last second.
 
 If you're used to writing machine learning code in one-off scripts, you may ask
 why do I need to use this metric? Can't I just count and divide myself? While
@@ -204,10 +206,10 @@ combine and reduce the across multiple nodes, enabling us to train on hundreds
 of GPUs, while still ensuring correctness in all our metrics.
 
 In addition to AverageMetric, there is also
-[SumMetric](parlai.core.metrics.SumMetric), which keeps a running sum. SumMetric
-and AverageMetric are the most common ways to construct custom metrics, but others
-exist as well. For a full list (and views into advanced cases), please
-see the [metrics API documentation](parlai.core.metrics).
+[SumMetric](parlai.core.metrics.SumMetric), which keeps a running
+sum. SumMetric and AverageMetric are the most common ways to construct custom
+metrics, but others exist as well. For a full list (and views into advanced
+cases), please see the [metrics API documentation](metrics_api).
 
 ## Agent (model) level metrics
 
@@ -311,7 +313,7 @@ where we often want to instrument specific behavior.
 Let's look at an example. We'll add a metric inside the `batchify` function,
 which is called from within `batch_act`, and is used to convert from a list of
 [Messages](messages) objects to a
-[Batch](parlai.core.torch_agent.Batch) object. It is where we do things like
+[Batch](torch_agent.html#parlai.core.torch_agent.Batch) object. It is where we do things like
 padding, etc. We'll do something slightly different than our previous runs.
 In this case, we'll count the number of _tokens_ which are the word "hello".
 
diff --git a/parlai/agents/bart/README.md b/parlai/agents/bart/README.md
index 7c4420edcfa..1135ff3d65a 100644
--- a/parlai/agents/bart/README.md
+++ b/parlai/agents/bart/README.md
@@ -9,7 +9,7 @@ The BART agent can be instantiated as simply `-m bart`, however it is recommende
 
 ## Basic Examples
 
-#### Train BART on convai2.
+### Train BART on convai2.
 ```bash
 parlai train_model -m bart -mf /tmp/model_file -t convai2 -bs 24 --fp16 true -eps 1 -lr 1e-5 --optimizer adam
 ```
@@ -20,7 +20,7 @@ or
 parlai train_model -m bart --init-model zoo:bart/bart_large/model -mf /tmp/model_file -t convai2 -bs 24 --fp16 true -eps 1 -lr 1e-5 --optimizer adam
 ```
 
-#### Interact with a BART Model fine-tuned in fairseq
+### Interact with a BART Model fine-tuned in fairseq
 
 ```bash
 parlai interactive -m bart --init-fairseq-model /path/to/fairseq/model
diff --git a/requirements.txt b/requirements.txt
index 4e5edc75efd..52b71029aec 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -19,7 +19,7 @@ py-rouge==1.1
 pyyaml==5.3.1
 pyzmq==18.1.0
 regex==2020.1.8
-recommonmark==0.6.0
+myst-parser==0.11.1
 requests-mock==1.7.0
 requests==2.22.0
 scikit-learn==0.23.1

From 6d8eabb4eff3a09a46bdfea0c14ef3ce9877d974 Mon Sep 17 00:00:00 2001
From: Stephen Roller <roller@fb.com>
Date: Wed, 12 Aug 2020 18:12:48 -0400
Subject: [PATCH 03/22] Radically update the autodocs.

---
 docs/source/_static/css/parlai_theme.css      |  4 +
 docs/source/agents.rst                        |  2 +-
 .../{agents_list.md => agents_list.rst}       |  5 +-
 docs/source/build_data.rst                    |  2 +-
 docs/source/dict.rst                          |  2 +-
 docs/source/generate_agent_list.py            | 36 +++++---
 docs/source/index.rst                         |  8 +-
 docs/source/messages.rst                      |  9 +-
 docs/source/metrics_api.rst                   |  3 -
 docs/source/params.rst                        |  7 +-
 docs/source/teachers.rst                      |  2 +-
 docs/source/torch_agent.rst                   | 19 +++-
 docs/source/utils.md                          | 27 ------
 docs/source/utils.rst                         | 88 +++++++++++++++++++
 docs/source/worlds.rst                        |  2 +-
 parlai/core/loader.py                         | 20 +++--
 parlai/utils/conversations.py                 | 39 ++++----
 17 files changed, 189 insertions(+), 86 deletions(-)
 rename docs/source/{agents_list.md => agents_list.rst} (91%)
 delete mode 100644 docs/source/utils.md
 create mode 100644 docs/source/utils.rst

diff --git a/docs/source/_static/css/parlai_theme.css b/docs/source/_static/css/parlai_theme.css
index 6ea066e4ef6..f526b97b56d 100644
--- a/docs/source/_static/css/parlai_theme.css
+++ b/docs/source/_static/css/parlai_theme.css
@@ -26,6 +26,10 @@ h1, h2, .rst-content .toctree-wrapper p.caption, h3, h4, h5, h6, legend, p.capti
     background-color: #fff;
 }
 
+th > p {
+    margin-bottom: 0px !important;
+}
+
 @media screen and (min-width: 1400px) {
     .wy-nav-content-wrap {
         background-color: rgba(0, 0, 0, 0.0470588);
diff --git a/docs/source/agents.rst b/docs/source/agents.rst
index 86fad1659a0..9d68fe91d61 100644
--- a/docs/source/agents.rst
+++ b/docs/source/agents.rst
@@ -1,4 +1,4 @@
-core.agents
+parlai.core.agents
 ===================================
 .. automodule:: parlai.core.agents
   :members:
diff --git a/docs/source/agents_list.md b/docs/source/agents_list.rst
similarity index 91%
rename from docs/source/agents_list.md
rename to docs/source/agents_list.rst
index d87f79ae9b0..db14bb57ee2 100644
--- a/docs/source/agents_list.md
+++ b/docs/source/agents_list.rst
@@ -1,15 +1,14 @@
-# Standard Agents
+Standard Agents
+---------------
 
 ParlAI comes with a wide variety of standardized agents. Some of these are
 common baselines made available for quicker research; some can be made
 state-of-the-art with careful tuning and experimentation. Each of these contains
 a README with examples of usage and description of the agent.
 
-```eval_rst
 .. toctree::
    :maxdepth: 1
    :caption: List of Agents
    :glob:
 
    agent_refs/*
-```
diff --git a/docs/source/build_data.rst b/docs/source/build_data.rst
index f8d4ce58e8c..766e6d064f9 100644
--- a/docs/source/build_data.rst
+++ b/docs/source/build_data.rst
@@ -1,4 +1,4 @@
-core.build_data
+parlai.core.build_data
 ===================================
 .. automodule:: parlai.core.build_data
   :members:
diff --git a/docs/source/dict.rst b/docs/source/dict.rst
index a6630515831..38eba670247 100644
--- a/docs/source/dict.rst
+++ b/docs/source/dict.rst
@@ -1,4 +1,4 @@
-core.dict
+parlai.core.dict
 ===================================
 .. automodule:: parlai.core.dict
   :members:
diff --git a/docs/source/generate_agent_list.py b/docs/source/generate_agent_list.py
index 6c59a57d64a..039db3411a5 100644
--- a/docs/source/generate_agent_list.py
+++ b/docs/source/generate_agent_list.py
@@ -7,6 +7,7 @@
 import parlai.agents
 from parlai.core.params import ParlaiParser
 
+import argparse
 import os
 import pkgutil
 import importlib
@@ -35,7 +36,7 @@ def _make_argparse_table(class_):
             if hasattr(action, 'hidden') and action.hidden:
                 # some options are marked hidden
                 continue
-            action_strings = ",  ".join(f'``{a}``' for a in action.option_strings)
+            action_strings = ",  ".join(f'`{a}`' for a in action.option_strings)
             description = []
             # start with the help message
             if action.help:
@@ -46,11 +47,11 @@ def _make_argparse_table(class_):
             # list choices if there are any
             if action.choices:
                 description += [
-                    "Choices: " + ", ".join(f'``{c}``' for c in action.choices) + "."
+                    "Choices: " + ", ".join(f'`{c}`' for c in action.choices) + "."
                 ]
             # list default and recommended values.
             default_value = ""
-            if action.default is not None:
+            if action.default is not None and action.default is not argparse.SUPPRESS:
                 default_value += f"Default: ``{action.default}``.  "
             if hasattr(action, 'recommended') and action.recommended:
                 default_value += f"Recommended: ``{action.recommended}``. "
@@ -68,17 +69,26 @@ def _make_argparse_table(class_):
         if not actions:
             continue
 
-        # render the table
-        readme.append(f"```eval_rst\n")
-        readme.append(f".. csv-table:: {ag.title}\n")
-        readme.append(f'   :widths: 35, 65\n\n')
-        cout = io.StringIO()
-        csvw = csv.writer(cout, csv.unix_dialect, delimiter=",")
+        readme.append(f'_{ag.title}_\n\n')
+        readme.append("| Argument | Description |\n")
+        readme.append("|----------|----------|\n")
         for row in actions:
-            cout.write("   ")
-            csvw.writerow(row)
-        readme.append(cout.getvalue())
-        readme.append("```\n\n")
+            text = ("| " + " | ".join(row) + " |")
+            text = text.replace("\n", "<br>")
+            readme.append(f"{text}\n")
+        readme.append("\n\n")
+
+        # render the table
+        # readme.append(f"```eval_rst\n")
+        # readme.append(f".. csv-table:: {ag.title}\n")
+        # readme.append(f'   :widths: 35, 65\n\n')
+        # cout = io.StringIO()
+        # csvw = csv.writer(cout, csv.unix_dialect, delimiter=",")
+        # for row in actions:
+        #     cout.write("   ")
+        #     csvw.writerow(row)
+        # readme.append(cout.getvalue())
+        # readme.append("```\n\n")
     return readme
 
 
diff --git a/docs/source/index.rst b/docs/source/index.rst
index 01f36e2165b..9021eaea339 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -42,16 +42,18 @@ ParlAI is a one-stop-shop for dialog research.
 
 .. toctree::
    :maxdepth: 1
-   :caption: Core Library API
+   :caption: API Reference
 
-   messages
    agents
    torch_agent
-   build_data
+   messages
    dict
+   build_data
    metrics_api
    params
    teachers
+   loader
+   script
    worlds
    utils
 
diff --git a/docs/source/messages.rst b/docs/source/messages.rst
index 2836f55007d..ad9152de9b5 100644
--- a/docs/source/messages.rst
+++ b/docs/source/messages.rst
@@ -1,5 +1,10 @@
-messages
-============
+parlai.core.message
+===================================
+
+.. automodule:: parlai.core.message
+  :members:
+
+
 
 .. image:: _static/img/act-obs-dict.png
     :width: 60 %
diff --git a/docs/source/metrics_api.rst b/docs/source/metrics_api.rst
index 7e864701cc6..77bb7e56b52 100644
--- a/docs/source/metrics_api.rst
+++ b/docs/source/metrics_api.rst
@@ -1,6 +1,3 @@
-Top level :any:`parlai.core.metrics.AverageMetric`
-
-
 parlai.core.metrics
 ===================================
 .. automodule:: parlai.core.metrics
diff --git a/docs/source/params.rst b/docs/source/params.rst
index 6673e5a0010..18f3c7dfb28 100644
--- a/docs/source/params.rst
+++ b/docs/source/params.rst
@@ -1,4 +1,9 @@
-core.params
+parlai.core.opt
+===================================
+.. automodule:: parlai.core.opt
+  :members:
+
+parlai.core.params
 ===================================
 .. automodule:: parlai.core.params
   :members:
diff --git a/docs/source/teachers.rst b/docs/source/teachers.rst
index 7824a34f066..4512ebad71d 100644
--- a/docs/source/teachers.rst
+++ b/docs/source/teachers.rst
@@ -1,4 +1,4 @@
-core.teachers
+parlai.core.teachers
 ===================================
 .. automodule:: parlai.core.teachers
   :members:
diff --git a/docs/source/torch_agent.rst b/docs/source/torch_agent.rst
index 3b4eda1db4f..94043182165 100644
--- a/docs/source/torch_agent.rst
+++ b/docs/source/torch_agent.rst
@@ -1,4 +1,4 @@
-core.torch_agent
+parlai.core.torch_agent
 ===================================
 
 Torch Agent implements much of the boilerplate necessary for creating
@@ -21,13 +21,26 @@ Torch Agent
   :members:
 
 
+Torch Generator Agent
+-----------------------------------
+.. automodule:: parlai.core.torch_generator_agent
+  :members:
+
+
 Torch Ranker Agent
 -----------------------------------
 .. automodule:: parlai.core.torch_ranker_agent
   :members:
 
 
-Torch Generator Agent
+Torch Classifier Agent
 -----------------------------------
-.. automodule:: parlai.core.torch_generator_agent
+.. automodule:: parlai.core.torch_classifier_agent
+  :members:
+
+
+
+Torch Image Agent
+-----------------------------------
+.. automodule:: parlai.core.torch_image_agent
   :members:
diff --git a/docs/source/utils.md b/docs/source/utils.md
deleted file mode 100644
index 0a990cf8c8c..00000000000
--- a/docs/source/utils.md
+++ /dev/null
@@ -1,27 +0,0 @@
-<!--
-  Copyright (c) Facebook, Inc. and its affiliates.
-  This source code is licensed under the MIT license found in the
-  LICENSE file in the root directory of this source tree.
--->
-
-# Utilties
-ParlAI has very many utilities, roughly organized by function.
-
-## Thread Utilities
-```eval_rst
-.. automodule:: parlai.utils.thread
-  :members:
-```
-
-
-## Torch Utilities
-```eval_rst
-.. automodule:: parlai.utils.torch
-  :members:
-```
-
-## Uncategorized Utils
-```eval_rst
-.. automodule:: parlai.utils.misc
-  :members:
-```
diff --git a/docs/source/utils.rst b/docs/source/utils.rst
new file mode 100644
index 00000000000..b19546ff90b
--- /dev/null
+++ b/docs/source/utils.rst
@@ -0,0 +1,88 @@
+parlai.utils
+------------
+ParlAI has very many utilities, roughly organized by function.
+
+parlai.utils.bpe
+================
+.. automodule:: parlai.utils.bpe
+  :members:
+
+
+parlai.utils.conversations
+==========================
+.. automodule:: parlai.utils.conversations
+  :members:
+
+
+parlai.utils.data
+=================
+.. automodule:: parlai.utils.data
+  :members:
+
+
+parlai.utils.distributed
+========================
+.. automodule:: parlai.utils.distributed
+  :members:
+
+
+parlai.utils.fp16
+=================
+.. automodule:: parlai.utils.fp16
+  :members:
+
+
+parlai.utils.logging
+====================
+.. automodule:: parlai.utils.logging
+  :members:
+
+
+parlai.utils.misc
+=================
+.. automodule:: parlai.utils.misc
+  :members:
+
+
+parlai.utils.pickle
+===================
+.. automodule:: parlai.utils.pickle
+  :members:
+
+
+parlai.utils.safety
+===================
+.. automodule:: parlai.utils.safety
+  :members:
+
+
+parlai.utils.strings
+====================
+.. automodule:: parlai.utils.strings
+  :members:
+
+
+parlai.utils.testing
+====================
+.. automodule:: parlai.utils.testing
+  :members:
+
+
+parlai.utils.torch
+==================
+.. automodule:: parlai.utils.torch
+  :members:
+
+
+parlai.utils.typing
+===================
+.. automodule:: parlai.utils.typing
+  :members:
+
+
+parlai.utils.world_logging
+==========================
+.. automodule:: parlai.utils.world_logging
+  :members:
+
+
diff --git a/docs/source/worlds.rst b/docs/source/worlds.rst
index d6c34f27b6d..b1de8194529 100644
--- a/docs/source/worlds.rst
+++ b/docs/source/worlds.rst
@@ -1,4 +1,4 @@
-core.worlds
+parlai.core.worlds
 ===================================
 .. automodule:: parlai.core.worlds
   :members:
diff --git a/parlai/core/loader.py b/parlai/core/loader.py
index 7772ddd9096..158d9b47aa1 100644
--- a/parlai/core/loader.py
+++ b/parlai/core/loader.py
@@ -221,12 +221,16 @@ def load_task_module(taskname: str):
     """
     Get the module containing all teacher agents for the task specified by `--task`.
 
-    :param taskname: path to task class in one of the following formats:
-        * full: ``-t parlai.tasks.babi.agents:DefaultTeacher``
-        * shorthand: ``-t babi``, which will check
-            ``parlai.tasks.babi.agents:DefaultTeacher``
-        * shorthand specific: ``-t babi:task10k``, which will check
-            ``parlai.tasks.babi.agents:Task10kTeacher``
+    :param taskname:
+        path to task class in one of the formats
+
+    The valid formats for taskname are:
+
+      - full: ``-t parlai.tasks.babi.agents:DefaultTeacher``
+      - shorthand: ``-t babi``, which will check
+        ``parlai.tasks.babi.agents:DefaultTeacher``
+      - shorthand specific: ``-t babi:task10k``, which will check
+        ``parlai.tasks.babi.agents:Task10kTeacher``
 
     :return:
         module containing all teacher agents for a task
@@ -253,9 +257,9 @@ def load_teacher_module(taskname: str):
 
     * full: ``-t parlai.tasks.babi.agents:DefaultTeacher``
     * shorthand: ``-t babi``, which will check
-        ``parlai.tasks.babi.agents:DefaultTeacher``
+      ``parlai.tasks.babi.agents:DefaultTeacher``
     * shorthand specific: ``-t babi:task10k``, which will check
-        ``parlai.tasks.babi.agents:Task10kTeacher``
+      ``parlai.tasks.babi.agents:Task10kTeacher``
 
     The base path to search when using shorthand formats can be changed from
     "parlai" to "parlai_internal" by prepending "internal:" to the path, e.g.
diff --git a/parlai/utils/conversations.py b/parlai/utils/conversations.py
index a1b8741d571..7b2b6d73353 100644
--- a/parlai/utils/conversations.py
+++ b/parlai/utils/conversations.py
@@ -22,7 +22,7 @@ class Metadata:
     """
     Utility class for conversation metadata.
 
-    Metadata should be saved at <datapath>.metadata.
+    Metadata should be saved at ``<datapath>.metadata``.
     """
 
     def __init__(self, datapath):
@@ -169,24 +169,27 @@ class Conversations:
 
     Conversations should be saved in JSONL format, where each line is
     a JSON of the following form:
-    {
-        'possible_conversation_level_info': True,
-        'dialog':
-            [   [
-                    {
-                        'id': 'speaker_1',
-                        'text': <first utterance>,
-                    },
-                    {
-                        'id': 'speaker_2',
-                        'text': <second utterance>,
-                    },
+
+    .. code-block:
+
+        {
+            'possible_conversation_level_info': True,
+            'dialog':
+                [   [
+                        {
+                            'id': 'speaker_1',
+                            'text': <first utterance>,
+                        },
+                        {
+                            'id': 'speaker_2',
+                            'text': <second utterance>,
+                        },
+                        ...
+                    ],
                     ...
-                ],
-                ...
-            ]
-        ...
-    }
+                ]
+            ...
+        }
     """
 
     def __init__(self, datapath):

From d4faec573fe8ceef697ad35b49620478ce2c2522 Mon Sep 17 00:00:00 2001
From: Stephen Roller <roller@fb.com>
Date: Wed, 12 Aug 2020 18:17:19 -0400
Subject: [PATCH 04/22] new files.

---
 docs/source/loader.rst | 4 ++++
 docs/source/script.rst | 4 ++++
 2 files changed, 8 insertions(+)
 create mode 100644 docs/source/loader.rst
 create mode 100644 docs/source/script.rst

diff --git a/docs/source/loader.rst b/docs/source/loader.rst
new file mode 100644
index 00000000000..84755768f27
--- /dev/null
+++ b/docs/source/loader.rst
@@ -0,0 +1,4 @@
+parlai.core.loader
+===================================
+.. automodule:: parlai.core.loader
+  :members:
diff --git a/docs/source/script.rst b/docs/source/script.rst
new file mode 100644
index 00000000000..a6a109ab093
--- /dev/null
+++ b/docs/source/script.rst
@@ -0,0 +1,4 @@
+parlai.core.script
+===================================
+.. automodule:: parlai.core.script
+  :members:

From 9f2454839a52b14f4dfc44cc47c9e7443866acf1 Mon Sep 17 00:00:00 2001
From: Stephen Roller <roller@fb.com>
Date: Wed, 12 Aug 2020 18:29:12 -0400
Subject: [PATCH 05/22] Black

---
 docs/source/generate_agent_list.py | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/docs/source/generate_agent_list.py b/docs/source/generate_agent_list.py
index 039db3411a5..615fccd3670 100644
--- a/docs/source/generate_agent_list.py
+++ b/docs/source/generate_agent_list.py
@@ -12,8 +12,6 @@
 import pkgutil
 import importlib
 import inspect
-import io
-import csv
 
 
 """
@@ -73,7 +71,7 @@ def _make_argparse_table(class_):
         readme.append("| Argument | Description |\n")
         readme.append("|----------|----------|\n")
         for row in actions:
-            text = ("| " + " | ".join(row) + " |")
+            text = "| " + " | ".join(row) + " |"
             text = text.replace("\n", "<br>")
             readme.append(f"{text}\n")
         readme.append("\n\n")

From 2ec1b354a162d317d362d6aaad7ce99aeee40b0a Mon Sep 17 00:00:00 2001
From: Stephen Roller <roller@fb.com>
Date: Wed, 12 Aug 2020 19:25:09 -0400
Subject: [PATCH 06/22] Remove metrics tutorial from PR.

---
 docs/source/tutorial_metrics.md | 372 --------------------------------
 1 file changed, 372 deletions(-)
 delete mode 100644 docs/source/tutorial_metrics.md

diff --git a/docs/source/tutorial_metrics.md b/docs/source/tutorial_metrics.md
deleted file mode 100644
index 650ba4ceb7e..00000000000
--- a/docs/source/tutorial_metrics.md
+++ /dev/null
@@ -1,372 +0,0 @@
-# Understanding and adding new metrics
-
-Author: Stephen Roller
-
-## Introduction and Standard Metrics
-
-ParlAI contains a number of built-in metrics that are automatically computed when
-we train and evaluate models. Some of these metrics are _text generation_ metrics,
-which happen any time we generate a text: this includes F1, BLEU and Accuracy.
-
-For example, let's try a Fixed Response model, which always returns a given fixed
-response, and evaluate on the DailyDialog dataset:
-
-```
-$ parlai eval_model -m fixed_response -t dailydialog --fixed-response "how may i help you ?"
-... after a while ...
-14:41:40 | Evaluating task dailydialog using datatype valid.
-14:41:40 | creating task(s): dailydialog
-14:41:41 | Finished evaluating tasks ['dailydialog'] using datatype valid
-    accuracy  bleu-4  exs    f1
-    .0001239 .002617 8069 .1163
-```
-
-We see that we got 0.01239% accuracy, 0.26% BLEU-4 score, and 11.63% F1 across
-8069 examples. What do those metrics means?
-
-- Accuracy: this is perfect, exact, matching of the response, averaged across
-  all examples in the dataset
-- BLEU-4: this is the [BLEU score](https://en.wikipedia.org/wiki/BLEU) between
-  the predicted response and the reference response. It is measured on
-  tokenized text, and uses NLTK to compute it.
-- F1: This is the [Unigram](https://en.wikipedia.org/wiki/N-gram) F1 overlap
-  between your text and the reference response.
-- exs: the number of examples we have evaluated
-
-If you don't see the BLEU-4 score, you may need to install NLTK with
-`pip install nltk`.
-
-We can also measure ROUGE. Note that we need to `pip install py-rouge` for this
-functionality:
-
-```
-$ parlai eval_model -m fixed_response -t dailydialog --fixed-response "how may i help you ?" --metrics rouge
-14:47:24 | creating task(s): dailydialog
-14:47:31 | Finished evaluating tasks ['dailydialog'] using datatype valid
-    accuracy  exs    f1  rouge_1  rouge_2  rouge_L
-    .0001239 8069 .1163   .09887  .007285   .09525
-```
-
-### Agent-specific metrics
-
-Some agents include their own metrics that are computed for them. For example,
-generative models automatically compute `ppl`
-([perplexity](https://en.wikipedia.org/wiki/Perplexity)) and `token_acc`, both
-which measure the generative model's ability to predict indivdual tokens.  As
-an example, let's evaluate the [BlenderBot](https://parl.ai/projects/recipes/)
-90M model on DailyDialog:
-
-```
-$ parlai eval_model --task dailydialog -mf zoo:blender/blender_90M/model -bs 32
-...
-14:54:14 | Evaluating task dailydialog using datatype valid.
-14:54:14 | creating task(s): dailydialog
-...
-15:26:19 | Finished evaluating tasks ['dailydialog'] using datatype valid
-    accuracy  bleu-4  ctpb  ctps  exps  exs    f1  gpu_mem  loss      lr  ltpb  ltps   ppl  token_acc  total_train_updates   tpb   tps
-           0 .002097 14202 442.5 6.446 8069 .1345    .0384 2.979 7.5e-06  3242   101 19.67      .4133               339012 17445 543.5
-```
-
-Here we see a number of extra metrics, each of which we explain below:
-- `tpb`, `ctpb`, `ltpb`: stand for tokens per batch, context-tokens per batch,
-  and label-tokens per batch. These are useful for measuring how dense the
-  batches are, and are helpful when experimenting with [dynamic
-  batching](tutorial_fast).  tpb is always the sum of ctpb and lptb.
-- `tps`, `ctps`, `ltps`: are similar, but stand for "tokens per second". They
-  measure how fast we are training. Similarly, `exps` measures examples per
-  second.
-- `gpu_mem`: measures _roughly_ how much GPU memory your model is using, but it
-  is only approximate. This is useful for determining if you can possibly increase
-  the model size or the batch size.
-- `loss`: the loss metric
-- `ppl` and `token_acc`: the perplexity and per-token accuracy. these are generative
-  performance metrics.
-- `total_train_updates`: the number of SGD updates this model was trained for.
-  You will see this increase during training, but not during evaluation.
-
-## Adding custom metrics
-
-Of course, you may wish to add your own custom metrics: whether this is because
-you are developing a special model, special dataset, or otherwise want other
-information accessible to you. Metrics can be computed by either _the teacher_ OR
-_the model_. Within the model, they may be computed either _locally_ or _globally_.
-There are different reasons for why and where you would want to choose each
-location:
-
-- __Teacher metrics__: This is the best spot for computing metrics that depend
-  on a specific dataset. These metrics will only be available when evaluating
-  on this dataset. They have the advantage of being easy to compute and
-  understand.
-- __Global metrics__: Global metrics are computed by the model, and are globally
-  tracked. These metrics are easy to understand and track, but work poorly
-  when doing multitasking.
-- __Local metrics__: Local metrics are the model-analogue of teacher metrics.
-  They are computed and recorded on a per-example basis, and so they work well
-  when multitasking. They can be extremely complicated for some models, however.
-
-We will take you through writing each of these methods in turn, and demonstrate
-examples of how to add these metrics in your setup.
-
-## Teacher metrics
-
-Teacher metrics are useful for items that depend on a specific dataset.
-For example, in some of our task oriented datasets, like
-[`google_sgd`](https://github.com/facebookresearch/ParlAI/blob/master/parlai/tasks/google_sgd/agents.py),
-we want to additionally compute metrics around slots.
-
-Teacher metrics can be added by adding the following method to your teacher:
-
-```python
-    def custom_evaluation(
-        self,
-        teacher_action: Message,
-        labels: Optional[Tuple[str]],
-        model_response: Message,
-    ) -> None:
-        pass
-```
-
-The signature for this method is as follows:
-- `teacher_action`: this is the last message the teacher sent to the model. This likely
-   contains a "text" and "labels" field, as well as any custom fields you might
-   have.
-- `labels`: The gold label(s). This can also be found as information in the
-  `teacher_action`, but it is conveniently extracted for you.
-- `model_response`: The full model response, including any extra fields the model
-   may have sent.
-
-Let's take an actual example. We will add a custom metric which calculates
-how often the model says the word "hello", and call it `hello_avg`.
-
-We will add a [custom teacher](tutorial_task). For this example, we will use
-the `@register` syntax you may have seen in our [quickstart
-tutorial](tutorial_quick).
-
-```python
-from parlai.core.loader import register_teacher
-from parlai.core.metrics import AverageMetric
-from parlai.tasks.dailydialog.agents import DefaultTeacher as DailyDialogTeacher
-
-@register_teacher("hello_daily")
-class CustomDailyDialogTeacher(DailyDialogTeacher):
-    def custom_evaluation(
-        self, teacher_action, labels, model_response
-    ) -> None:
-        if 'text' not in model_response:
-            # model didn't speak, skip this example
-            return
-        model_text = model_response['text']
-        if 'hello' in model_text:
-            # count 1 / 1 messages having "hello"
-            self.metrics.add('hello_avg', AverageMetric(1, 1))
-        else:
-            # count 0 / 1 messages having "hello"
-            self.metrics.add('hello_avg', AverageMetric(0, 1))
-
-if __name__ == '__main__':
-    from parlai.scripts.eval_model import EvalModel
-
-    EvalModel.main(
-       task='hello_daily',
-       model_file='zoo:blender/blender_90M/model',
-       batchsize=32,
-    )
-```
-
-If we run the script, we will have a new metric in our output:
-
-```
-18:07:30 | Finished evaluating tasks ['hello_daily'] using datatype valid
-    accuracy  bleu-4  ctpb  ctps  exps  exs    f1  gpu_mem  hello_avg  loss  ltpb  ltps   ppl  token_acc  tpb   tps
-           0 .002035  2172   230 3.351 8069 .1346   .05211      .1228 2.979 495.9 52.52 19.67      .4133 2668 282.6
-```
-
-__What is AverageMetric?__
-
-Wait, what is this
-[AverageMetric](parlai.core.metrics.AverageMetric)? All metrics
-you want to create in ParlAI should be a
-[Metric](Metric) object. Metric objects
-define a way of instantiating the metric, a way of combining it with a
-like-metric, and a way of rendering it as a single float value. For an
-AverageMetric, this means we need to define a numerator and a denominator; the
-combination of AverageMetrics adds their numerators and denominators
-separately. As we do this across all examples, the numerator will be the number
-of examples with "hello" in it, and the denominator will be the total number of
-examples. When we go to print the metric, the division will be computed at the
-last second.
-
-If you're used to writing machine learning code in one-off scripts, you may ask
-why do I need to use this metric? Can't I just count and divide myself? While
-you can do this, your code could not be run in [_distributed
-mode_](tutorial_fast). If we only returned a single float, we would not be able
-to know if some distributed workers received more or fewer examples than
-others. However, when we explicitly store the numerator and denominator, we can
-combine and reduce the across multiple nodes, enabling us to train on hundreds
-of GPUs, while still ensuring correctness in all our metrics.
-
-In addition to AverageMetric, there is also
-[SumMetric](parlai.core.metrics.SumMetric), which keeps a running
-sum. SumMetric and AverageMetric are the most common ways to construct custom
-metrics, but others exist as well. For a full list (and views into advanced
-cases), please see the [metrics API documentation](metrics_api).
-
-## Agent (model) level metrics
-
-In the above example, we worked on a metric defined by a Teacher. However,
-sometimes our models will have special metrics that only they want to compute,
-which we call an Agent-level metric. Perplexity is one example.
-
-To compute model-level metrics, we can define either a global metric, or a
-local metric. Global metrics can be computed anywhere, and are easy to use,
-but cannot distinguish between different teachers when multitasking. We'll
-look at another example, counting the number of times the teacher says "hello".
-
-### Global metrics
-
-A global metric is computed anywhere in the model, and has an
-interface similar to that of the teacher:
-
-```python
-agent.global_metrics.add('my_metric', AverageMetric(1, 2))
-```
-
-Global metrics are called as such because they can be called anywhere in agent
-code. For example, we can add a metric that counts the number of times the
-model sees the word "hello" in `observe`. We'll do this while extending
-the `TransformerGeneratorAgent`, so that we can combined it with the BlenderBot
-model we used earlier.
-
-```python
-from parlai.core.metrics import AverageMetric
-from parlai.core.loader import register_agent
-from parlai.agents.transformer.transformer import TransformerGeneratorAgent
-
-
-@register_agent('GlobalHelloCounter')
-class GlobalHelloCounterAgent(TransformerGeneratorAgent):
-    def observe(self, observation):
-        retval = super().observe(observation)
-        if 'text' in observation:
-            text = observation['text']
-            self.global_metrics.add(
-                'global_hello', AverageMetric(int('hello' in text), 1)
-            )
-        return retval
-
-
-if __name__ == '__main__':
-    from parlai.scripts.eval_model import EvalModel
-
-    EvalModel.main(
-        task='dailydialog',
-        model='GlobalHelloCounter',
-        model_file='zoo:blender/blender_90M/model',
-        batchsize=32,
-    )
-```
-
-Running the script, we see that our new metric appears. Note that it is
-different than the metric in the first half of the tutorial: that is because
-previously we were counting the number of times the model said hello (a lot),
-but now we are counting how often the dataset says hello.
-
-```
-21:57:50 | Finished evaluating tasks ['dailydialog'] using datatype valid
-    accuracy  bleu-4  ctpb  ctps  exps  exs    f1  global_hello  gpu_mem  loss  ltpb  ltps   ppl  token_acc   tpb   tps
-           0 .002097 14202 435.1 6.338 8069 .1345      .0009914   .02795 2.979  3242 99.32 19.67      .4133 17445 534.4
-```
-
-The global metric works well, but have some drawbacks: if we were to start
-training on a multitask datasets, we would not be able to distinguish the
-`global_hello` of the two datasets, and we could only compute the micro-average
-of the combination of the two. Below is an excerpt from a training log with
-the above agents:
-
-```
-09:14:52 | time:112s total_exs:90180 epochs:0.41
-                clip  ctpb  ctps  exps  exs  global_hello  gnorm  gpu_mem  loss  lr  ltpb  ltps   ppl  token_acc  total_train_updates   tpb   tps   ups
-   all             1  9831 66874 841.9 8416        .01081  2.018    .3474 5.078   1  1746 11878 163.9      .2370                  729 11577 78752 6.803
-   convai2                             3434        .01081                 5.288                 197.9      .2120
-   dailydialog                         4982        .01081                 4.868                   130      .2620
-```
-
-Notice how `global_hello` is the same in both, because the model is unable to
-distinguish between the two settings. In the next section we'll show how to fix
-this with local metrics.
-
-__On placement__: In the example above, we recorded the global metric inside
-the `observe` function. However, global metrics can be recorded from anywhere.
-
-### Local metrics
-
-Having observed the limitation of global metrics being unable to distinguish
-settings in multitasking, we would like to improve upon this. Let's add a local
-metric, which is recorded _per example_. By recording this metric per example,
-we can unambiguously identify which metrics came from which dataset, and report
-averages correctly.
-
-Local metrics have a limitation: they can only be computed inside the scope of
-`batch_act`. This includes common places like `compute_loss` or `generate`,
-where we often want to instrument specific behavior.
-
-Let's look at an example. We'll add a metric inside the `batchify` function,
-which is called from within `batch_act`, and is used to convert from a list of
-[Messages](messages) objects to a
-[Batch](torch_agent.html#parlai.core.torch_agent.Batch) object. It is where we do things like
-padding, etc. We'll do something slightly different than our previous runs.
-In this case, we'll count the number of _tokens_ which are the word "hello".
-
-```python
-from parlai.core.metrics import AverageMetric
-from parlai.core.loader import register_agent
-from parlai.agents.transformer.transformer import TransformerGeneratorAgent
-
-
-@register_agent('LocalHelloCounter')
-class LocalHelloCounterAgent(TransformerGeneratorAgent):
-    def batchify(self, observations):
-        batch = super().batchify(observations)
-        if hasattr(batch, 'text_vec'):
-            num_hello = ["hello" in o['text'] for o in observations]
-            self.record_local_metric(
-                'local_hello',
-                # AverageMetric.many(seq1) is shorthand for
-                # [AverageMetric(item) for item in seq)
-                AverageMetric.many(num_hello),
-            )
-        return batch
-
-
-if __name__ == '__main__':
-    from parlai.scripts.train_model import TrainModel
-
-    TrainModel.main(
-        task='dailydialog,convai2',
-        model='LocalHelloCounter',
-        dict_file='zoo:blender/blender_90M/model.dict',
-        batchsize=32,
-    )
-```
-
-When we run this training script, we get one such output:
-```
-09:49:00 | time:101s total_exs:56160 epochs:0.26
-                clip  ctpb  ctps  exps  exs  gnorm  gpu_mem  local_hello  loss  lr  ltpb  ltps   ppl  token_acc  total_train_updates  tpb   tps  ups
-   all             1  3676 63204 550.2 5504  2.146    .1512       .01423 4.623   1 436.2  7500 101.8      .2757                 1755 4112 70704 17.2
-   convai2                             3652                       .02793 4.659                 105.5      .2651
-   dailydialog                         1852                       .00054 4.587                 98.17      .2863
-```
-
-Notice how the `local_hello` metric can now distinguish between hellos coming from
-convai2 and those coming from daily dialog? The average hides the fact that one
-dataset has many hellos, and the other does not.
-
-Local metrics are primarily worth the implementation when you care about the
-fidelity of _train time metrics_. During evaluation time, we evaluate each
-dataset individually, so we can ensure global metrics are not mixed up.
-
-__Under the hood__: Local metrics work by including a "metrics" field in the
-return message. This is a dictionary which maps field name to a metric value.
-When the teacher receives the response from the model, it utilizes the metrics
-field to update counters on its side.

From 37f5a96fb649f5c2336dbea7286c8c9408403b86 Mon Sep 17 00:00:00 2001
From: Stephen Roller <roller@fb.com>
Date: Wed, 12 Aug 2020 19:40:23 -0400
Subject: [PATCH 07/22] Fix the table.

---
 docs/source/_static/css/parlai_theme.css |  4 ++++
 docs/source/index.rst                    |  1 -
 docs/source/tutorial_fast.md             | 18 ++++++++----------
 3 files changed, 12 insertions(+), 11 deletions(-)

diff --git a/docs/source/_static/css/parlai_theme.css b/docs/source/_static/css/parlai_theme.css
index f526b97b56d..f3d2d5dec2e 100644
--- a/docs/source/_static/css/parlai_theme.css
+++ b/docs/source/_static/css/parlai_theme.css
@@ -30,6 +30,10 @@ th > p {
     margin-bottom: 0px !important;
 }
 
+.text-align\:right {
+    text-align: right;
+}
+
 @media screen and (min-width: 1400px) {
     .wy-nav-content-wrap {
         background-color: rgba(0, 0, 0, 0.0470588);
diff --git a/docs/source/index.rst b/docs/source/index.rst
index 9021eaea339..facc8f5475f 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -20,7 +20,6 @@ ParlAI is a one-stop-shop for dialog research.
    tutorial_worlds
    tutorial_torch_generator_agent
    tutorial_torch_ranker_agent
-   tutorial_metrics
    tutorial_fast
    tutorial_tipsntricks
    tutorial_mturk
diff --git a/docs/source/tutorial_fast.md b/docs/source/tutorial_fast.md
index e8b3ef7e9c6..bc260cb28ff 100644
--- a/docs/source/tutorial_fast.md
+++ b/docs/source/tutorial_fast.md
@@ -8,16 +8,14 @@ used with others.
 
 A summary of the speedups is in this table:
 
-    +-------------------------+-------+------+-------+---------+
-    | Method                  | Train | Eval | Total | Speedup |
-    +-------------------------+-------+------+-------+---------+
-    | Baseline                |  504s |  48s |  552s |    1.0x |
-    | Skip generation         |  504s |  16s |  520s |    1.1x |
-    | Dynamic batching        |  254s |  11s |  265s |    2.1x |
-    | FP16                    |  197s |   8s |  205s |    2.7x |
-    | Larger batchsize (FP16) |  151s |   7s |  158s |    3.5x |
-    | Using 4 GPUs            |   47s |   3s |   50s |   11.0x |
-    +-------------------------+-------+------+-------+---------+
+| Method                  | Train | Eval | Total | Speedup |
+| ----------------------- | ----: | ---: | ----: | ------: |
+| Baseline                |  504s |  48s |  552s |    1.0x |
+| Skip generation         |  504s |  16s |  520s |    1.1x |
+| Dynamic batching        |  254s |  11s |  265s |    2.1x |
+| FP16                    |  197s |   8s |  205s |    2.7x |
+| Larger batchsize (FP16) |  151s |   7s |  158s |    3.5x |
+| Using 4 GPUs            |   47s |   3s |   50s |   11.0x |
 
 ## Setting a baseline
 

From b2d4ebb672218b8127c4083f2e96480b3be961a3 Mon Sep 17 00:00:00 2001
From: Stephen Roller <roller@fb.com>
Date: Wed, 12 Aug 2020 21:33:12 -0400
Subject: [PATCH 08/22] Muck with css, generated tables.

---
 docs/source/_static/css/parlai_theme.css |  18 +++-
 docs/source/cli_usage.rst                |  35 +++++-
 docs/source/generate_agent_list.py       |   5 +-
 docs/source/generate_cli.py              | 131 +++++++++++++++--------
 docs/source/index.rst                    |  19 ++--
 5 files changed, 149 insertions(+), 59 deletions(-)

diff --git a/docs/source/_static/css/parlai_theme.css b/docs/source/_static/css/parlai_theme.css
index f3d2d5dec2e..173194b797c 100644
--- a/docs/source/_static/css/parlai_theme.css
+++ b/docs/source/_static/css/parlai_theme.css
@@ -13,6 +13,11 @@ h1, h2, .rst-content .toctree-wrapper p.caption, h3, h4, h5, h6, legend, p.capti
     font-family: "Lato","proxima-nova","Helvetica Neue",Arial,sans-serif;
 }
 
+p { margin-bottom: 16px; }
+h1 { size: 250%; }
+h2 { size: 200%; }
+h3 { size: 150%; }
+
 .wy-menu.wy-menu-vertical p.caption {
     color: #fcfcfc;
 }
@@ -26,8 +31,8 @@ h1, h2, .rst-content .toctree-wrapper p.caption, h3, h4, h5, h6, legend, p.capti
     background-color: #fff;
 }
 
-th > p {
-    margin-bottom: 0px !important;
+th p, td div.line-block {
+    margin-bottom: 0px;
 }
 
 .text-align\:right {
@@ -149,3 +154,12 @@ nav .hidden-section {
 .wy-table-responsive table {
     width: 100%;
 }
+
+.section {
+}
+.section .section {
+    margin-bottom: 5em;
+}
+.section .section .section {
+    margin-bottom: 2em;
+}
diff --git a/docs/source/cli_usage.rst b/docs/source/cli_usage.rst
index f634ac6c8e0..dce6b21288d 100644
--- a/docs/source/cli_usage.rst
+++ b/docs/source/cli_usage.rst
@@ -1,7 +1,38 @@
-Command Line Usage
+Main scripts
 ==================
 
 This contains the command line usage for each of the standard scripts we
-release. These are each included in ``parlai/scripts``.
+release. These are each included in ``parlai/scripts``, and all may
+be invoked with the "parlai" supercommand.
+
+The parlai supercommand may be invoked from the command line by running
+``parlai`` after installing ParlAI. Its default output looks like this:
+
+.. code-block::txt
+
+  usage: parlai [-h] [--helpall] COMMAND ...
+
+         _
+        /")
+       //)
+    ==//'=== ParlAI
+     /
+
+  optional arguments:
+    -h, --help               show this help message and exit
+    --helpall                show all commands, including advanced ones.
+
+  Commands:
+
+    display_data (dd)        Display data from a task
+    display_model (dm)       Display model predictions.
+    eval_model (em, eval)    Evaluate a model
+    train_model (tm, train)  Train a model
+    interactive (i)          Interactive chat with a model on the command line
+    safe_interactive         Like interactive, but adds a safety filter
+    self_chat                Generate self-chats of a model
+
+The remainder of this page describes each of the commands, their possible arguments,
+and some examples of their usage.
 
 .. include:: cli_usage.inc
diff --git a/docs/source/generate_agent_list.py b/docs/source/generate_agent_list.py
index 615fccd3670..8b0ef373c05 100644
--- a/docs/source/generate_agent_list.py
+++ b/docs/source/generate_agent_list.py
@@ -34,9 +34,10 @@ def _make_argparse_table(class_):
             if hasattr(action, 'hidden') and action.hidden:
                 # some options are marked hidden
                 continue
+            if action.dest == argparse.SUPPRESS or action.dest == 'help':
+                continue
             action_strings = ",  ".join(f'`{a}`' for a in action.option_strings)
             description = []
-            # start with the help message
             if action.help:
                 h = action.help
                 if not h[0].isupper():
@@ -67,7 +68,7 @@ def _make_argparse_table(class_):
         if not actions:
             continue
 
-        readme.append(f'_{ag.title}_\n\n')
+        readme.append(f'__{ag.title}__\n\n')
         readme.append("| Argument | Description |\n")
         readme.append("|----------|----------|\n")
         for row in actions:
diff --git a/docs/source/generate_cli.py b/docs/source/generate_cli.py
index 01bdc35eb3b..16af59c33e5 100755
--- a/docs/source/generate_cli.py
+++ b/docs/source/generate_cli.py
@@ -4,59 +4,102 @@
 # This source code is licensed under the MIT license found in the
 # LICENSE file in the root directory of this source tree.
 
-import os
-import parlai.scripts
-import io
-import importlib
+import argparse
+import parlai.core.script as pcs
 
+def render_script(fout, key, registration):
+    script_parser = registration.klass.setup_args()
+    description = script_parser.description
+    underline = "#" * (5 + len(key))
+    fout.write(f"{key}\n{underline}\n\n")
+    if description:
+        fout.write(f"**Short description:** {description}\n\n")
+    if registration.aliases:
+        aliases = ", ".join(f'``{a}``' for a in registration.aliases)
+        fout.write(f"**Aliases:** {aliases}\n")
 
-def indent(rawstr, indentstr='    '):
-    lines = rawstr.split('\n')
-    indented = [(indentstr + l) for l in lines]
-    return '\n'.join(indented).replace('\\', '\\\\').rstrip()
+    fout.write(f'\n.. automodule:: {registration.klass.__module__}\n\n')
 
+    actions = []
+    for action in script_parser._actions:
+        if hasattr(action, 'hidden') and action.hidden:
+            # some options are marked hidden
+            continue
+        action_strings = ",  ".join(f'``{a}``' for a in action.option_strings)
+        description = []
+        if action.dest == argparse.SUPPRESS:
+            continue
+        # list choices if there are any
+        if action.choices:
+            description += [
+                "Choices: " + ", ".join(f'``{c}``' for c in action.choices) + "."
+            ]
+        # list default and recommended values.
+        if hasattr(action, 'hidden') and action.hidden:
+            continue
+        default_value = ""
+        if action.default and action.default != argparse.SUPPRESS:
+            default_value += f"Default: ``{action.default}``.  "
+        if hasattr(action, 'recommended') and action.recommended:
+            default_value += f"Recommended: ``{action.recommended}``. "
+
+        # special escape for a few args which use a literal newline as their default
+        if default_value:
+            default_value = default_value.replace("\n", "\\n")
+            description.append(default_value)
+
+        # escape for the fact that we're inserting this inside a table
+        if len(description) > 1:
+            description = [' | ' + d for d in description]
+        actions.append((action_strings, description))
+
+    if not actions:
+        return
+
+    action_width = max(max(len(a) for a, d in actions), len("Argument"))
+    desc_width = len("Description")
+    for _, desc in actions:
+        for line in desc:
+            width = len(line)
+            if width > desc_width:
+                desc_width = width
+
+    fout.write("CLI Arguments\n")
+    fout.write("-------------\n")
+    fout.write("+" + "-" * action_width + "+" + "-" * desc_width + "+\n")
+    fstr = f'|{{:{action_width}s}}|{{:{desc_width}}}|\n'
+    fout.write(fstr.format("Argument", "Description"))
+    fout.write("+" + "=" * action_width + "+" + "=" * desc_width + "+\n")
+    for action, description in actions:
+        for i, line in enumerate(description):
+            aval = action if i == 0 else ""
+            fout.write(fstr.format(aval, line))
+        fout.write("+" + "-" * action_width + "+" + "-" * desc_width + "+\n")
+    fout.write('\n\n')
 
-def get_scripts():
-    pathname = os.path.dirname(parlai.scripts.__file__)
-    for fn in os.listdir(pathname):
-        if fn.endswith('.py') and not fn.startswith('__'):
-            yield os.path.join(pathname, fn)
 
 
 def main():
     fout = open('cli_usage.inc', 'w')
+    pcs.setup_script_registry()
 
-    for script_path in get_scripts():
-        script_name = os.path.basename(script_path).replace(".py", "")
-        try:
-            module = importlib.import_module("parlai.scripts." + script_name)
-        except ModuleNotFoundError:
-            continue
-        if not hasattr(module, 'setup_args'):
-            continue
-        # header
-        fout.write(script_name)
-        fout.write('\n')
-        fout.write('-' * len(script_name))
-        fout.write('\n')
-
-        # docs from the module
-        fout.write('.. automodule:: parlai.scripts.{}\n'.format(script_name))
-
-        # fout.write('   :members:\n')
-        # fout.write('   :exclude-members: setup_args\n')
-        fout.write('\n')
-        fout.write('CLI help\n')
-        fout.write('~~~~~~~~\n\n\n')
-
-        # output the --help
-        fout.write('.. code-block:: text\n\n')  # literal block
-        capture = io.StringIO()
-        parser = module.setup_args()
-        parser.prog = 'parlai {}'.format(script_name)
-        parser.print_help(capture)
-        fout.write(indent(capture.getvalue()))
-        fout.write('\n\n')
+    first = []
+    second = []
+    for key, registration in sorted(pcs.SCRIPT_REGISTRY.items()):
+        if not registration.hidden:
+            first.append((key, registration))
+        else:
+            second.append((key, registration))
+
+    for key, registration in first:
+        render_script(fout, key, registration)
+
+    fout.close()
+
+    fout = open('cli_advanced.inc', 'w')
+
+    for key, registration in second:
+        render_script(fout, key, registration)
 
 
 if __name__ == '__main__':
diff --git a/docs/source/index.rst b/docs/source/index.rst
index facc8f5475f..2747b3c1afb 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -12,7 +12,7 @@ ParlAI is a one-stop-shop for dialog research.
 
 .. toctree::
    :maxdepth: 1
-   :caption: Tutorials
+   :caption: Tutorials & Explanations
 
    tutorial_quick
    tutorial_basic
@@ -21,9 +21,9 @@ ParlAI is a one-stop-shop for dialog research.
    tutorial_torch_generator_agent
    tutorial_torch_ranker_agent
    tutorial_fast
-   tutorial_tipsntricks
    tutorial_mturk
    tutorial_chat_service
+   tutorial_tipsntricks
 
 .. toctree::
    :maxdepth: 1
@@ -40,7 +40,14 @@ ParlAI is a one-stop-shop for dialog research.
   agents_list
 
 .. toctree::
-   :maxdepth: 1
+   :maxdepth: 2
+   :caption: Scripts & CLI
+
+   cli_usage
+   cli_advanced
+
+.. toctree::
+   :maxdepth: 2
    :caption: API Reference
 
    agents
@@ -57,12 +64,6 @@ ParlAI is a one-stop-shop for dialog research.
    utils
 
 
-.. toctree::
-   :maxdepth: 2
-   :caption: Scripts
-
-   cli_usage
-
 
 Indices and tables
 ==================

From 02cfe9931583af5f47447a61740e7fc22bd6b828 Mon Sep 17 00:00:00 2001
From: Stephen Roller <roller@fb.com>
Date: Wed, 12 Aug 2020 21:33:32 -0400
Subject: [PATCH 09/22] Special docs.

---
 docs/source/cli_advanced.inc | 652 +++++++++++++++++++++++++++++++++++
 docs/source/cli_advanced.rst |   6 +
 2 files changed, 658 insertions(+)
 create mode 100644 docs/source/cli_advanced.inc
 create mode 100644 docs/source/cli_advanced.rst

diff --git a/docs/source/cli_advanced.inc b/docs/source/cli_advanced.inc
new file mode 100644
index 00000000000..bef2663f092
--- /dev/null
+++ b/docs/source/cli_advanced.inc
@@ -0,0 +1,652 @@
+build_candidates
+#####################
+
+**Short description:** Build the candidate responses for a retrieval model
+
+
+.. automodule:: parlai.scripts.build_candidates
+
+CLI Arguments
+-------------
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|Argument                          |Description                                                                                                                                                                                                                                                                                                             |
++==================================+========================================================================================================================================================================================================================================================================================================================+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-dt``,  ``--datatype``          | | Choices: ``train``, ``train:stream``, ``train:ordered``, ``train:ordered:stream``, ``train:stream:ordered``, ``train:evalmode``, ``train:evalmode:stream``, ``train:evalmode:ordered``, ``train:evalmode:ordered:stream``, ``train:evalmode:stream:ordered``, ``valid``, ``valid:stream``, ``test``, ``test:stream``.|
+|                                  | | Default: ``train:evalmode``.                                                                                                                                                                                                                                                                                         |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-nt``,  ``--numthreads``        |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-bs``,  ``--batchsize``         |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-dynb``,  ``--dynamic-batching``|Choices: ``None``, ``batchsort``, ``full``.                                                                                                                                                                                                                                                                             |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-n``,  ``--num-examples``       |Default: ``-1``.                                                                                                                                                                                                                                                                                                        |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-ltim``,  ``--log-every-n-secs``|Default: ``2``.                                                                                                                                                                                                                                                                                                         |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+
+build_dict
+###############
+
+**Short description:** Build a dictionary.
+
+
+.. automodule:: parlai.scripts.build_dict
+
+CLI Arguments
+-------------
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|Argument                          |Description                                                                                                                                                                                                                                                                                                             |
++==================================+========================================================================================================================================================================================================================================================================================================================+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-dt``,  ``--datatype``          | | Choices: ``train``, ``train:stream``, ``train:ordered``, ``train:ordered:stream``, ``train:stream:ordered``, ``train:evalmode``, ``train:evalmode:stream``, ``train:evalmode:ordered``, ``train:evalmode:ordered:stream``, ``train:evalmode:stream:ordered``, ``valid``, ``valid:stream``, ``test``, ``test:stream``.|
+|                                  | | Default: ``train``.                                                                                                                                                                                                                                                                                                  |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-nt``,  ``--numthreads``        |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-bs``,  ``--batchsize``         |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-dynb``,  ``--dynamic-batching``|Choices: ``None``, ``batchsort``, ``full``.                                                                                                                                                                                                                                                                             |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``--dict-maxexs``                 |Default: ``-1``.                                                                                                                                                                                                                                                                                                        |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-ltim``,  ``--log-every-n-secs``|Default: ``10``.                                                                                                                                                                                                                                                                                                        |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+
+convert_to_fasttext
+########################
+
+**Short description:** Convert data for ingestion in fastText
+
+
+.. automodule:: parlai.scripts.convert_data_to_fasttext_format
+
+CLI Arguments
+-------------
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|Argument                          |Description                                                                                                                                                                                                                                                                                                             |
++==================================+========================================================================================================================================================================================================================================================================================================================+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-dt``,  ``--datatype``          | | Choices: ``train``, ``train:stream``, ``train:ordered``, ``train:ordered:stream``, ``train:stream:ordered``, ``train:evalmode``, ``train:evalmode:stream``, ``train:evalmode:ordered``, ``train:evalmode:ordered:stream``, ``train:evalmode:stream:ordered``, ``valid``, ``valid:stream``, ``test``, ``test:stream``.|
+|                                  | | Default: ``train:ordered``.                                                                                                                                                                                                                                                                                          |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-nt``,  ``--numthreads``        |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-bs``,  ``--batchsize``         |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-dynb``,  ``--dynamic-batching``|Choices: ``None``, ``batchsort``, ``full``.                                                                                                                                                                                                                                                                             |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-n``,  ``--num-examples``       |Default: ``-1``.                                                                                                                                                                                                                                                                                                        |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-ltim``,  ``--log-every-n-secs``|Default: ``2``.                                                                                                                                                                                                                                                                                                         |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+
+convert_to_parlai
+######################
+
+**Short description:** Dump a task to a standardized format
+
+
+.. automodule:: parlai.scripts.convert_data_to_parlai_format
+
+CLI Arguments
+-------------
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|Argument                          |Description                                                                                                                                                                                                                                                                                                             |
++==================================+========================================================================================================================================================================================================================================================================================================================+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-dt``,  ``--datatype``          | | Choices: ``train``, ``train:stream``, ``train:ordered``, ``train:ordered:stream``, ``train:stream:ordered``, ``train:evalmode``, ``train:evalmode:stream``, ``train:evalmode:ordered``, ``train:evalmode:ordered:stream``, ``train:evalmode:stream:ordered``, ``valid``, ``valid:stream``, ``test``, ``test:stream``.|
+|                                  | | Default: ``train:stream``.                                                                                                                                                                                                                                                                                           |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-nt``,  ``--numthreads``        |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-bs``,  ``--batchsize``         |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-dynb``,  ``--dynamic-batching``|Choices: ``None``, ``batchsort``, ``full``.                                                                                                                                                                                                                                                                             |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-n``,  ``--num-examples``       |Default: ``-1``.                                                                                                                                                                                                                                                                                                        |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-if``,  ``--ignore-fields``     |Default: ``id``.                                                                                                                                                                                                                                                                                                        |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-ltim``,  ``--log-every-n-secs``|Default: ``2``.                                                                                                                                                                                                                                                                                                         |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+
+convo_render
+#################
+
+**Short description:** Render data as HTML
+
+
+.. automodule:: parlai.scripts.convo_render
+
+CLI Arguments
+-------------
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|Argument                          |Description                                                                                                                                                                                                                                                                                                             |
++==================================+========================================================================================================================================================================================================================================================================================================================+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-dt``,  ``--datatype``          | | Choices: ``train``, ``train:stream``, ``train:ordered``, ``train:ordered:stream``, ``train:stream:ordered``, ``train:evalmode``, ``train:evalmode:stream``, ``train:evalmode:ordered``, ``train:evalmode:ordered:stream``, ``train:evalmode:stream:ordered``, ``valid``, ``valid:stream``, ``test``, ``test:stream``.|
+|                                  | | Default: ``train``.                                                                                                                                                                                                                                                                                                  |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-nt``,  ``--numthreads``        |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-bs``,  ``--batchsize``         |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-dynb``,  ``--dynamic-batching``|Choices: ``None``, ``batchsort``, ``full``.                                                                                                                                                                                                                                                                             |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``--width``,  ``-wd``             |Default: ``8``.                                                                                                                                                                                                                                                                                                         |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``--height``,  ``-ht``            |Default: ``10``.                                                                                                                                                                                                                                                                                                        |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``--user-icon``,  ``-uic``        |Default: ``https://emojipedia-us.s3.dualstack.us-west-1.amazonaws.com/thumbs/160/apple/76/woman_1f469.png``.                                                                                                                                                                                                            |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``--alt-icon``,  ``-aic``         |Default: ``https://emojipedia-us.s3.dualstack.us-west-1.amazonaws.com/thumbs/160/facebook/230/parrot_1f99c.png``.                                                                                                                                                                                                       |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``--num-examples``,  ``-ne``      |Default: ``10``.                                                                                                                                                                                                                                                                                                        |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+
+data_stats
+###############
+
+**Short description:** Compute data statistics
+
+
+.. automodule:: parlai.scripts.data_stats
+
+CLI Arguments
+-------------
++-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|Argument                             |Description                                                                                                                                                                                                                                                                                                             |
++=====================================+========================================================================================================================================================================================================================================================================================================================+
++-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-dt``,  ``--datatype``             | | Choices: ``train``, ``train:stream``, ``train:ordered``, ``train:ordered:stream``, ``train:stream:ordered``, ``train:evalmode``, ``train:evalmode:stream``, ``train:evalmode:ordered``, ``train:evalmode:ordered:stream``, ``train:evalmode:stream:ordered``, ``valid``, ``valid:stream``, ``test``, ``test:stream``.|
+|                                     | | Default: ``train:ordered``.                                                                                                                                                                                                                                                                                          |
++-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-nt``,  ``--numthreads``           |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
++-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-bs``,  ``--batchsize``            |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
++-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-dynb``,  ``--dynamic-batching``   |Choices: ``None``, ``batchsort``, ``full``.                                                                                                                                                                                                                                                                             |
++-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-n``,  ``-ne``,  ``--num-examples``|Default: ``-1``.                                                                                                                                                                                                                                                                                                        |
++-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-ltim``,  ``--log-every-n-secs``   |Default: ``2``.                                                                                                                                                                                                                                                                                                         |
++-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``--agent``                          |Choices: ``0``, ``1``.                                                                                                                                                                                                                                                                                                  |
++-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+
+detect_offensive
+#####################
+
+**Short description:** Check task for offensive language
+
+
+.. automodule:: parlai.scripts.detect_offensive_language
+
+CLI Arguments
+-------------
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|Argument                          |Description                                                                                                                                                                                                                                                                                                             |
++==================================+========================================================================================================================================================================================================================================================================================================================+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-dt``,  ``--datatype``          | | Choices: ``train``, ``train:stream``, ``train:ordered``, ``train:ordered:stream``, ``train:stream:ordered``, ``train:evalmode``, ``train:evalmode:stream``, ``train:evalmode:ordered``, ``train:evalmode:ordered:stream``, ``train:evalmode:stream:ordered``, ``valid``, ``valid:stream``, ``test``, ``test:stream``.|
+|                                  | | Default: ``train:ordered``.                                                                                                                                                                                                                                                                                          |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-nt``,  ``--numthreads``        |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-bs``,  ``--batchsize``         |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-dynb``,  ``--dynamic-batching``|Choices: ``None``, ``batchsort``, ``full``.                                                                                                                                                                                                                                                                             |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-m``,  ``--model``              |Default: ``repeat_query``.                                                                                                                                                                                                                                                                                              |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-ltim``,  ``--log-every-n-secs``|Default: ``2``.                                                                                                                                                                                                                                                                                                         |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``--safety``                      | | Choices: ``all``, ``classifier``, ``string_matcher``.                                                                                                                                                                                                                                                                |
+|                                  | | Default: ``all``.                                                                                                                                                                                                                                                                                                    |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+
+eval_wordstat
+##################
+
+**Short description:** Compute statistics from model predictions
+
+
+.. automodule:: parlai.scripts.eval_wordstat
+
+CLI Arguments
+-------------
++--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|Argument                              |Description                                                                                                                                                                                                                                                                                                             |
++======================================+========================================================================================================================================================================================================================================================================================================================+
++--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-dt``,  ``--datatype``              | | Choices: ``train``, ``train:stream``, ``train:ordered``, ``train:ordered:stream``, ``train:stream:ordered``, ``train:evalmode``, ``train:evalmode:stream``, ``train:evalmode:ordered``, ``train:evalmode:ordered:stream``, ``train:evalmode:stream:ordered``, ``valid``, ``valid:stream``, ``test``, ``test:stream``.|
+|                                      | | Default: ``valid``.                                                                                                                                                                                                                                                                                                  |
++--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-nt``,  ``--numthreads``            |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
++--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-bs``,  ``--batchsize``             |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
++--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-dynb``,  ``--dynamic-batching``    |Choices: ``None``, ``batchsort``, ``full``.                                                                                                                                                                                                                                                                             |
++--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-ne``,  ``--num-examples``          |Default: ``-1``.                                                                                                                                                                                                                                                                                                        |
++--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-ltim``,  ``--log-every-n-secs``    |Default: ``2``.                                                                                                                                                                                                                                                                                                         |
++--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-fb``,  ``--freq-bins``             |Default: ``0,100,1000,10000``.                                                                                                                                                                                                                                                                                          |
++--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-cun``,  ``--compute-unique``       |Default: ``True``.                                                                                                                                                                                                                                                                                                      |
++--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+
+extract_image_feature
+##########################
+
+**Short description:** Load/extract image features
+
+
+.. automodule:: parlai.scripts.extract_image_feature
+
+CLI Arguments
+-------------
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|Argument                          |Description                                                                                                                                                                                                                                                                                                             |
++==================================+========================================================================================================================================================================================================================================================================================================================+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-dt``,  ``--datatype``          | | Choices: ``train``, ``train:stream``, ``train:ordered``, ``train:ordered:stream``, ``train:stream:ordered``, ``train:evalmode``, ``train:evalmode:stream``, ``train:evalmode:ordered``, ``train:evalmode:ordered:stream``, ``train:evalmode:stream:ordered``, ``valid``, ``valid:stream``, ``test``, ``test:stream``.|
+|                                  | | Default: ``train``.                                                                                                                                                                                                                                                                                                  |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-nt``,  ``--numthreads``        |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-bs``,  ``--batchsize``         |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-dynb``,  ``--dynamic-batching``|Choices: ``None``, ``batchsort``, ``full``.                                                                                                                                                                                                                                                                             |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+
+interactive_web
+####################
+
+**Short description:** Interactive chat with a model in a web browser
+
+**Aliases:** ``iweb``
+
+.. automodule:: parlai.scripts.interactive_web
+
+CLI Arguments
+-------------
++---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|Argument                                           |Description                                                                                                                                                                                                                                                                                                             |
++===================================================+========================================================================================================================================================================================================================================================================================================================+
++---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-t``,  ``--task``                                |Default: ``interactive``.                                                                                                                                                                                                                                                                                               |
++---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-dt``,  ``--datatype``                           | | Choices: ``train``, ``train:stream``, ``train:ordered``, ``train:ordered:stream``, ``train:stream:ordered``, ``train:evalmode``, ``train:evalmode:stream``, ``train:evalmode:ordered``, ``train:evalmode:ordered:stream``, ``train:evalmode:stream:ordered``, ``valid``, ``valid:stream``, ``test``, ``test:stream``.|
+|                                                   | | Default: ``train``.                                                                                                                                                                                                                                                                                                  |
++---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-nt``,  ``--numthreads``                         |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
++---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-bs``,  ``--batchsize``                          |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
++---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-dynb``,  ``--dynamic-batching``                 |Choices: ``None``, ``batchsort``, ``full``.                                                                                                                                                                                                                                                                             |
++---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``--display-ignore-fields``                        |Default: ``label_candidates,text_candidates``.                                                                                                                                                                                                                                                                          |
++---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-it``,  ``--interactive-task``                   |Default: ``True``.                                                                                                                                                                                                                                                                                                      |
++---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``--save-format``                                  | | Choices: ``conversations``, ``parlai``.                                                                                                                                                                                                                                                                              |
+|                                                   | | Default: ``conversations``.                                                                                                                                                                                                                                                                                          |
++---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``--log-keep-fields``                              |Default: ``all``.                                                                                                                                                                                                                                                                                                       |
++---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``--port``                                         |Default: ``8080``.                                                                                                                                                                                                                                                                                                      |
++---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``--host``                                         |Default: ``localhost``.                                                                                                                                                                                                                                                                                                 |
++---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+
+multiprocessing_eval
+#########################
+
+**Short description:** Evaluate a model
+
+**Aliases:** ``mp_eval``
+
+.. automodule:: parlai.scripts.multiprocessing_eval
+
+CLI Arguments
+-------------
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|Argument                          |Description                                                                                                                                                                                                                                                                                                             |
++==================================+========================================================================================================================================================================================================================================================================================================================+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-dt``,  ``--datatype``          | | Choices: ``train``, ``train:stream``, ``train:ordered``, ``train:ordered:stream``, ``train:stream:ordered``, ``train:evalmode``, ``train:evalmode:stream``, ``train:evalmode:ordered``, ``train:evalmode:ordered:stream``, ``train:evalmode:stream:ordered``, ``valid``, ``valid:stream``, ``test``, ``test:stream``.|
+|                                  | | Default: ``valid``.                                                                                                                                                                                                                                                                                                  |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-nt``,  ``--numthreads``        |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-bs``,  ``--batchsize``         |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-dynb``,  ``--dynamic-batching``|Choices: ``None``, ``batchsort``, ``full``.                                                                                                                                                                                                                                                                             |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``--save-format``                 | | Choices: ``conversations``, ``parlai``.                                                                                                                                                                                                                                                                              |
+|                                  | | Default: ``conversations``.                                                                                                                                                                                                                                                                                          |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-ne``,  ``--num-examples``      |Default: ``-1``.                                                                                                                                                                                                                                                                                                        |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-ltim``,  ``--log-every-n-secs``|Default: ``10``.                                                                                                                                                                                                                                                                                                        |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-mcs``,  ``--metrics``          |Default: ``default``.                                                                                                                                                                                                                                                                                                   |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``--log-keep-fields``             |Default: ``all``.                                                                                                                                                                                                                                                                                                       |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``--distributed-world-size``      |Default: ``2``.                                                                                                                                                                                                                                                                                                         |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+
+multiprocessing_train
+##########################
+
+**Short description:** Train a model
+
+**Aliases:** ``mp_train``
+
+.. automodule:: parlai.scripts.multiprocessing_train
+
+CLI Arguments
+-------------
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|Argument                                   |Description                                                                                                                                                                                                                                                                                                             |
++===========================================+========================================================================================================================================================================================================================================================================================================================+
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-dt``,  ``--datatype``                   | | Choices: ``train``, ``train:stream``, ``train:ordered``, ``train:ordered:stream``, ``train:stream:ordered``, ``train:evalmode``, ``train:evalmode:stream``, ``train:evalmode:ordered``, ``train:evalmode:ordered:stream``, ``train:evalmode:stream:ordered``, ``valid``, ``valid:stream``, ``test``, ``test:stream``.|
+|                                           | | Default: ``train``.                                                                                                                                                                                                                                                                                                  |
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-nt``,  ``--numthreads``                 |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-bs``,  ``--batchsize``                  |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-dynb``,  ``--dynamic-batching``         |Choices: ``None``, ``batchsort``, ``full``.                                                                                                                                                                                                                                                                             |
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-eps``,  ``--num-epochs``                |Default: ``-1``.                                                                                                                                                                                                                                                                                                        |
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-ttim``,  ``--max-train-time``           |Default: ``-1``.                                                                                                                                                                                                                                                                                                        |
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-vtim``,  ``--validation-every-n-secs``  |Default: ``-1``.                                                                                                                                                                                                                                                                                                        |
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-stim``,  ``--save-every-n-secs``        |Default: ``-1``.                                                                                                                                                                                                                                                                                                        |
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-veps``,  ``--validation-every-n-epochs``|Default: ``-1``.                                                                                                                                                                                                                                                                                                        |
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-vp``,  ``--validation-patience``        |Default: ``10``.                                                                                                                                                                                                                                                                                                        |
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-vmt``,  ``--validation-metric``         |Default: ``accuracy``.                                                                                                                                                                                                                                                                                                  |
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-vmm``,  ``--validation-metric-mode``    |Choices: ``max``, ``min``.                                                                                                                                                                                                                                                                                              |
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-mcs``,  ``--metrics``                   |Default: ``default``.                                                                                                                                                                                                                                                                                                   |
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``--distributed-world-size``               |Default: ``2``.                                                                                                                                                                                                                                                                                                         |
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+
+party
+##########
+
+**Short description:** Throw a party!
+
+**Aliases:** ``parrot``
+
+.. automodule:: parlai.scripts.party
+
+CLI Arguments
+-------------
++-------------------+-----------+
+|Argument           |Description|
++===================+===========+
++-------------------+-----------+
+
+
+profile_interactive
+########################
+
+**Short description:** Interactive chat with a model
+
+
+.. automodule:: parlai.scripts.profile_interactive
+
+CLI Arguments
+-------------
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|Argument                          |Description                                                                                                                                                                                                                                                                                                             |
++==================================+========================================================================================================================================================================================================================================================================================================================+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-t``,  ``--task``               |Default: ``interactive``.                                                                                                                                                                                                                                                                                               |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-dt``,  ``--datatype``          | | Choices: ``train``, ``train:stream``, ``train:ordered``, ``train:ordered:stream``, ``train:stream:ordered``, ``train:evalmode``, ``train:evalmode:stream``, ``train:evalmode:ordered``, ``train:evalmode:ordered:stream``, ``train:evalmode:stream:ordered``, ``valid``, ``valid:stream``, ``test``, ``test:stream``.|
+|                                  | | Default: ``train``.                                                                                                                                                                                                                                                                                                  |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-nt``,  ``--numthreads``        |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-bs``,  ``--batchsize``         |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-dynb``,  ``--dynamic-batching``|Choices: ``None``, ``batchsort``, ``full``.                                                                                                                                                                                                                                                                             |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-d``,  ``--display-examples``   |Default: ``True``.                                                                                                                                                                                                                                                                                                      |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-ne``,  ``--num-examples``      |Default: ``5``.                                                                                                                                                                                                                                                                                                         |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``--display-ignore-fields``       |Default: ``label_candidates,text_candidates``.                                                                                                                                                                                                                                                                          |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-it``,  ``--interactive-task``  |Default: ``True``.                                                                                                                                                                                                                                                                                                      |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+
+profile_train
+##################
+
+**Short description:** cProfile a training run
+
+
+.. automodule:: parlai.scripts.profile_train
+
+CLI Arguments
+-------------
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|Argument                                   |Description                                                                                                                                                                                                                                                                                                             |
++===========================================+========================================================================================================================================================================================================================================================================================================================+
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-dt``,  ``--datatype``                   | | Choices: ``train``, ``train:stream``, ``train:ordered``, ``train:ordered:stream``, ``train:stream:ordered``, ``train:evalmode``, ``train:evalmode:stream``, ``train:evalmode:ordered``, ``train:evalmode:ordered:stream``, ``train:evalmode:stream:ordered``, ``valid``, ``valid:stream``, ``test``, ``test:stream``.|
+|                                           | | Default: ``train``.                                                                                                                                                                                                                                                                                                  |
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-nt``,  ``--numthreads``                 |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-bs``,  ``--batchsize``                  |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-dynb``,  ``--dynamic-batching``         |Choices: ``None``, ``batchsort``, ``full``.                                                                                                                                                                                                                                                                             |
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-eps``,  ``--num-epochs``                |Default: ``-1``.                                                                                                                                                                                                                                                                                                        |
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-ttim``,  ``--max-train-time``           |Default: ``-1``.                                                                                                                                                                                                                                                                                                        |
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-vtim``,  ``--validation-every-n-secs``  |Default: ``-1``.                                                                                                                                                                                                                                                                                                        |
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-stim``,  ``--save-every-n-secs``        |Default: ``-1``.                                                                                                                                                                                                                                                                                                        |
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-veps``,  ``--validation-every-n-epochs``|Default: ``-1``.                                                                                                                                                                                                                                                                                                        |
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-vp``,  ``--validation-patience``        |Default: ``10``.                                                                                                                                                                                                                                                                                                        |
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-vmt``,  ``--validation-metric``         |Default: ``accuracy``.                                                                                                                                                                                                                                                                                                  |
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-vmm``,  ``--validation-metric-mode``    |Choices: ``max``, ``min``.                                                                                                                                                                                                                                                                                              |
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-mcs``,  ``--metrics``                   |Default: ``default``.                                                                                                                                                                                                                                                                                                   |
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+
+verify_data
+################
+
+**Short description:** Check tasks for common errors
+
+
+.. automodule:: parlai.scripts.verify_data
+
+CLI Arguments
+-------------
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|Argument                          |Description                                                                                                                                                                                                                                                                                                             |
++==================================+========================================================================================================================================================================================================================================================================================================================+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-dt``,  ``--datatype``          | | Choices: ``train``, ``train:stream``, ``train:ordered``, ``train:ordered:stream``, ``train:stream:ordered``, ``train:evalmode``, ``train:evalmode:stream``, ``train:evalmode:ordered``, ``train:evalmode:ordered:stream``, ``train:evalmode:stream:ordered``, ``valid``, ``valid:stream``, ``test``, ``test:stream``.|
+|                                  | | Default: ``train:ordered``.                                                                                                                                                                                                                                                                                          |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-nt``,  ``--numthreads``        |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-bs``,  ``--batchsize``         |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-dynb``,  ``--dynamic-batching``|Choices: ``None``, ``batchsort``, ``full``.                                                                                                                                                                                                                                                                             |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+|``-ltim``,  ``--log-every-n-secs``|Default: ``2``.                                                                                                                                                                                                                                                                                                         |
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
++----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+
diff --git a/docs/source/cli_advanced.rst b/docs/source/cli_advanced.rst
new file mode 100644
index 00000000000..94480e7385b
--- /dev/null
+++ b/docs/source/cli_advanced.rst
@@ -0,0 +1,6 @@
+Advance Scripts
+==================
+
+These are the more obscure and advanced scripts in parlai.
+
+.. include:: cli_advanced.inc

From 07e4b64e9386571130d335bdd56c930522530e8a Mon Sep 17 00:00:00 2001
From: Stephen Roller <roller@fb.com>
Date: Wed, 12 Aug 2020 21:41:04 -0400
Subject: [PATCH 10/22] Get rid of cli_advanced

---
 docs/source/cli_advanced.inc           | 652 -------------------------
 docs/source/{ => core}/agents.rst      |   0
 docs/source/{ => core}/build_data.rst  |   0
 docs/source/{ => core}/dict.rst        |   0
 docs/source/{ => core}/loader.rst      |   0
 docs/source/{ => core}/messages.rst    |   0
 docs/source/{ => core}/metrics_api.rst |   0
 docs/source/{ => core}/params.rst      |   0
 docs/source/{ => core}/script.rst      |   0
 docs/source/{ => core}/teachers.rst    |   0
 docs/source/{ => core}/torch_agent.rst |   0
 docs/source/{ => core}/worlds.rst      |   0
 12 files changed, 652 deletions(-)
 delete mode 100644 docs/source/cli_advanced.inc
 rename docs/source/{ => core}/agents.rst (100%)
 rename docs/source/{ => core}/build_data.rst (100%)
 rename docs/source/{ => core}/dict.rst (100%)
 rename docs/source/{ => core}/loader.rst (100%)
 rename docs/source/{ => core}/messages.rst (100%)
 rename docs/source/{ => core}/metrics_api.rst (100%)
 rename docs/source/{ => core}/params.rst (100%)
 rename docs/source/{ => core}/script.rst (100%)
 rename docs/source/{ => core}/teachers.rst (100%)
 rename docs/source/{ => core}/torch_agent.rst (100%)
 rename docs/source/{ => core}/worlds.rst (100%)

diff --git a/docs/source/cli_advanced.inc b/docs/source/cli_advanced.inc
deleted file mode 100644
index bef2663f092..00000000000
--- a/docs/source/cli_advanced.inc
+++ /dev/null
@@ -1,652 +0,0 @@
-build_candidates
-#####################
-
-**Short description:** Build the candidate responses for a retrieval model
-
-
-.. automodule:: parlai.scripts.build_candidates
-
-CLI Arguments
--------------
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|Argument                          |Description                                                                                                                                                                                                                                                                                                             |
-+==================================+========================================================================================================================================================================================================================================================================================================================+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-dt``,  ``--datatype``          | | Choices: ``train``, ``train:stream``, ``train:ordered``, ``train:ordered:stream``, ``train:stream:ordered``, ``train:evalmode``, ``train:evalmode:stream``, ``train:evalmode:ordered``, ``train:evalmode:ordered:stream``, ``train:evalmode:stream:ordered``, ``valid``, ``valid:stream``, ``test``, ``test:stream``.|
-|                                  | | Default: ``train:evalmode``.                                                                                                                                                                                                                                                                                         |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-nt``,  ``--numthreads``        |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-bs``,  ``--batchsize``         |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-dynb``,  ``--dynamic-batching``|Choices: ``None``, ``batchsort``, ``full``.                                                                                                                                                                                                                                                                             |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-n``,  ``--num-examples``       |Default: ``-1``.                                                                                                                                                                                                                                                                                                        |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-ltim``,  ``--log-every-n-secs``|Default: ``2``.                                                                                                                                                                                                                                                                                                         |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-
-
-build_dict
-###############
-
-**Short description:** Build a dictionary.
-
-
-.. automodule:: parlai.scripts.build_dict
-
-CLI Arguments
--------------
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|Argument                          |Description                                                                                                                                                                                                                                                                                                             |
-+==================================+========================================================================================================================================================================================================================================================================================================================+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-dt``,  ``--datatype``          | | Choices: ``train``, ``train:stream``, ``train:ordered``, ``train:ordered:stream``, ``train:stream:ordered``, ``train:evalmode``, ``train:evalmode:stream``, ``train:evalmode:ordered``, ``train:evalmode:ordered:stream``, ``train:evalmode:stream:ordered``, ``valid``, ``valid:stream``, ``test``, ``test:stream``.|
-|                                  | | Default: ``train``.                                                                                                                                                                                                                                                                                                  |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-nt``,  ``--numthreads``        |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-bs``,  ``--batchsize``         |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-dynb``,  ``--dynamic-batching``|Choices: ``None``, ``batchsort``, ``full``.                                                                                                                                                                                                                                                                             |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``--dict-maxexs``                 |Default: ``-1``.                                                                                                                                                                                                                                                                                                        |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-ltim``,  ``--log-every-n-secs``|Default: ``10``.                                                                                                                                                                                                                                                                                                        |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-
-
-convert_to_fasttext
-########################
-
-**Short description:** Convert data for ingestion in fastText
-
-
-.. automodule:: parlai.scripts.convert_data_to_fasttext_format
-
-CLI Arguments
--------------
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|Argument                          |Description                                                                                                                                                                                                                                                                                                             |
-+==================================+========================================================================================================================================================================================================================================================================================================================+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-dt``,  ``--datatype``          | | Choices: ``train``, ``train:stream``, ``train:ordered``, ``train:ordered:stream``, ``train:stream:ordered``, ``train:evalmode``, ``train:evalmode:stream``, ``train:evalmode:ordered``, ``train:evalmode:ordered:stream``, ``train:evalmode:stream:ordered``, ``valid``, ``valid:stream``, ``test``, ``test:stream``.|
-|                                  | | Default: ``train:ordered``.                                                                                                                                                                                                                                                                                          |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-nt``,  ``--numthreads``        |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-bs``,  ``--batchsize``         |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-dynb``,  ``--dynamic-batching``|Choices: ``None``, ``batchsort``, ``full``.                                                                                                                                                                                                                                                                             |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-n``,  ``--num-examples``       |Default: ``-1``.                                                                                                                                                                                                                                                                                                        |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-ltim``,  ``--log-every-n-secs``|Default: ``2``.                                                                                                                                                                                                                                                                                                         |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-
-
-convert_to_parlai
-######################
-
-**Short description:** Dump a task to a standardized format
-
-
-.. automodule:: parlai.scripts.convert_data_to_parlai_format
-
-CLI Arguments
--------------
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|Argument                          |Description                                                                                                                                                                                                                                                                                                             |
-+==================================+========================================================================================================================================================================================================================================================================================================================+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-dt``,  ``--datatype``          | | Choices: ``train``, ``train:stream``, ``train:ordered``, ``train:ordered:stream``, ``train:stream:ordered``, ``train:evalmode``, ``train:evalmode:stream``, ``train:evalmode:ordered``, ``train:evalmode:ordered:stream``, ``train:evalmode:stream:ordered``, ``valid``, ``valid:stream``, ``test``, ``test:stream``.|
-|                                  | | Default: ``train:stream``.                                                                                                                                                                                                                                                                                           |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-nt``,  ``--numthreads``        |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-bs``,  ``--batchsize``         |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-dynb``,  ``--dynamic-batching``|Choices: ``None``, ``batchsort``, ``full``.                                                                                                                                                                                                                                                                             |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-n``,  ``--num-examples``       |Default: ``-1``.                                                                                                                                                                                                                                                                                                        |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-if``,  ``--ignore-fields``     |Default: ``id``.                                                                                                                                                                                                                                                                                                        |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-ltim``,  ``--log-every-n-secs``|Default: ``2``.                                                                                                                                                                                                                                                                                                         |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-
-
-convo_render
-#################
-
-**Short description:** Render data as HTML
-
-
-.. automodule:: parlai.scripts.convo_render
-
-CLI Arguments
--------------
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|Argument                          |Description                                                                                                                                                                                                                                                                                                             |
-+==================================+========================================================================================================================================================================================================================================================================================================================+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-dt``,  ``--datatype``          | | Choices: ``train``, ``train:stream``, ``train:ordered``, ``train:ordered:stream``, ``train:stream:ordered``, ``train:evalmode``, ``train:evalmode:stream``, ``train:evalmode:ordered``, ``train:evalmode:ordered:stream``, ``train:evalmode:stream:ordered``, ``valid``, ``valid:stream``, ``test``, ``test:stream``.|
-|                                  | | Default: ``train``.                                                                                                                                                                                                                                                                                                  |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-nt``,  ``--numthreads``        |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-bs``,  ``--batchsize``         |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-dynb``,  ``--dynamic-batching``|Choices: ``None``, ``batchsort``, ``full``.                                                                                                                                                                                                                                                                             |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``--width``,  ``-wd``             |Default: ``8``.                                                                                                                                                                                                                                                                                                         |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``--height``,  ``-ht``            |Default: ``10``.                                                                                                                                                                                                                                                                                                        |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``--user-icon``,  ``-uic``        |Default: ``https://emojipedia-us.s3.dualstack.us-west-1.amazonaws.com/thumbs/160/apple/76/woman_1f469.png``.                                                                                                                                                                                                            |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``--alt-icon``,  ``-aic``         |Default: ``https://emojipedia-us.s3.dualstack.us-west-1.amazonaws.com/thumbs/160/facebook/230/parrot_1f99c.png``.                                                                                                                                                                                                       |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``--num-examples``,  ``-ne``      |Default: ``10``.                                                                                                                                                                                                                                                                                                        |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-
-
-data_stats
-###############
-
-**Short description:** Compute data statistics
-
-
-.. automodule:: parlai.scripts.data_stats
-
-CLI Arguments
--------------
-+-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|Argument                             |Description                                                                                                                                                                                                                                                                                                             |
-+=====================================+========================================================================================================================================================================================================================================================================================================================+
-+-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-dt``,  ``--datatype``             | | Choices: ``train``, ``train:stream``, ``train:ordered``, ``train:ordered:stream``, ``train:stream:ordered``, ``train:evalmode``, ``train:evalmode:stream``, ``train:evalmode:ordered``, ``train:evalmode:ordered:stream``, ``train:evalmode:stream:ordered``, ``valid``, ``valid:stream``, ``test``, ``test:stream``.|
-|                                     | | Default: ``train:ordered``.                                                                                                                                                                                                                                                                                          |
-+-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-nt``,  ``--numthreads``           |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
-+-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-bs``,  ``--batchsize``            |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
-+-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-dynb``,  ``--dynamic-batching``   |Choices: ``None``, ``batchsort``, ``full``.                                                                                                                                                                                                                                                                             |
-+-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-n``,  ``-ne``,  ``--num-examples``|Default: ``-1``.                                                                                                                                                                                                                                                                                                        |
-+-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-ltim``,  ``--log-every-n-secs``   |Default: ``2``.                                                                                                                                                                                                                                                                                                         |
-+-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``--agent``                          |Choices: ``0``, ``1``.                                                                                                                                                                                                                                                                                                  |
-+-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-
-
-detect_offensive
-#####################
-
-**Short description:** Check task for offensive language
-
-
-.. automodule:: parlai.scripts.detect_offensive_language
-
-CLI Arguments
--------------
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|Argument                          |Description                                                                                                                                                                                                                                                                                                             |
-+==================================+========================================================================================================================================================================================================================================================================================================================+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-dt``,  ``--datatype``          | | Choices: ``train``, ``train:stream``, ``train:ordered``, ``train:ordered:stream``, ``train:stream:ordered``, ``train:evalmode``, ``train:evalmode:stream``, ``train:evalmode:ordered``, ``train:evalmode:ordered:stream``, ``train:evalmode:stream:ordered``, ``valid``, ``valid:stream``, ``test``, ``test:stream``.|
-|                                  | | Default: ``train:ordered``.                                                                                                                                                                                                                                                                                          |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-nt``,  ``--numthreads``        |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-bs``,  ``--batchsize``         |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-dynb``,  ``--dynamic-batching``|Choices: ``None``, ``batchsort``, ``full``.                                                                                                                                                                                                                                                                             |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-m``,  ``--model``              |Default: ``repeat_query``.                                                                                                                                                                                                                                                                                              |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-ltim``,  ``--log-every-n-secs``|Default: ``2``.                                                                                                                                                                                                                                                                                                         |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``--safety``                      | | Choices: ``all``, ``classifier``, ``string_matcher``.                                                                                                                                                                                                                                                                |
-|                                  | | Default: ``all``.                                                                                                                                                                                                                                                                                                    |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-
-
-eval_wordstat
-##################
-
-**Short description:** Compute statistics from model predictions
-
-
-.. automodule:: parlai.scripts.eval_wordstat
-
-CLI Arguments
--------------
-+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|Argument                              |Description                                                                                                                                                                                                                                                                                                             |
-+======================================+========================================================================================================================================================================================================================================================================================================================+
-+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-dt``,  ``--datatype``              | | Choices: ``train``, ``train:stream``, ``train:ordered``, ``train:ordered:stream``, ``train:stream:ordered``, ``train:evalmode``, ``train:evalmode:stream``, ``train:evalmode:ordered``, ``train:evalmode:ordered:stream``, ``train:evalmode:stream:ordered``, ``valid``, ``valid:stream``, ``test``, ``test:stream``.|
-|                                      | | Default: ``valid``.                                                                                                                                                                                                                                                                                                  |
-+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-nt``,  ``--numthreads``            |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
-+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-bs``,  ``--batchsize``             |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
-+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-dynb``,  ``--dynamic-batching``    |Choices: ``None``, ``batchsort``, ``full``.                                                                                                                                                                                                                                                                             |
-+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-ne``,  ``--num-examples``          |Default: ``-1``.                                                                                                                                                                                                                                                                                                        |
-+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-ltim``,  ``--log-every-n-secs``    |Default: ``2``.                                                                                                                                                                                                                                                                                                         |
-+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-fb``,  ``--freq-bins``             |Default: ``0,100,1000,10000``.                                                                                                                                                                                                                                                                                          |
-+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-cun``,  ``--compute-unique``       |Default: ``True``.                                                                                                                                                                                                                                                                                                      |
-+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-
-
-extract_image_feature
-##########################
-
-**Short description:** Load/extract image features
-
-
-.. automodule:: parlai.scripts.extract_image_feature
-
-CLI Arguments
--------------
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|Argument                          |Description                                                                                                                                                                                                                                                                                                             |
-+==================================+========================================================================================================================================================================================================================================================================================================================+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-dt``,  ``--datatype``          | | Choices: ``train``, ``train:stream``, ``train:ordered``, ``train:ordered:stream``, ``train:stream:ordered``, ``train:evalmode``, ``train:evalmode:stream``, ``train:evalmode:ordered``, ``train:evalmode:ordered:stream``, ``train:evalmode:stream:ordered``, ``valid``, ``valid:stream``, ``test``, ``test:stream``.|
-|                                  | | Default: ``train``.                                                                                                                                                                                                                                                                                                  |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-nt``,  ``--numthreads``        |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-bs``,  ``--batchsize``         |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-dynb``,  ``--dynamic-batching``|Choices: ``None``, ``batchsort``, ``full``.                                                                                                                                                                                                                                                                             |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-
-
-interactive_web
-####################
-
-**Short description:** Interactive chat with a model in a web browser
-
-**Aliases:** ``iweb``
-
-.. automodule:: parlai.scripts.interactive_web
-
-CLI Arguments
--------------
-+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|Argument                                           |Description                                                                                                                                                                                                                                                                                                             |
-+===================================================+========================================================================================================================================================================================================================================================================================================================+
-+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-t``,  ``--task``                                |Default: ``interactive``.                                                                                                                                                                                                                                                                                               |
-+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-dt``,  ``--datatype``                           | | Choices: ``train``, ``train:stream``, ``train:ordered``, ``train:ordered:stream``, ``train:stream:ordered``, ``train:evalmode``, ``train:evalmode:stream``, ``train:evalmode:ordered``, ``train:evalmode:ordered:stream``, ``train:evalmode:stream:ordered``, ``valid``, ``valid:stream``, ``test``, ``test:stream``.|
-|                                                   | | Default: ``train``.                                                                                                                                                                                                                                                                                                  |
-+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-nt``,  ``--numthreads``                         |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
-+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-bs``,  ``--batchsize``                          |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
-+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-dynb``,  ``--dynamic-batching``                 |Choices: ``None``, ``batchsort``, ``full``.                                                                                                                                                                                                                                                                             |
-+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``--display-ignore-fields``                        |Default: ``label_candidates,text_candidates``.                                                                                                                                                                                                                                                                          |
-+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-it``,  ``--interactive-task``                   |Default: ``True``.                                                                                                                                                                                                                                                                                                      |
-+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``--save-format``                                  | | Choices: ``conversations``, ``parlai``.                                                                                                                                                                                                                                                                              |
-|                                                   | | Default: ``conversations``.                                                                                                                                                                                                                                                                                          |
-+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``--log-keep-fields``                              |Default: ``all``.                                                                                                                                                                                                                                                                                                       |
-+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``--port``                                         |Default: ``8080``.                                                                                                                                                                                                                                                                                                      |
-+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``--host``                                         |Default: ``localhost``.                                                                                                                                                                                                                                                                                                 |
-+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-
-
-multiprocessing_eval
-#########################
-
-**Short description:** Evaluate a model
-
-**Aliases:** ``mp_eval``
-
-.. automodule:: parlai.scripts.multiprocessing_eval
-
-CLI Arguments
--------------
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|Argument                          |Description                                                                                                                                                                                                                                                                                                             |
-+==================================+========================================================================================================================================================================================================================================================================================================================+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-dt``,  ``--datatype``          | | Choices: ``train``, ``train:stream``, ``train:ordered``, ``train:ordered:stream``, ``train:stream:ordered``, ``train:evalmode``, ``train:evalmode:stream``, ``train:evalmode:ordered``, ``train:evalmode:ordered:stream``, ``train:evalmode:stream:ordered``, ``valid``, ``valid:stream``, ``test``, ``test:stream``.|
-|                                  | | Default: ``valid``.                                                                                                                                                                                                                                                                                                  |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-nt``,  ``--numthreads``        |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-bs``,  ``--batchsize``         |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-dynb``,  ``--dynamic-batching``|Choices: ``None``, ``batchsort``, ``full``.                                                                                                                                                                                                                                                                             |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``--save-format``                 | | Choices: ``conversations``, ``parlai``.                                                                                                                                                                                                                                                                              |
-|                                  | | Default: ``conversations``.                                                                                                                                                                                                                                                                                          |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-ne``,  ``--num-examples``      |Default: ``-1``.                                                                                                                                                                                                                                                                                                        |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-ltim``,  ``--log-every-n-secs``|Default: ``10``.                                                                                                                                                                                                                                                                                                        |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-mcs``,  ``--metrics``          |Default: ``default``.                                                                                                                                                                                                                                                                                                   |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``--log-keep-fields``             |Default: ``all``.                                                                                                                                                                                                                                                                                                       |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``--distributed-world-size``      |Default: ``2``.                                                                                                                                                                                                                                                                                                         |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-
-
-multiprocessing_train
-##########################
-
-**Short description:** Train a model
-
-**Aliases:** ``mp_train``
-
-.. automodule:: parlai.scripts.multiprocessing_train
-
-CLI Arguments
--------------
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|Argument                                   |Description                                                                                                                                                                                                                                                                                                             |
-+===========================================+========================================================================================================================================================================================================================================================================================================================+
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-dt``,  ``--datatype``                   | | Choices: ``train``, ``train:stream``, ``train:ordered``, ``train:ordered:stream``, ``train:stream:ordered``, ``train:evalmode``, ``train:evalmode:stream``, ``train:evalmode:ordered``, ``train:evalmode:ordered:stream``, ``train:evalmode:stream:ordered``, ``valid``, ``valid:stream``, ``test``, ``test:stream``.|
-|                                           | | Default: ``train``.                                                                                                                                                                                                                                                                                                  |
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-nt``,  ``--numthreads``                 |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-bs``,  ``--batchsize``                  |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-dynb``,  ``--dynamic-batching``         |Choices: ``None``, ``batchsort``, ``full``.                                                                                                                                                                                                                                                                             |
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-eps``,  ``--num-epochs``                |Default: ``-1``.                                                                                                                                                                                                                                                                                                        |
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-ttim``,  ``--max-train-time``           |Default: ``-1``.                                                                                                                                                                                                                                                                                                        |
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-vtim``,  ``--validation-every-n-secs``  |Default: ``-1``.                                                                                                                                                                                                                                                                                                        |
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-stim``,  ``--save-every-n-secs``        |Default: ``-1``.                                                                                                                                                                                                                                                                                                        |
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-veps``,  ``--validation-every-n-epochs``|Default: ``-1``.                                                                                                                                                                                                                                                                                                        |
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-vp``,  ``--validation-patience``        |Default: ``10``.                                                                                                                                                                                                                                                                                                        |
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-vmt``,  ``--validation-metric``         |Default: ``accuracy``.                                                                                                                                                                                                                                                                                                  |
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-vmm``,  ``--validation-metric-mode``    |Choices: ``max``, ``min``.                                                                                                                                                                                                                                                                                              |
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-mcs``,  ``--metrics``                   |Default: ``default``.                                                                                                                                                                                                                                                                                                   |
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``--distributed-world-size``               |Default: ``2``.                                                                                                                                                                                                                                                                                                         |
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-
-
-party
-##########
-
-**Short description:** Throw a party!
-
-**Aliases:** ``parrot``
-
-.. automodule:: parlai.scripts.party
-
-CLI Arguments
--------------
-+-------------------+-----------+
-|Argument           |Description|
-+===================+===========+
-+-------------------+-----------+
-
-
-profile_interactive
-########################
-
-**Short description:** Interactive chat with a model
-
-
-.. automodule:: parlai.scripts.profile_interactive
-
-CLI Arguments
--------------
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|Argument                          |Description                                                                                                                                                                                                                                                                                                             |
-+==================================+========================================================================================================================================================================================================================================================================================================================+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-t``,  ``--task``               |Default: ``interactive``.                                                                                                                                                                                                                                                                                               |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-dt``,  ``--datatype``          | | Choices: ``train``, ``train:stream``, ``train:ordered``, ``train:ordered:stream``, ``train:stream:ordered``, ``train:evalmode``, ``train:evalmode:stream``, ``train:evalmode:ordered``, ``train:evalmode:ordered:stream``, ``train:evalmode:stream:ordered``, ``valid``, ``valid:stream``, ``test``, ``test:stream``.|
-|                                  | | Default: ``train``.                                                                                                                                                                                                                                                                                                  |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-nt``,  ``--numthreads``        |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-bs``,  ``--batchsize``         |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-dynb``,  ``--dynamic-batching``|Choices: ``None``, ``batchsort``, ``full``.                                                                                                                                                                                                                                                                             |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-d``,  ``--display-examples``   |Default: ``True``.                                                                                                                                                                                                                                                                                                      |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-ne``,  ``--num-examples``      |Default: ``5``.                                                                                                                                                                                                                                                                                                         |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``--display-ignore-fields``       |Default: ``label_candidates,text_candidates``.                                                                                                                                                                                                                                                                          |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-it``,  ``--interactive-task``  |Default: ``True``.                                                                                                                                                                                                                                                                                                      |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-
-
-profile_train
-##################
-
-**Short description:** cProfile a training run
-
-
-.. automodule:: parlai.scripts.profile_train
-
-CLI Arguments
--------------
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|Argument                                   |Description                                                                                                                                                                                                                                                                                                             |
-+===========================================+========================================================================================================================================================================================================================================================================================================================+
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-dt``,  ``--datatype``                   | | Choices: ``train``, ``train:stream``, ``train:ordered``, ``train:ordered:stream``, ``train:stream:ordered``, ``train:evalmode``, ``train:evalmode:stream``, ``train:evalmode:ordered``, ``train:evalmode:ordered:stream``, ``train:evalmode:stream:ordered``, ``valid``, ``valid:stream``, ``test``, ``test:stream``.|
-|                                           | | Default: ``train``.                                                                                                                                                                                                                                                                                                  |
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-nt``,  ``--numthreads``                 |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-bs``,  ``--batchsize``                  |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-dynb``,  ``--dynamic-batching``         |Choices: ``None``, ``batchsort``, ``full``.                                                                                                                                                                                                                                                                             |
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-eps``,  ``--num-epochs``                |Default: ``-1``.                                                                                                                                                                                                                                                                                                        |
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-ttim``,  ``--max-train-time``           |Default: ``-1``.                                                                                                                                                                                                                                                                                                        |
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-vtim``,  ``--validation-every-n-secs``  |Default: ``-1``.                                                                                                                                                                                                                                                                                                        |
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-stim``,  ``--save-every-n-secs``        |Default: ``-1``.                                                                                                                                                                                                                                                                                                        |
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-veps``,  ``--validation-every-n-epochs``|Default: ``-1``.                                                                                                                                                                                                                                                                                                        |
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-vp``,  ``--validation-patience``        |Default: ``10``.                                                                                                                                                                                                                                                                                                        |
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-vmt``,  ``--validation-metric``         |Default: ``accuracy``.                                                                                                                                                                                                                                                                                                  |
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-vmm``,  ``--validation-metric-mode``    |Choices: ``max``, ``min``.                                                                                                                                                                                                                                                                                              |
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-mcs``,  ``--metrics``                   |Default: ``default``.                                                                                                                                                                                                                                                                                                   |
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+-------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-
-
-verify_data
-################
-
-**Short description:** Check tasks for common errors
-
-
-.. automodule:: parlai.scripts.verify_data
-
-CLI Arguments
--------------
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|Argument                          |Description                                                                                                                                                                                                                                                                                                             |
-+==================================+========================================================================================================================================================================================================================================================================================================================+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-dt``,  ``--datatype``          | | Choices: ``train``, ``train:stream``, ``train:ordered``, ``train:ordered:stream``, ``train:stream:ordered``, ``train:evalmode``, ``train:evalmode:stream``, ``train:evalmode:ordered``, ``train:evalmode:ordered:stream``, ``train:evalmode:stream:ordered``, ``valid``, ``valid:stream``, ``test``, ``test:stream``.|
-|                                  | | Default: ``train:ordered``.                                                                                                                                                                                                                                                                                          |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-nt``,  ``--numthreads``        |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-bs``,  ``--batchsize``         |Default: ``1``.                                                                                                                                                                                                                                                                                                         |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-dynb``,  ``--dynamic-batching``|Choices: ``None``, ``batchsort``, ``full``.                                                                                                                                                                                                                                                                             |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-|``-ltim``,  ``--log-every-n-secs``|Default: ``2``.                                                                                                                                                                                                                                                                                                         |
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-
-
diff --git a/docs/source/agents.rst b/docs/source/core/agents.rst
similarity index 100%
rename from docs/source/agents.rst
rename to docs/source/core/agents.rst
diff --git a/docs/source/build_data.rst b/docs/source/core/build_data.rst
similarity index 100%
rename from docs/source/build_data.rst
rename to docs/source/core/build_data.rst
diff --git a/docs/source/dict.rst b/docs/source/core/dict.rst
similarity index 100%
rename from docs/source/dict.rst
rename to docs/source/core/dict.rst
diff --git a/docs/source/loader.rst b/docs/source/core/loader.rst
similarity index 100%
rename from docs/source/loader.rst
rename to docs/source/core/loader.rst
diff --git a/docs/source/messages.rst b/docs/source/core/messages.rst
similarity index 100%
rename from docs/source/messages.rst
rename to docs/source/core/messages.rst
diff --git a/docs/source/metrics_api.rst b/docs/source/core/metrics_api.rst
similarity index 100%
rename from docs/source/metrics_api.rst
rename to docs/source/core/metrics_api.rst
diff --git a/docs/source/params.rst b/docs/source/core/params.rst
similarity index 100%
rename from docs/source/params.rst
rename to docs/source/core/params.rst
diff --git a/docs/source/script.rst b/docs/source/core/script.rst
similarity index 100%
rename from docs/source/script.rst
rename to docs/source/core/script.rst
diff --git a/docs/source/teachers.rst b/docs/source/core/teachers.rst
similarity index 100%
rename from docs/source/teachers.rst
rename to docs/source/core/teachers.rst
diff --git a/docs/source/torch_agent.rst b/docs/source/core/torch_agent.rst
similarity index 100%
rename from docs/source/torch_agent.rst
rename to docs/source/core/torch_agent.rst
diff --git a/docs/source/worlds.rst b/docs/source/core/worlds.rst
similarity index 100%
rename from docs/source/worlds.rst
rename to docs/source/core/worlds.rst

From d41449a11c8791345aab85f34a7dd0134ad9cf10 Mon Sep 17 00:00:00 2001
From: Stephen Roller <roller@fb.com>
Date: Wed, 12 Aug 2020 21:43:30 -0400
Subject: [PATCH 11/22] Minor changes

---
 .gitignore                     |  1 +
 docs/source/core/messages.rst  |  2 +-
 docs/source/index.rst          | 12 +-----------
 docs/source/tutorial_basic.rst |  2 +-
 docs/source/tutorial_task.rst  |  8 ++++----
 5 files changed, 8 insertions(+), 17 deletions(-)

diff --git a/.gitignore b/.gitignore
index 54f0983f990..f31dd56aa16 100644
--- a/.gitignore
+++ b/.gitignore
@@ -86,6 +86,7 @@ docs/_build/
 docs/source/task_list.inc
 docs/source/zoo_list.inc
 docs/source/cli_usage.inc
+docs/source/cli_advance.inc
 docs/source/agent_refs
 
 # PyBuilder
diff --git a/docs/source/core/messages.rst b/docs/source/core/messages.rst
index ad9152de9b5..849c0b7dfac 100644
--- a/docs/source/core/messages.rst
+++ b/docs/source/core/messages.rst
@@ -6,7 +6,7 @@ parlai.core.message
 
 
 
-.. image:: _static/img/act-obs-dict.png
+.. image:: /_static/img/act-obs-dict.png
     :width: 60 %
 
 The primary medium for information flow (messages between agents and the environment)
diff --git a/docs/source/index.rst b/docs/source/index.rst
index 2747b3c1afb..ef20b2a562f 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -50,17 +50,7 @@ ParlAI is a one-stop-shop for dialog research.
    :maxdepth: 2
    :caption: API Reference
 
-   agents
-   torch_agent
-   messages
-   dict
-   build_data
-   metrics_api
-   params
-   teachers
-   loader
-   script
-   worlds
+   core
    utils
 
 
diff --git a/docs/source/tutorial_basic.rst b/docs/source/tutorial_basic.rst
index 06ae77b08bf..863eb725c92 100644
--- a/docs/source/tutorial_basic.rst
+++ b/docs/source/tutorial_basic.rst
@@ -65,7 +65,7 @@ These observations and actions are the primary way agents in ParlAI communicate
 The Message object is a subclass of a python dict, whose key function is to prevent users from editing
 fields in an action or observation unintentionally.
 
-The :doc:`messages <messages>` documentation goes into more detail about
+The :doc:`messages <core/messages>` documentation goes into more detail about
 each field, but the following table shows the basics.
 
 
diff --git a/docs/source/tutorial_task.rst b/docs/source/tutorial_task.rst
index 6301ecdd3f5..a448bdc010e 100644
--- a/docs/source/tutorial_task.rst
+++ b/docs/source/tutorial_task.rst
@@ -53,9 +53,9 @@ We could look at that data using the usual display data script:
 	[ loaded 1 episodes with a total of 2 examples ]
 
 The text file data format is called ParlAI Dialog format, and is described
-in the :doc:`teachers documentation <teachers>` (parlai.core.teachers.ParlAIDialogTeacher)
+in the :doc:`teachers documentation <core/teachers>` (parlai.core.teachers.ParlAIDialogTeacher)
 and
-in the `core/teachers.py file <https://github.com/facebookresearch/ParlAI/blob/master/parlai/core/teachers.py#L1098>`_.
+in the `core/teachers.py file <parlai.core.teachers.ParlAIDialogTeacher>`_.
 Essentially, there is one training example every line, and each field in a
 ParlAI message is tab separated with the name of the field, followed by a colon.
 E.g. the usual fields like 'text', 'labels', 'label_candidates' etc. can all
@@ -214,7 +214,7 @@ In this section we will illustrate the process of using the ``ParlAIDialogTeache
 class by adding the Twitter dataset.
 This task has data in textual form and has been formatted to follow the ParlAI Dialog format.
 It is thus very simple to implement it using ``ParlAIDialogTeacher``.
-More information on this class and the dialog format can be found in the :doc:`teachers documentation <teachers>`.
+More information on this class and the dialog format can be found in the :doc:`teachers documentation <core/teachers>`.
 
 In this task, the agent is presented with questions about movies that are answerable from Wikipedia.
 A sample dialog is demonstrated below.
@@ -385,7 +385,7 @@ The second is a boolean flag ``new_episode?`` which indicates if the current
 query starts a new episode or not.
 
 More information on this format can be found in the documentation under ``DialogData``
-in the :doc:`teachers documentation <teachers>`
+in the :doc:`teachers documentation <core/teachers>`
 (``setup_data`` is provided as a data_loader to ``DialogData``).
 
 The sample ``setup_data`` method for our task is presented below.

From 11c116ff00d6f42f6e6fc506808f169062220160 Mon Sep 17 00:00:00 2001
From: Stephen Roller <roller@fb.com>
Date: Wed, 12 Aug 2020 21:55:54 -0400
Subject: [PATCH 12/22] grr.

---
 docs/source/_static/css/parlai_theme.css |  2 +-
 docs/source/generate_cli.py              | 16 +++++++++++-----
 2 files changed, 12 insertions(+), 6 deletions(-)

diff --git a/docs/source/_static/css/parlai_theme.css b/docs/source/_static/css/parlai_theme.css
index 173194b797c..39a16aea976 100644
--- a/docs/source/_static/css/parlai_theme.css
+++ b/docs/source/_static/css/parlai_theme.css
@@ -32,7 +32,7 @@ h3 { size: 150%; }
 }
 
 th p, td div.line-block {
-    margin-bottom: 0px;
+    margin-bottom: 0px !important;
 }
 
 .text-align\:right {
diff --git a/docs/source/generate_cli.py b/docs/source/generate_cli.py
index 16af59c33e5..2c7aaa86057 100755
--- a/docs/source/generate_cli.py
+++ b/docs/source/generate_cli.py
@@ -25,18 +25,22 @@ def render_script(fout, key, registration):
         if hasattr(action, 'hidden') and action.hidden:
             # some options are marked hidden
             continue
-        action_strings = ",  ".join(f'``{a}``' for a in action.option_strings)
-        description = []
         if action.dest == argparse.SUPPRESS:
             continue
+        action_strings = ",  ".join(f'``{a}``' for a in action.option_strings)
+        if not action_strings:
+            continue
+        description = []
+        if action.help:
+            h = action.help
+            if not h[0].isupper():
+                h = h[0].upper() + h[1:]
+            description += [h]
         # list choices if there are any
         if action.choices:
             description += [
                 "Choices: " + ", ".join(f'``{c}``' for c in action.choices) + "."
             ]
-        # list default and recommended values.
-        if hasattr(action, 'hidden') and action.hidden:
-            continue
         default_value = ""
         if action.default and action.default != argparse.SUPPRESS:
             default_value += f"Default: ``{action.default}``.  "
@@ -51,6 +55,8 @@ def render_script(fout, key, registration):
         # escape for the fact that we're inserting this inside a table
         if len(description) > 1:
             description = [' | ' + d for d in description]
+        if len(description) == 0:
+            description = [""]
         actions.append((action_strings, description))
 
     if not actions:

From a32b229e1bd17b0343d87a06b515d664dd25eaac Mon Sep 17 00:00:00 2001
From: Stephen Roller <roller@fb.com>
Date: Wed, 12 Aug 2020 21:59:37 -0400
Subject: [PATCH 13/22] One more.

---
 docs/source/generate_agent_list.py | 1 +
 docs/source/generate_cli.py        | 1 +
 2 files changed, 2 insertions(+)

diff --git a/docs/source/generate_agent_list.py b/docs/source/generate_agent_list.py
index 8b0ef373c05..79341e82bf2 100644
--- a/docs/source/generate_agent_list.py
+++ b/docs/source/generate_agent_list.py
@@ -42,6 +42,7 @@ def _make_argparse_table(class_):
                 h = action.help
                 if not h[0].isupper():
                     h = h[0].upper() + h[1:]
+                h = h.replace("%(default)s", str(action.default))
                 description += [h]
             # list choices if there are any
             if action.choices:
diff --git a/docs/source/generate_cli.py b/docs/source/generate_cli.py
index 2c7aaa86057..05ac0ed6f05 100755
--- a/docs/source/generate_cli.py
+++ b/docs/source/generate_cli.py
@@ -35,6 +35,7 @@ def render_script(fout, key, registration):
             h = action.help
             if not h[0].isupper():
                 h = h[0].upper() + h[1:]
+            h = h.replace("%(default)s", f'``{action.default}``')
             description += [h]
         # list choices if there are any
         if action.choices:

From 41a4934ca392b24eaace8dff1a47434c2ab09612 Mon Sep 17 00:00:00 2001
From: Stephen Roller <roller@fb.com>
Date: Wed, 12 Aug 2020 22:24:30 -0400
Subject: [PATCH 14/22] Missing.

---
 .gitignore           |  2 +-
 docs/source/core.rst | 11 +++++++++++
 2 files changed, 12 insertions(+), 1 deletion(-)
 create mode 100644 docs/source/core.rst

diff --git a/.gitignore b/.gitignore
index f31dd56aa16..9f04a8bb2cf 100644
--- a/.gitignore
+++ b/.gitignore
@@ -86,7 +86,7 @@ docs/_build/
 docs/source/task_list.inc
 docs/source/zoo_list.inc
 docs/source/cli_usage.inc
-docs/source/cli_advance.inc
+docs/source/cli_advanced.inc
 docs/source/agent_refs
 
 # PyBuilder
diff --git a/docs/source/core.rst b/docs/source/core.rst
new file mode 100644
index 00000000000..004e7bf8ef2
--- /dev/null
+++ b/docs/source/core.rst
@@ -0,0 +1,11 @@
+parlai.core
+---------------
+
+This contains the full, comprehensive docstrings of all of ``parlai.core``,
+which powers the vast majority of ParlAI.
+
+.. toctree::
+   :maxdepth: 1
+   :glob:
+
+   core/*

From d39b53aadacf15f1a9469a3e586a00dce89c9363 Mon Sep 17 00:00:00 2001
From: Stephen Roller <roller@fb.com>
Date: Wed, 12 Aug 2020 22:46:17 -0400
Subject: [PATCH 15/22] Tweak colors.

---
 docs/source/_static/css/parlai_theme.css |  2 +-
 docs/source/generate_cli.py              |  2 +-
 docs/source/tutorial_task.rst            | 15 +++++++--------
 3 files changed, 9 insertions(+), 10 deletions(-)

diff --git a/docs/source/_static/css/parlai_theme.css b/docs/source/_static/css/parlai_theme.css
index 39a16aea976..d974f8d5b11 100644
--- a/docs/source/_static/css/parlai_theme.css
+++ b/docs/source/_static/css/parlai_theme.css
@@ -80,7 +80,7 @@ th p, td div.line-block {
 
 /* Use our red for literals (it's very similar to the original color) */
 .rst-content tt.literal, .rst-content tt.literal, .rst-content code.literal {
-    color: #CE3239;
+    color: #222;
 }
 
 .rst-content tt.xref, a .rst-content tt, .rst-content tt.xref,
diff --git a/docs/source/generate_cli.py b/docs/source/generate_cli.py
index 05ac0ed6f05..2c996cb47b4 100755
--- a/docs/source/generate_cli.py
+++ b/docs/source/generate_cli.py
@@ -7,6 +7,7 @@
 import argparse
 import parlai.core.script as pcs
 
+
 def render_script(fout, key, registration):
     script_parser = registration.klass.setup_args()
     description = script_parser.description
@@ -85,7 +86,6 @@ def render_script(fout, key, registration):
     fout.write('\n\n')
 
 
-
 def main():
     fout = open('cli_usage.inc', 'w')
     pcs.setup_script_registry()
diff --git a/docs/source/tutorial_task.rst b/docs/source/tutorial_task.rst
index a448bdc010e..f3dae97a3bb 100644
--- a/docs/source/tutorial_task.rst
+++ b/docs/source/tutorial_task.rst
@@ -52,14 +52,13 @@ We could look at that data using the usual display data script:
 	EPOCH DONE
 	[ loaded 1 episodes with a total of 2 examples ]
 
-The text file data format is called ParlAI Dialog format, and is described
-in the :doc:`teachers documentation <core/teachers>` (parlai.core.teachers.ParlAIDialogTeacher)
-and
-in the `core/teachers.py file <parlai.core.teachers.ParlAIDialogTeacher>`_.
-Essentially, there is one training example every line, and each field in a
-ParlAI message is tab separated with the name of the field, followed by a colon.
-E.g. the usual fields like 'text', 'labels', 'label_candidates' etc. can all
-be used, or you can add your own fields too if you have a special use for them.
+The text file data format is called ParlAI Dialog format, and is described in
+the :doc:`teachers documentation <core/teachers>` and
+:py:class:`parlai.core.teachers.ParlAIDialogTeacher`.  Essentially, there is one
+training example every line, and each field in a ParlAI message is tab
+separated with the name of the field, followed by a colon.  E.g. the usual
+fields like 'text', 'labels', 'label_candidates' etc. can all be used, or you
+can add your own fields too if you have a special use for them.
 
 Handling Separate Train/Valid/Test data
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

From 969e37a556167d25edd83e59bf916a380210df1b Mon Sep 17 00:00:00 2001
From: Stephen Roller <roller@fb.com>
Date: Wed, 12 Aug 2020 23:16:04 -0400
Subject: [PATCH 16/22] Drop some comments.

---
 docs/source/_static/css/parlai_theme.css |  2 +-
 docs/source/generate_agent_list.py       | 12 ------------
 2 files changed, 1 insertion(+), 13 deletions(-)

diff --git a/docs/source/_static/css/parlai_theme.css b/docs/source/_static/css/parlai_theme.css
index d974f8d5b11..97e4c9dc393 100644
--- a/docs/source/_static/css/parlai_theme.css
+++ b/docs/source/_static/css/parlai_theme.css
@@ -80,7 +80,7 @@ th p, td div.line-block {
 
 /* Use our red for literals (it's very similar to the original color) */
 .rst-content tt.literal, .rst-content tt.literal, .rst-content code.literal {
-    color: #222;
+    color: #333;
 }
 
 .rst-content tt.xref, a .rst-content tt, .rst-content tt.xref,
diff --git a/docs/source/generate_agent_list.py b/docs/source/generate_agent_list.py
index 79341e82bf2..2fc6435b5f2 100644
--- a/docs/source/generate_agent_list.py
+++ b/docs/source/generate_agent_list.py
@@ -77,18 +77,6 @@ def _make_argparse_table(class_):
             text = text.replace("\n", "<br>")
             readme.append(f"{text}\n")
         readme.append("\n\n")
-
-        # render the table
-        # readme.append(f"```eval_rst\n")
-        # readme.append(f".. csv-table:: {ag.title}\n")
-        # readme.append(f'   :widths: 35, 65\n\n')
-        # cout = io.StringIO()
-        # csvw = csv.writer(cout, csv.unix_dialect, delimiter=",")
-        # for row in actions:
-        #     cout.write("   ")
-        #     csvw.writerow(row)
-        # readme.append(cout.getvalue())
-        # readme.append("```\n\n")
     return readme
 
 

From 3601d27f602b82423b8b8bea0120da1ef230b79b Mon Sep 17 00:00:00 2001
From: Stephen Roller <roller@fb.com>
Date: Wed, 12 Aug 2020 19:25:41 -0400
Subject: [PATCH 17/22] Add metrics tutorial.

---
 docs/source/tutorial_metrics.md | 372 ++++++++++++++++++++++++++++++++
 1 file changed, 372 insertions(+)
 create mode 100644 docs/source/tutorial_metrics.md

diff --git a/docs/source/tutorial_metrics.md b/docs/source/tutorial_metrics.md
new file mode 100644
index 00000000000..650ba4ceb7e
--- /dev/null
+++ b/docs/source/tutorial_metrics.md
@@ -0,0 +1,372 @@
+# Understanding and adding new metrics
+
+Author: Stephen Roller
+
+## Introduction and Standard Metrics
+
+ParlAI contains a number of built-in metrics that are automatically computed when
+we train and evaluate models. Some of these metrics are _text generation_ metrics,
+which happen any time we generate a text: this includes F1, BLEU and Accuracy.
+
+For example, let's try a Fixed Response model, which always returns a given fixed
+response, and evaluate on the DailyDialog dataset:
+
+```
+$ parlai eval_model -m fixed_response -t dailydialog --fixed-response "how may i help you ?"
+... after a while ...
+14:41:40 | Evaluating task dailydialog using datatype valid.
+14:41:40 | creating task(s): dailydialog
+14:41:41 | Finished evaluating tasks ['dailydialog'] using datatype valid
+    accuracy  bleu-4  exs    f1
+    .0001239 .002617 8069 .1163
+```
+
+We see that we got 0.01239% accuracy, 0.26% BLEU-4 score, and 11.63% F1 across
+8069 examples. What do those metrics means?
+
+- Accuracy: this is perfect, exact, matching of the response, averaged across
+  all examples in the dataset
+- BLEU-4: this is the [BLEU score](https://en.wikipedia.org/wiki/BLEU) between
+  the predicted response and the reference response. It is measured on
+  tokenized text, and uses NLTK to compute it.
+- F1: This is the [Unigram](https://en.wikipedia.org/wiki/N-gram) F1 overlap
+  between your text and the reference response.
+- exs: the number of examples we have evaluated
+
+If you don't see the BLEU-4 score, you may need to install NLTK with
+`pip install nltk`.
+
+We can also measure ROUGE. Note that we need to `pip install py-rouge` for this
+functionality:
+
+```
+$ parlai eval_model -m fixed_response -t dailydialog --fixed-response "how may i help you ?" --metrics rouge
+14:47:24 | creating task(s): dailydialog
+14:47:31 | Finished evaluating tasks ['dailydialog'] using datatype valid
+    accuracy  exs    f1  rouge_1  rouge_2  rouge_L
+    .0001239 8069 .1163   .09887  .007285   .09525
+```
+
+### Agent-specific metrics
+
+Some agents include their own metrics that are computed for them. For example,
+generative models automatically compute `ppl`
+([perplexity](https://en.wikipedia.org/wiki/Perplexity)) and `token_acc`, both
+which measure the generative model's ability to predict indivdual tokens.  As
+an example, let's evaluate the [BlenderBot](https://parl.ai/projects/recipes/)
+90M model on DailyDialog:
+
+```
+$ parlai eval_model --task dailydialog -mf zoo:blender/blender_90M/model -bs 32
+...
+14:54:14 | Evaluating task dailydialog using datatype valid.
+14:54:14 | creating task(s): dailydialog
+...
+15:26:19 | Finished evaluating tasks ['dailydialog'] using datatype valid
+    accuracy  bleu-4  ctpb  ctps  exps  exs    f1  gpu_mem  loss      lr  ltpb  ltps   ppl  token_acc  total_train_updates   tpb   tps
+           0 .002097 14202 442.5 6.446 8069 .1345    .0384 2.979 7.5e-06  3242   101 19.67      .4133               339012 17445 543.5
+```
+
+Here we see a number of extra metrics, each of which we explain below:
+- `tpb`, `ctpb`, `ltpb`: stand for tokens per batch, context-tokens per batch,
+  and label-tokens per batch. These are useful for measuring how dense the
+  batches are, and are helpful when experimenting with [dynamic
+  batching](tutorial_fast).  tpb is always the sum of ctpb and lptb.
+- `tps`, `ctps`, `ltps`: are similar, but stand for "tokens per second". They
+  measure how fast we are training. Similarly, `exps` measures examples per
+  second.
+- `gpu_mem`: measures _roughly_ how much GPU memory your model is using, but it
+  is only approximate. This is useful for determining if you can possibly increase
+  the model size or the batch size.
+- `loss`: the loss metric
+- `ppl` and `token_acc`: the perplexity and per-token accuracy. these are generative
+  performance metrics.
+- `total_train_updates`: the number of SGD updates this model was trained for.
+  You will see this increase during training, but not during evaluation.
+
+## Adding custom metrics
+
+Of course, you may wish to add your own custom metrics: whether this is because
+you are developing a special model, special dataset, or otherwise want other
+information accessible to you. Metrics can be computed by either _the teacher_ OR
+_the model_. Within the model, they may be computed either _locally_ or _globally_.
+There are different reasons for why and where you would want to choose each
+location:
+
+- __Teacher metrics__: This is the best spot for computing metrics that depend
+  on a specific dataset. These metrics will only be available when evaluating
+  on this dataset. They have the advantage of being easy to compute and
+  understand.
+- __Global metrics__: Global metrics are computed by the model, and are globally
+  tracked. These metrics are easy to understand and track, but work poorly
+  when doing multitasking.
+- __Local metrics__: Local metrics are the model-analogue of teacher metrics.
+  They are computed and recorded on a per-example basis, and so they work well
+  when multitasking. They can be extremely complicated for some models, however.
+
+We will take you through writing each of these methods in turn, and demonstrate
+examples of how to add these metrics in your setup.
+
+## Teacher metrics
+
+Teacher metrics are useful for items that depend on a specific dataset.
+For example, in some of our task oriented datasets, like
+[`google_sgd`](https://github.com/facebookresearch/ParlAI/blob/master/parlai/tasks/google_sgd/agents.py),
+we want to additionally compute metrics around slots.
+
+Teacher metrics can be added by adding the following method to your teacher:
+
+```python
+    def custom_evaluation(
+        self,
+        teacher_action: Message,
+        labels: Optional[Tuple[str]],
+        model_response: Message,
+    ) -> None:
+        pass
+```
+
+The signature for this method is as follows:
+- `teacher_action`: this is the last message the teacher sent to the model. This likely
+   contains a "text" and "labels" field, as well as any custom fields you might
+   have.
+- `labels`: The gold label(s). This can also be found as information in the
+  `teacher_action`, but it is conveniently extracted for you.
+- `model_response`: The full model response, including any extra fields the model
+   may have sent.
+
+Let's take an actual example. We will add a custom metric which calculates
+how often the model says the word "hello", and call it `hello_avg`.
+
+We will add a [custom teacher](tutorial_task). For this example, we will use
+the `@register` syntax you may have seen in our [quickstart
+tutorial](tutorial_quick).
+
+```python
+from parlai.core.loader import register_teacher
+from parlai.core.metrics import AverageMetric
+from parlai.tasks.dailydialog.agents import DefaultTeacher as DailyDialogTeacher
+
+@register_teacher("hello_daily")
+class CustomDailyDialogTeacher(DailyDialogTeacher):
+    def custom_evaluation(
+        self, teacher_action, labels, model_response
+    ) -> None:
+        if 'text' not in model_response:
+            # model didn't speak, skip this example
+            return
+        model_text = model_response['text']
+        if 'hello' in model_text:
+            # count 1 / 1 messages having "hello"
+            self.metrics.add('hello_avg', AverageMetric(1, 1))
+        else:
+            # count 0 / 1 messages having "hello"
+            self.metrics.add('hello_avg', AverageMetric(0, 1))
+
+if __name__ == '__main__':
+    from parlai.scripts.eval_model import EvalModel
+
+    EvalModel.main(
+       task='hello_daily',
+       model_file='zoo:blender/blender_90M/model',
+       batchsize=32,
+    )
+```
+
+If we run the script, we will have a new metric in our output:
+
+```
+18:07:30 | Finished evaluating tasks ['hello_daily'] using datatype valid
+    accuracy  bleu-4  ctpb  ctps  exps  exs    f1  gpu_mem  hello_avg  loss  ltpb  ltps   ppl  token_acc  tpb   tps
+           0 .002035  2172   230 3.351 8069 .1346   .05211      .1228 2.979 495.9 52.52 19.67      .4133 2668 282.6
+```
+
+__What is AverageMetric?__
+
+Wait, what is this
+[AverageMetric](parlai.core.metrics.AverageMetric)? All metrics
+you want to create in ParlAI should be a
+[Metric](Metric) object. Metric objects
+define a way of instantiating the metric, a way of combining it with a
+like-metric, and a way of rendering it as a single float value. For an
+AverageMetric, this means we need to define a numerator and a denominator; the
+combination of AverageMetrics adds their numerators and denominators
+separately. As we do this across all examples, the numerator will be the number
+of examples with "hello" in it, and the denominator will be the total number of
+examples. When we go to print the metric, the division will be computed at the
+last second.
+
+If you're used to writing machine learning code in one-off scripts, you may ask
+why do I need to use this metric? Can't I just count and divide myself? While
+you can do this, your code could not be run in [_distributed
+mode_](tutorial_fast). If we only returned a single float, we would not be able
+to know if some distributed workers received more or fewer examples than
+others. However, when we explicitly store the numerator and denominator, we can
+combine and reduce the across multiple nodes, enabling us to train on hundreds
+of GPUs, while still ensuring correctness in all our metrics.
+
+In addition to AverageMetric, there is also
+[SumMetric](parlai.core.metrics.SumMetric), which keeps a running
+sum. SumMetric and AverageMetric are the most common ways to construct custom
+metrics, but others exist as well. For a full list (and views into advanced
+cases), please see the [metrics API documentation](metrics_api).
+
+## Agent (model) level metrics
+
+In the above example, we worked on a metric defined by a Teacher. However,
+sometimes our models will have special metrics that only they want to compute,
+which we call an Agent-level metric. Perplexity is one example.
+
+To compute model-level metrics, we can define either a global metric, or a
+local metric. Global metrics can be computed anywhere, and are easy to use,
+but cannot distinguish between different teachers when multitasking. We'll
+look at another example, counting the number of times the teacher says "hello".
+
+### Global metrics
+
+A global metric is computed anywhere in the model, and has an
+interface similar to that of the teacher:
+
+```python
+agent.global_metrics.add('my_metric', AverageMetric(1, 2))
+```
+
+Global metrics are called as such because they can be called anywhere in agent
+code. For example, we can add a metric that counts the number of times the
+model sees the word "hello" in `observe`. We'll do this while extending
+the `TransformerGeneratorAgent`, so that we can combined it with the BlenderBot
+model we used earlier.
+
+```python
+from parlai.core.metrics import AverageMetric
+from parlai.core.loader import register_agent
+from parlai.agents.transformer.transformer import TransformerGeneratorAgent
+
+
+@register_agent('GlobalHelloCounter')
+class GlobalHelloCounterAgent(TransformerGeneratorAgent):
+    def observe(self, observation):
+        retval = super().observe(observation)
+        if 'text' in observation:
+            text = observation['text']
+            self.global_metrics.add(
+                'global_hello', AverageMetric(int('hello' in text), 1)
+            )
+        return retval
+
+
+if __name__ == '__main__':
+    from parlai.scripts.eval_model import EvalModel
+
+    EvalModel.main(
+        task='dailydialog',
+        model='GlobalHelloCounter',
+        model_file='zoo:blender/blender_90M/model',
+        batchsize=32,
+    )
+```
+
+Running the script, we see that our new metric appears. Note that it is
+different than the metric in the first half of the tutorial: that is because
+previously we were counting the number of times the model said hello (a lot),
+but now we are counting how often the dataset says hello.
+
+```
+21:57:50 | Finished evaluating tasks ['dailydialog'] using datatype valid
+    accuracy  bleu-4  ctpb  ctps  exps  exs    f1  global_hello  gpu_mem  loss  ltpb  ltps   ppl  token_acc   tpb   tps
+           0 .002097 14202 435.1 6.338 8069 .1345      .0009914   .02795 2.979  3242 99.32 19.67      .4133 17445 534.4
+```
+
+The global metric works well, but have some drawbacks: if we were to start
+training on a multitask datasets, we would not be able to distinguish the
+`global_hello` of the two datasets, and we could only compute the micro-average
+of the combination of the two. Below is an excerpt from a training log with
+the above agents:
+
+```
+09:14:52 | time:112s total_exs:90180 epochs:0.41
+                clip  ctpb  ctps  exps  exs  global_hello  gnorm  gpu_mem  loss  lr  ltpb  ltps   ppl  token_acc  total_train_updates   tpb   tps   ups
+   all             1  9831 66874 841.9 8416        .01081  2.018    .3474 5.078   1  1746 11878 163.9      .2370                  729 11577 78752 6.803
+   convai2                             3434        .01081                 5.288                 197.9      .2120
+   dailydialog                         4982        .01081                 4.868                   130      .2620
+```
+
+Notice how `global_hello` is the same in both, because the model is unable to
+distinguish between the two settings. In the next section we'll show how to fix
+this with local metrics.
+
+__On placement__: In the example above, we recorded the global metric inside
+the `observe` function. However, global metrics can be recorded from anywhere.
+
+### Local metrics
+
+Having observed the limitation of global metrics being unable to distinguish
+settings in multitasking, we would like to improve upon this. Let's add a local
+metric, which is recorded _per example_. By recording this metric per example,
+we can unambiguously identify which metrics came from which dataset, and report
+averages correctly.
+
+Local metrics have a limitation: they can only be computed inside the scope of
+`batch_act`. This includes common places like `compute_loss` or `generate`,
+where we often want to instrument specific behavior.
+
+Let's look at an example. We'll add a metric inside the `batchify` function,
+which is called from within `batch_act`, and is used to convert from a list of
+[Messages](messages) objects to a
+[Batch](torch_agent.html#parlai.core.torch_agent.Batch) object. It is where we do things like
+padding, etc. We'll do something slightly different than our previous runs.
+In this case, we'll count the number of _tokens_ which are the word "hello".
+
+```python
+from parlai.core.metrics import AverageMetric
+from parlai.core.loader import register_agent
+from parlai.agents.transformer.transformer import TransformerGeneratorAgent
+
+
+@register_agent('LocalHelloCounter')
+class LocalHelloCounterAgent(TransformerGeneratorAgent):
+    def batchify(self, observations):
+        batch = super().batchify(observations)
+        if hasattr(batch, 'text_vec'):
+            num_hello = ["hello" in o['text'] for o in observations]
+            self.record_local_metric(
+                'local_hello',
+                # AverageMetric.many(seq1) is shorthand for
+                # [AverageMetric(item) for item in seq)
+                AverageMetric.many(num_hello),
+            )
+        return batch
+
+
+if __name__ == '__main__':
+    from parlai.scripts.train_model import TrainModel
+
+    TrainModel.main(
+        task='dailydialog,convai2',
+        model='LocalHelloCounter',
+        dict_file='zoo:blender/blender_90M/model.dict',
+        batchsize=32,
+    )
+```
+
+When we run this training script, we get one such output:
+```
+09:49:00 | time:101s total_exs:56160 epochs:0.26
+                clip  ctpb  ctps  exps  exs  gnorm  gpu_mem  local_hello  loss  lr  ltpb  ltps   ppl  token_acc  total_train_updates  tpb   tps  ups
+   all             1  3676 63204 550.2 5504  2.146    .1512       .01423 4.623   1 436.2  7500 101.8      .2757                 1755 4112 70704 17.2
+   convai2                             3652                       .02793 4.659                 105.5      .2651
+   dailydialog                         1852                       .00054 4.587                 98.17      .2863
+```
+
+Notice how the `local_hello` metric can now distinguish between hellos coming from
+convai2 and those coming from daily dialog? The average hides the fact that one
+dataset has many hellos, and the other does not.
+
+Local metrics are primarily worth the implementation when you care about the
+fidelity of _train time metrics_. During evaluation time, we evaluate each
+dataset individually, so we can ensure global metrics are not mixed up.
+
+__Under the hood__: Local metrics work by including a "metrics" field in the
+return message. This is a dictionary which maps field name to a metric value.
+When the teacher receives the response from the model, it utilizes the metrics
+field to update counters on its side.

From 948e669fb54adff7db2d68929cadf6766b8cfaf2 Mon Sep 17 00:00:00 2001
From: Stephen Roller <roller@fb.com>
Date: Wed, 12 Aug 2020 22:25:08 -0400
Subject: [PATCH 18/22] add index.

---
 docs/source/index.rst | 1 +
 1 file changed, 1 insertion(+)

diff --git a/docs/source/index.rst b/docs/source/index.rst
index ef20b2a562f..ea73bec4375 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -20,6 +20,7 @@ ParlAI is a one-stop-shop for dialog research.
    tutorial_worlds
    tutorial_torch_generator_agent
    tutorial_torch_ranker_agent
+   tutorial_metrics
    tutorial_fast
    tutorial_mturk
    tutorial_chat_service

From d90aa6084b8eae3a95671331faddc16122dd48fc Mon Sep 17 00:00:00 2001
From: Stephen Roller <roller@fb.com>
Date: Sat, 15 Aug 2020 09:45:26 -0400
Subject: [PATCH 19/22] Update links.

---
 docs/source/tutorial_metrics.md | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/docs/source/tutorial_metrics.md b/docs/source/tutorial_metrics.md
index 650ba4ceb7e..a33067d3237 100644
--- a/docs/source/tutorial_metrics.md
+++ b/docs/source/tutorial_metrics.md
@@ -186,7 +186,7 @@ __What is AverageMetric?__
 Wait, what is this
 [AverageMetric](parlai.core.metrics.AverageMetric)? All metrics
 you want to create in ParlAI should be a
-[Metric](Metric) object. Metric objects
+[Metric](parlai.core.metrics.Metric) object. Metric objects
 define a way of instantiating the metric, a way of combining it with a
 like-metric, and a way of rendering it as a single float value. For an
 AverageMetric, this means we need to define a numerator and a denominator; the
@@ -209,7 +209,7 @@ In addition to AverageMetric, there is also
 [SumMetric](parlai.core.metrics.SumMetric), which keeps a running
 sum. SumMetric and AverageMetric are the most common ways to construct custom
 metrics, but others exist as well. For a full list (and views into advanced
-cases), please see the [metrics API documentation](metrics_api).
+cases), please see the [metrics API documentation](core/metrics_api).
 
 ## Agent (model) level metrics
 
@@ -312,8 +312,8 @@ where we often want to instrument specific behavior.
 
 Let's look at an example. We'll add a metric inside the `batchify` function,
 which is called from within `batch_act`, and is used to convert from a list of
-[Messages](messages) objects to a
-[Batch](torch_agent.html#parlai.core.torch_agent.Batch) object. It is where we do things like
+[Messages](core/messages) objects to a
+[Batch](parlai.core.torch_agent.Batch) object. It is where we do things like
 padding, etc. We'll do something slightly different than our previous runs.
 In this case, we'll count the number of _tokens_ which are the word "hello".
 

From 52116e16300d6849768eda5c7800606d78edd6bd Mon Sep 17 00:00:00 2001
From: Stephen Roller <roller@fb.com>
Date: Sun, 23 Aug 2020 12:43:53 -0400
Subject: [PATCH 20/22] Address some of Mojtaba's comments.

---
 docs/source/tutorial_metrics.md | 49 ++++++++++++++++++++++-----------
 1 file changed, 33 insertions(+), 16 deletions(-)

diff --git a/docs/source/tutorial_metrics.md b/docs/source/tutorial_metrics.md
index a33067d3237..316da08da2b 100644
--- a/docs/source/tutorial_metrics.md
+++ b/docs/source/tutorial_metrics.md
@@ -36,8 +36,8 @@ We see that we got 0.01239% accuracy, 0.26% BLEU-4 score, and 11.63% F1 across
 If you don't see the BLEU-4 score, you may need to install NLTK with
 `pip install nltk`.
 
-We can also measure ROUGE. Note that we need to `pip install py-rouge` for this
-functionality:
+We can also measure [ROUGE](https://en.wikipedia.org/wiki/ROUGE_%28metric%29). Note
+that we need to `pip install py-rouge` for this functionality:
 
 ```
 $ parlai eval_model -m fixed_response -t dailydialog --fixed-response "how may i help you ?" --metrics rouge
@@ -47,6 +47,12 @@ $ parlai eval_model -m fixed_response -t dailydialog --fixed-response "how may i
     .0001239 8069 .1163   .09887  .007285   .09525
 ```
 
+One nice thing about metrics is that they are automatically logged to the
+`.trainstats` file, and within Tensorboard (when enabled with
+`--tensorboard-log true`. As such, metrics are more reliable than adding print
+statements into your code.
+
+
 ### Agent-specific metrics
 
 Some agents include their own metrics that are computed for them. For example,
@@ -67,7 +73,13 @@ $ parlai eval_model --task dailydialog -mf zoo:blender/blender_90M/model -bs 32
            0 .002097 14202 442.5 6.446 8069 .1345    .0384 2.979 7.5e-06  3242   101 19.67      .4133               339012 17445 543.5
 ```
 
-Here we see a number of extra metrics, each of which we explain below:
+Here we see a number of extra metrics, each of which we explain below. They may be
+roughly divided into diagnostic/performance metrics, and modeling metrics. The
+modeling metrics are:
+- `ppl` and `token_acc`: the perplexity and per-token accuracy. these are generative
+  performance metrics.
+
+The diagnostic metrics are:
 - `tpb`, `ctpb`, `ltpb`: stand for tokens per batch, context-tokens per batch,
   and label-tokens per batch. These are useful for measuring how dense the
   batches are, and are helpful when experimenting with [dynamic
@@ -79,8 +91,6 @@ Here we see a number of extra metrics, each of which we explain below:
   is only approximate. This is useful for determining if you can possibly increase
   the model size or the batch size.
 - `loss`: the loss metric
-- `ppl` and `token_acc`: the perplexity and per-token accuracy. these are generative
-  performance metrics.
 - `total_train_updates`: the number of SGD updates this model was trained for.
   You will see this increase during training, but not during evaluation.
 
@@ -96,13 +106,20 @@ location:
 - __Teacher metrics__: This is the best spot for computing metrics that depend
   on a specific dataset. These metrics will only be available when evaluating
   on this dataset. They have the advantage of being easy to compute and
-  understand.
-- __Global metrics__: Global metrics are computed by the model, and are globally
-  tracked. These metrics are easy to understand and track, but work poorly
-  when doing multitasking.
-- __Local metrics__: Local metrics are the model-analogue of teacher metrics.
-  They are computed and recorded on a per-example basis, and so they work well
-  when multitasking. They can be extremely complicated for some models, however.
+  understand. An example of a modeling metric is `slot_p`, which is part of
+  some of our Task Oriented Datasets, such as
+  [`google_sgd`](https://github.com/facebookresearch/ParlAI/blob/master/parlai/tasks/google_sgd/agents.py)
+- __Global metrics__ (model metric): Global metrics are computed by the model,
+  and are globally tracked. These metrics are easy to understand and track, but
+  work poorly when doing multitasking. One example of a global metric includes
+  `gpu_mem`, which depends on a system-wide memory usage, and cannot be tied to
+  a specific task.
+- __Local metrics__ (model metric): Local metrics are the model-analogue of
+  teacher metrics.  They are computed and recorded on a per-example basis, and
+  so they work well when multitasking. They can be extremely complicated for
+  some models, however. An example of a local metric includes perplexity, which
+  should be computed on a per-example basis, but must be computed by the model,
+  and therefore cannot be a teacher metric.
 
 We will take you through writing each of these methods in turn, and demonstrate
 examples of how to add these metrics in your setup.
@@ -217,10 +234,10 @@ In the above example, we worked on a metric defined by a Teacher. However,
 sometimes our models will have special metrics that only they want to compute,
 which we call an Agent-level metric. Perplexity is one example.
 
-To compute model-level metrics, we can define either a global metric, or a
-local metric. Global metrics can be computed anywhere, and are easy to use,
-but cannot distinguish between different teachers when multitasking. We'll
-look at another example, counting the number of times the teacher says "hello".
+To compute model-level metrics, we can define either a _Global_ metric, or a
+_Local metric_. Global metrics can be computed anywhere, and are easy to use,
+but cannot distinguish between different teachers when multitasking. We'll look
+at another example, counting the number of times the teacher says "hello".
 
 ### Global metrics
 

From aa45ec62933b7af464487ac0811a2fa671a55d72 Mon Sep 17 00:00:00 2001
From: Stephen Roller <roller@fb.com>
Date: Sun, 23 Aug 2020 13:32:29 -0400
Subject: [PATCH 21/22] Emphasize the difference between the locations of
 metrics.

---
 docs/source/conf.py             |  6 ++++++
 docs/source/tutorial_metrics.md | 23 ++++++++++++++++++-----
 2 files changed, 24 insertions(+), 5 deletions(-)

diff --git a/docs/source/conf.py b/docs/source/conf.py
index e122bf74ad4..ea2ec93b548 100644
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -123,6 +123,12 @@
 html_css_files = ['css/parlai_theme.css']
 html_logo = '_static/img/parlai.png'
 
+# myst options
+myst_admonition_enable = True
+myst_amsmath_enable = True
+myst_html_img_enable = True
+myst_dmath_enable = True
+
 
 # -- Options for HTMLHelp output ------------------------------------------
 
diff --git a/docs/source/tutorial_metrics.md b/docs/source/tutorial_metrics.md
index 316da08da2b..2bf4b5c599e 100644
--- a/docs/source/tutorial_metrics.md
+++ b/docs/source/tutorial_metrics.md
@@ -76,10 +76,12 @@ $ parlai eval_model --task dailydialog -mf zoo:blender/blender_90M/model -bs 32
 Here we see a number of extra metrics, each of which we explain below. They may be
 roughly divided into diagnostic/performance metrics, and modeling metrics. The
 modeling metrics are:
+
 - `ppl` and `token_acc`: the perplexity and per-token accuracy. these are generative
   performance metrics.
 
 The diagnostic metrics are:
+
 - `tpb`, `ctpb`, `ltpb`: stand for tokens per batch, context-tokens per batch,
   and label-tokens per batch. These are useful for measuring how dense the
   batches are, and are helpful when experimenting with [dynamic
@@ -153,7 +155,7 @@ The signature for this method is as follows:
    may have sent.
 
 Let's take an actual example. We will add a custom metric which calculates
-how often the model says the word "hello", and call it `hello_avg`.
+how often the __model__ says the word "hello", and call it `hello_avg`.
 
 We will add a [custom teacher](tutorial_task). For this example, we will use
 the `@register` syntax you may have seen in our [quickstart
@@ -283,10 +285,21 @@ if __name__ == '__main__':
     )
 ```
 
-Running the script, we see that our new metric appears. Note that it is
-different than the metric in the first half of the tutorial: that is because
-previously we were counting the number of times the model said hello (a lot),
-but now we are counting how often the dataset says hello.
+__Note that this is very different than the Teacher metric we implemented in the
+first half of the tutorial__. In the teacher metric, we were counting the number
+of times the _model_ said hello. Here, we are counting the number of times the
+_teacher_ said hello.
+
+:::{admonition,tip} How to determine where to implement your custom metric:
+
+- If you want your metric to be _model_-agnostic, then it should be implemented
+  in the Teacher.
+- If you want your metric to be _dataset_-agnostic, then it should be
+  implemented in the Model agent.
+:::
+
+Running the script, we see that our new metric appears. As discussed above, the
+value differs slightly because of the difference in semantics.
 
 ```
 21:57:50 | Finished evaluating tasks ['dailydialog'] using datatype valid

From 45b767db9e04347997683aa1ba09491eb2185655 Mon Sep 17 00:00:00 2001
From: Stephen Roller <roller@fb.com>
Date: Sun, 23 Aug 2020 14:38:47 -0400
Subject: [PATCH 22/22] Jing's comments.

---
 docs/source/tutorial_metrics.md | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/docs/source/tutorial_metrics.md b/docs/source/tutorial_metrics.md
index 2bf4b5c599e..b26efd42b2a 100644
--- a/docs/source/tutorial_metrics.md
+++ b/docs/source/tutorial_metrics.md
@@ -296,6 +296,10 @@ _teacher_ said hello.
   in the Teacher.
 - If you want your metric to be _dataset_-agnostic, then it should be
   implemented in the Model agent.
+- If you need your metric to be both model and dataset agnostic, then you
+  should do it within the Model, using a
+  [mixin](https://www.residentmar.io/2019/07/07/python-mixins.html) or abstract
+  class.
 :::
 
 Running the script, we see that our new metric appears. As discussed above, the
@@ -361,8 +365,6 @@ class LocalHelloCounterAgent(TransformerGeneratorAgent):
             num_hello = ["hello" in o['text'] for o in observations]
             self.record_local_metric(
                 'local_hello',
-                # AverageMetric.many(seq1) is shorthand for
-                # [AverageMetric(item) for item in seq)
                 AverageMetric.many(num_hello),
             )
         return batch