Skip to content
This repository has been archived by the owner on Nov 3, 2023. It is now read-only.

Commit

Permalink
updated structure
Browse files Browse the repository at this point in the history
  • Loading branch information
liliarose committed Aug 5, 2021
1 parent 6ea08d4 commit 578f8a8
Showing 1 changed file with 27 additions and 26 deletions.
53 changes: 27 additions & 26 deletions docs/source/tutorial_model_cards.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,18 @@
# Generating Model Cards Semi-Automatically

**Author**: Wendy
**Author**: Wendy Zhang

Here's a sample model card for the [Blenderbot2 3B](#TODO).
Here's a sample model card for the [Blenderbot2_3B](https://github.com/facebookresearch/ParlAI/blob/master/parlai/zoo/blenderbot2/model_card.md).

## The Process
There are two steps in generating the model cards.
![imageonline-co-whitebackgroundremoved (3)](https://user-images.githubusercontent.com/14303605/128065136-9403281c-3124-488e-be1d-81b9262b7758.png)

For both steps, we should specify the following arguments:
- `--model-file / -mf`: the model file
- `--folder-to-save / -fts`: the location where we're saving reports

## Step 1: Generating reports
### Step 1: Generating reports
In general, we can use a command like this for report generation:
```
# template
Expand All @@ -22,15 +23,15 @@ parlai gmc -mf zoo:dialogue_safety/multi_turn/model -fts safety_single --mode ge

However, depending on the situiaton, we might need to add these arguments as well:
- `--wrapper / -w` **only if** the model is a generation model
- check the [safety bench](https://github.com/facebookresearch/ParlAI/tree/master/projects/safety_bench) for more info about the the wrappers and its implementation
- check the [safety bench](https://github.com/facebookresearch/ParlAI/tree/master/projects/safety_bench) for more info
- `--model-type / -mt` **only if** the model isn't added to or already in [`model_list.py`](https://github.com/facebookresearch/ParlAI/blob/master/parlai/zoo/model_list.py)
- possible choices include `ranker`, `generator`, `classifier`, `retriever`

In addition, if the model itself needs certain arguments (ie. `--search-server`), we should specify them at this stage too. We can also add `--batchsize` for faster generation.

Check out the section about [generating reports](#report-generation-process) explanations of the report generation process and how to generate single reports.
Check out the section about [generating reports](#details-of-report-generation) for more information on the report generation process and how to generate single reports.

## Step 2: Model Card Generation
### Step 2: Model Card Generation
If some kind of model description has already been added to the [model_list.py](https://github.com/facebookresearch/ParlAI/blob/master/parlai/zoo/model_list.py) (distinguished by `path`, which should be the same as `model_file`), and reports were sucessfully generated in the step before, then we can simply run the following command
```
# template
Expand All @@ -39,7 +40,7 @@ If some kind of model description has already been added to the [model_list.py](
parlai gmc -mf zoo:dialogue_safety/multi_turn/model -fts safety_multi
```

## Examples
### Examples
Here are some samples commands (click to see the results):
- Dialogue Safety (multi-turn)
```
Expand All @@ -53,30 +54,30 @@ parlai gmc -mf zoo:blender/blender_90M/model -fts blenderbot_90M
```
- Blenderbot2 400M
```
parlai gmc -mf zoo:blenderbot2/blenderbot2_400M/model -fts bb2_440M -bs 128 --mode gen:safety --search-server http://devfair0169:5000/bing_search --wrapper blenderbot2_400M
parlai gmc -mf zoo:blenderbot2/blenderbot2_400M/model -fts bb2_440M
parlai gmc -mf zoo:blenderbot2/blenderbot2_400M/model -fts bb2_400M -bs 128 --mode gen:safety --search-server http://devfair0169:5000/bing_search --wrapper blenderbot2_400M
parlai gmc -mf zoo:blenderbot2/blenderbot2_400M/model -fts bb2_400M
```


## Report Generation Process
## Details of Report Generation
![imageonline-co-whitebackgroundremoved (4)](https://user-images.githubusercontent.com/14303605/128233882-4c77770d-9703-466f-b1a2-7f2395c5c2f6.png)

**Successful generations should end with a green message like this:**
In the end, it should generate the following reports under the `--folder-to-save`
- a folder `data_stats/` that contains the data stats of the training set
- a `eval_results.json` that contains the evaluation results based on the evaltasks
- a `sample.json` file contain a sample input and output from the model
- for generators, it should generate a folder `safety_bench_res` that contains the safety_bench results ([click here to learn more about the safety bench](https://github.com/facebookresearch/ParlAI/tree/master/projects/safety_bench)).

<img width="679" alt="Screen Shot 2021-07-26 at 3 58 33 PM" src="https://user-images.githubusercontent.com/14303605/127069754-b99cec8c-6fac-4d32-bbca-f4972f6c5b5e.png">
**Successful generations should end with a green message like this:**

**Unsucessful generations will look like this, and should tell us which reports are missing and why.**
<img width="1790" alt="Screen Shot 2021-07-26 at 11 32 17 AM" src="https://user-images.githubusercontent.com/14303605/127040345-e8ec6afa-60da-484e-8e68-955f592cec8b.png">
<img width="679" alt="Screen Shot 2021-07-26 at 3 58 33 PM" src="https://user-images.githubusercontent.com/14303605/127069754-b99cec8c-6fac-4d32-bbca-f4972f6c5b5e.png">

- In the end, it should generate the following reports under the `--folder-to-save`
- a folder `data_stats/` that contains the data stats of the training set
- a `eval_results.json` that contains the evaluation results based on the evaltasks
- a `sample.json` file contain a sample input and output from the model
- for generators, it should generate a folder `safety_bench_res` that contains the safety_bench results ([click here to learn more about the safety bench](https://github.com/facebookresearch/ParlAI/tree/master/projects/safety_bench)).
**Unsucessful generations will look like this, and should tell us which reports are missing and why.**
<img width="1790" alt="Screen Shot 2021-07-26 at 11 32 17 AM" src="https://user-images.githubusercontent.com/14303605/127040345-e8ec6afa-60da-484e-8e68-955f592cec8b.png">

![imageonline-co-whitebackgroundremoved (4)](https://user-images.githubusercontent.com/14303605/128233882-4c77770d-9703-466f-b1a2-7f2395c5c2f6.png)

## Generating single reports
Sometimes, you might want to generate only certain reports. In this case, instead of using `--mode gen`, we should use `--mode gen:<report>`. Here are the possibilites:
### Generating single reports
Sometimes, you might want to generate only certain reports. In this case, instead of using `--mode gen`, we should use following possibilites:
- `--mode gen:data_stats` to generate the `data_stats/` folder
- `--mode gen:eval` to generate the `eval_results.json` file (evaluation results)
- `--mode gen:safety` to generate the `safety_bench_res` folder
Expand All @@ -96,11 +97,11 @@ Sometimes, you might want to generate only certain reports. In this case, instea



## Using `extra-args-path`
### Using `extra-args-path`

We can use `extra-args-path` to pass in longer arguments:
We can use `extra-args-path` to pass in longer arguments. By default, `extra-args-path` will be `<folder-to-save>/args.json`.

### Adding Custom Dataset and Model Info
#### Adding Custom Dataset and Model Info
By default, the code will try to find a sections in [`model_list.py`](https://github.com/facebookresearch/ParlAI/blob/master/parlai/zoo/model_list.py). However, instead of changing `model_list.py`, we can also pass in a `.json` file to `--extra-args-path` with out new section. Here's us trying to add the intended use section

```
Expand Down Expand Up @@ -130,7 +131,7 @@ Similarly, if we don't want to touch [`task_list.py`](https://github.com/faceboo
The information passed via this method can partially overwrite what's written in `task_list.py` and `model_list.py`.


### Add Custom Sections or Changing Section Order (static)
#### Add Custom Sections or Changing Section Order (static)

For static sections, there's two ways to do this.

Expand Down

0 comments on commit 578f8a8

Please sign in to comment.