Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add documentation step to add task guide #1154

Merged
merged 3 commits into from
Oct 19, 2020
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions guides/tasks/adding_tasks.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,12 @@ In this tutorial we’ll show you how to add a new task to `jiant`.

We’ll use a real task for this example: Senteval’s Tense task. Senteval’s tense task is a single sentence classification task with labels “PAST” or “PRES”, and it uses accuracy as its evaluation metric.

Adding Senteval’s Tense task will require touching three files:
Adding Senteval’s Tense task will require touching three files:
1. A (new) task file: `jiant/tasks/lib/senteval/tense.py`
2. The task retrieval library: `jiant/tasks/retrieval.py`
3. The evaluation library: `jiant/tasks/evaluate/core.py`

We’ll talk about these files in the following sections.
We’ll talk about these files in the following sections.

## 1. The Task file

Expand All @@ -21,7 +21,7 @@ Tasks files have 5 core components:
4. `DataRow` dataclass
5. `Batch` dataclass

In the following sections we’ll explain these components (and build up a working example in code blocks).
In the following sections we’ll explain these components (and build up a working example in code blocks).

#### 1. `Task` class

Expand Down Expand Up @@ -130,7 +130,7 @@ class Batch(BatchMixin):
segment_ids: torch.LongTensor
label_id: torch.LongTensor
tokens: list
```
```

Now that we've implemented required components of our SentevalTenseTask, we're ready to add it to the task retrieval library.

Expand Down Expand Up @@ -162,6 +162,10 @@ def get_evaluation_scheme_for_task(task) -> BaseEvaluationScheme:
return SimpleAccuracyEvaluationScheme()
```

## 4. Update the supported tasks documentation
Add your task to [`guides/tasks/supported_tasks.md`](../../guides/tasks/supported_tasks.md).

## Congratulations!
And that’s it. You’ve made all the core code changes required to include the `SentevalTenseTask` in your `jiant` experiments.

What's next? To tokenize and cache your `SentevalTenseTask` (which you shortnamed `senteval_tense`) for an experiement, you'll need to provide a task config `json` file pointing to your task's data:
Expand Down
1 change: 1 addition & 0 deletions guides/tasks/supported_tasks.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

| Name | `task_name` | `jiant` | Downloader | `jiant_task_name` | Misc |
|---|---|:---:|:---:|---|---|
| [Argument Reasoning Comprehension](https://arxiv.org/abs/1708.01425) | arct | ✅ | | arct | [Github](https://github.com/UKPLab/argument-reasoning-comprehension-task) |
| Abductive NLI | abductive_nli | ✅ | ✅ | abductive_nli | |
| SuperGLUE Winogender Diagnostic | superglue_axg | ✅ | ✅ | superglue_axg | SuperGLUE |
| Acceptability Definiteness | acceptability_definiteness | ✅ | | acceptability_definiteness | Function Words |
Expand Down