Skip to content

Commit 4f6db1a

Browse files
authored
documentation: minor update to adding_a_new_task.md
+ Added a link to the following URL. It points to a JSON file hosted by Hugging Face with the 21K ImageNet labels that were used to train Detic. https://huggingface.co/datasets/huggingface/label-files/raw/main/imagenet-22k-id2label.json + Edited the text to clarify the relationship between the objects detected by Stretch AI and these 21K labels. For example, "person" does not appear to be directly listed as one of the 21K classes. + Removed "several" from describing the number of new operations created for the simple handover task, since arguably only two wholly new operation types were created and the rest of the operations were modified or configured versions of existing operations. + Fixed a typo in a statement about the task tuples.
1 parent b15cb77 commit 4f6db1a

File tree

1 file changed

+3
-3
lines changed

1 file changed

+3
-3
lines changed

docs/adding_a_new_task.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -90,7 +90,7 @@ The NavigateToObjectOperation navigates to the current target object, which is s
9090

9191
Notably, many of the steps in the pick and place demo use the "feature" match method to find instances. The "feature" match method uses embeddings to match text to images of the instance. We did not have success using the "feature" match method to find a person instance, so we instead used the original class label assigned by the underlying object detector that results in the creation of instances.
9292

93-
**By default, the robot only creates instances for object categories listed in [example_cat_map.json](../src/stretch/config/example_cat_map.json) and detected by [Detic](https://github.com/facebookresearch/Detic).** Detic can attempt to detect the twenty-one-thousand classes used in the [ImageNet](https://en.wikipedia.org/wiki/ImageNet) database. By default, Stretch AI only uses a small subset of these classes (e.g., 108 classes when the handover task was originally created).
93+
**By default, the robot only creates instances for object categories listed in [example_cat_map.json](../src/stretch/config/example_cat_map.json) and detected by [Detic](https://github.com/facebookresearch/Detic).** Detic was trained with the [twenty-one-thousand classes](https://huggingface.co/datasets/huggingface/label-files/raw/main/imagenet-22k-id2label.json) used in the [ImageNet](https://en.wikipedia.org/wiki/ImageNet) database. By default, Stretch AI uses a relatively small number of classes (e.g., 108 classes when the handover task was originally created).
9494

9595
**To detect people for the handover task, we [added the "person" category](https://github.com/hello-robot/stretch_ai/blob/4be200f8ffa908bfe23b55139a8341916c4342f4/src/stretch/config/example_cat_map.json#L111) to example_cat_map.json.**
9696

@@ -113,7 +113,7 @@ found_a_person.configure(
113113

114114
The available operations can be found in the [/src/stretch/agent/operations](../src/stretch/agent/operations) directory. We recommend that you use existing operations when possible.
115115

116-
For the handover task, we defined several new operations to perform the actual handover. For example, the [extend_arm.py](../src/stretch/agent/operations/extend_arm.py) operation, found in the [/src/stretch/agent/operations](../src/stretch/agent/operations) directory, extends the arm during a handover.
116+
For the handover task, we defined new operations to perform the actual handover. For example, the [extend_arm.py](../src/stretch/agent/operations/extend_arm.py) operation, found in the [/src/stretch/agent/operations](../src/stretch/agent/operations) directory, extends the arm during a handover.
117117

118118
Like other operations, the ExtendArm operation class has a configure method and a run method. The configure method sets relevant parameters for the operation ahead of time. The task calls the run method to execute the operation.
119119

@@ -141,7 +141,7 @@ Some tasks called by the executor require an argument provided by the LLM. For e
141141

142142
## Modify the LLM Prompt
143143

144-
With Stretch AI's [pick-and-place demo](https://github.com/hello-robot/stretch_ai/blob/main/docs/llm_agent.md), you can provide a natural language request to an LLM and the LLM will output text that specifies the tasks that the robot should perform. The LLM's text output is then parsed into a list of tuples of tasks identifiers with task arguments. This list then goes to the executor described in the previous section, which processes the list of tuples and executes the tasks.
144+
With Stretch AI's [pick-and-place demo](https://github.com/hello-robot/stretch_ai/blob/main/docs/llm_agent.md), you can provide a natural language request to an LLM and the LLM will output text that specifies the tasks that the robot should perform. The LLM's text output is then parsed into a list of tuples with task identifiers and task arguments. This list then goes to the executor described in the previous section, which processes the list of tuples and executes the tasks.
145145

146146
### Tell the LLM How to Use Your Task
147147

0 commit comments

Comments
 (0)