You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
documentation: minor update to adding_a_new_task.md
+ Added a link to the following URL. It points to a JSON file hosted by Hugging Face with the 21K ImageNet labels that were used to train Detic.
https://huggingface.co/datasets/huggingface/label-files/raw/main/imagenet-22k-id2label.json
+ Edited the text to clarify the relationship between the objects detected by Stretch AI and these 21K labels. For example, "person" does not appear to be directly listed as one of the 21K classes.
+ Removed "several" from describing the number of new operations created for the simple handover task, since arguably only two wholly new operation types were created and the rest of the operations were modified or configured versions of existing operations.
+ Fixed a typo in a statement about the task tuples.
Copy file name to clipboardExpand all lines: docs/adding_a_new_task.md
+3-3Lines changed: 3 additions & 3 deletions
Original file line number
Diff line number
Diff line change
@@ -90,7 +90,7 @@ The NavigateToObjectOperation navigates to the current target object, which is s
90
90
91
91
Notably, many of the steps in the pick and place demo use the "feature" match method to find instances. The "feature" match method uses embeddings to match text to images of the instance. We did not have success using the "feature" match method to find a person instance, so we instead used the original class label assigned by the underlying object detector that results in the creation of instances.
92
92
93
-
**By default, the robot only creates instances for object categories listed in [example_cat_map.json](../src/stretch/config/example_cat_map.json) and detected by [Detic](https://github.com/facebookresearch/Detic).** Detic can attempt to detect the twenty-one-thousand classes used in the [ImageNet](https://en.wikipedia.org/wiki/ImageNet) database. By default, Stretch AI only uses a small subset of these classes (e.g., 108 classes when the handover task was originally created).
93
+
**By default, the robot only creates instances for object categories listed in [example_cat_map.json](../src/stretch/config/example_cat_map.json) and detected by [Detic](https://github.com/facebookresearch/Detic).** Detic was trained with the [twenty-one-thousand classes](https://huggingface.co/datasets/huggingface/label-files/raw/main/imagenet-22k-id2label.json) used in the [ImageNet](https://en.wikipedia.org/wiki/ImageNet) database. By default, Stretch AI uses a relatively small number of classes (e.g., 108 classes when the handover task was originally created).
94
94
95
95
**To detect people for the handover task, we [added the "person" category](https://github.com/hello-robot/stretch_ai/blob/4be200f8ffa908bfe23b55139a8341916c4342f4/src/stretch/config/example_cat_map.json#L111) to example_cat_map.json.**
96
96
@@ -113,7 +113,7 @@ found_a_person.configure(
113
113
114
114
The available operations can be found in the [/src/stretch/agent/operations](../src/stretch/agent/operations) directory. We recommend that you use existing operations when possible.
115
115
116
-
For the handover task, we defined several new operations to perform the actual handover. For example, the [extend_arm.py](../src/stretch/agent/operations/extend_arm.py) operation, found in the [/src/stretch/agent/operations](../src/stretch/agent/operations) directory, extends the arm during a handover.
116
+
For the handover task, we defined new operations to perform the actual handover. For example, the [extend_arm.py](../src/stretch/agent/operations/extend_arm.py) operation, found in the [/src/stretch/agent/operations](../src/stretch/agent/operations) directory, extends the arm during a handover.
117
117
118
118
Like other operations, the ExtendArm operation class has a configure method and a run method. The configure method sets relevant parameters for the operation ahead of time. The task calls the run method to execute the operation.
119
119
@@ -141,7 +141,7 @@ Some tasks called by the executor require an argument provided by the LLM. For e
141
141
142
142
## Modify the LLM Prompt
143
143
144
-
With Stretch AI's [pick-and-place demo](https://github.com/hello-robot/stretch_ai/blob/main/docs/llm_agent.md), you can provide a natural language request to an LLM and the LLM will output text that specifies the tasks that the robot should perform. The LLM's text output is then parsed into a list of tuples of tasks identifiers with task arguments. This list then goes to the executor described in the previous section, which processes the list of tuples and executes the tasks.
144
+
With Stretch AI's [pick-and-place demo](https://github.com/hello-robot/stretch_ai/blob/main/docs/llm_agent.md), you can provide a natural language request to an LLM and the LLM will output text that specifies the tasks that the robot should perform. The LLM's text output is then parsed into a list of tuples with task identifiers and task arguments. This list then goes to the executor described in the previous section, which processes the list of tuples and executes the tasks.
0 commit comments