Skip to content

Commit

Permalink
Update workflow template documentation (#6883)
Browse files Browse the repository at this point in the history
* Update workflow template documentation

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Tech review comments

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Explain custom and os-provided templates

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Add link to supported templates

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Update _automating-configurations/workflow-templates.md

Co-authored-by: Daniel Widdis <widdis@gmail.com>
Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>

---------

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>
Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>
Co-authored-by: Daniel Widdis <widdis@gmail.com>
(cherry picked from commit eb55593)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
  • Loading branch information
github-actions[bot] and dbwiddis committed Apr 4, 2024
1 parent a598dfa commit 13f1902
Show file tree
Hide file tree
Showing 2 changed files with 24 additions and 29 deletions.
2 changes: 1 addition & 1 deletion _automating-configurations/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ You can automate complex OpenSearch setup and preprocessing tasks by providing t
In OpenSearch 2.12, configuration automation is limited to ML tasks.
{: .info}

OpenSearch use case templates provide a compact description of the setup process in a JSON or YAML document. These templates describe automated workflow configurations for conversational chat or query generation, AI connectors, tools, agents, and other components that prepare OpenSearch as a backend for generative models. For template examples, see [Sample templates](https://github.com/opensearch-project/flow-framework/tree/main/sample-templates).
OpenSearch use case templates provide a compact description of the setup process in a JSON or YAML document. These templates describe automated workflow configurations for conversational chat or query generation, AI connectors, tools, agents, and other components that prepare OpenSearch as a backend for generative models. For custom template examples, see [Sample templates](https://github.com/opensearch-project/flow-framework/tree/main/sample-templates). For OpenSearch-provided templates, see [Workflow templates]({{site.url}}{{site.baseurl}}/automating-configurations/workflow-templates/).

## Key features

Expand Down
51 changes: 23 additions & 28 deletions _automating-configurations/workflow-templates.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,38 +6,15 @@ nav_order: 25

# Workflow templates

OpenSearch provides several workflow templates for some common machine learning (ML) use cases, such as semantic or conversational search.
OpenSearch provides several workflow templates for some common machine learning (ML) use cases. Using a template simplifies complex setups and provides many default values for use cases like semantic or conversational search.

You can specify a workflow template when you call the [Create Workflow API]({{site.url}}{{site.baseurl}}/automating-configurations/api/create-workflow/). To provision the workflow, specify `provision=true` as a query parameter. For example, you can configure [neural sparse search]({{site.url}}{{site.baseurl}}/search-plugins/neural-sparse-search/) by providing the `local_neural_sparse_search_bi_encoder` query parameter as `use_case`, as shown in the following request:
You can specify a workflow template when you call the [Create Workflow API]({{site.url}}{{site.baseurl}}/automating-configurations/api/create-workflow/):

```json
POST /_plugins/_flow_framework/workflow?use_case=local_neural_sparse_search_bi_encoder
```
{% include copy-curl.html %}

The workflow created using this template performs the following configuration steps:

- Deploys the default pretrained sparse encoding model (`amazon/neural-sparse/opensearch-neural-sparse-encoding-v1`).
- Creates an ingest pipeline that contains a `sparse_encoding` processor, which converts the text in a document field to vector embeddings using the deployed model.
- Creates a sample index for sparse search, specifying the default pipeline as the newly created ingest pipeline.
- To use an OpenSearch-provided workflow template, specify the template use case as the `use_case` query parameter (see the [Example](#example)). For a list of OpenSearch-provided templates, see [Supported workflow templates](#supported-workflow-templates).

## Parameters

Each workflow template has a defined schema and a set of APIs with predefined default values for each step. For more information about template parameter defaults, see [Supported workflow templates](#supported-workflow-templates).

### Overriding default values
- To use a custom workflow template, provide the complete template in the request body. For an example of a custom template, see [an example JSON template]({{site.url}}{{site.baseurl}}/automating-configurations/api/create-workflow/#example-request-register-and-deploy-a-remote-model-json) or [an example YAML template]({{site.url}}{{site.baseurl}}/automating-configurations/api/create-workflow/#example-request-register-and-deploy-an-externally-hosted-model-yaml).

To override a template's default values, provide the new values in the request body when sending a create workflow request. For example, the following request changes the Cohere model, the name of the `text_embedding` processor output field, and the name of the sparse index of the `semantic_search_with_cohere_embedding` template:

```json
POST /_plugins/_flow_framework/workflow?use_case=semantic_search_with_cohere_embedding
{
"create_connector.model" : "embed-multilingual-v3.0",
"text_embedding.field_map.output": "book_embedding",
"create_index.name": "sparse-book-index"
}
```
{% include copy-curl.html %}
To provision the workflow, specify `provision=true` as a query parameter.

## Example

Expand Down Expand Up @@ -114,6 +91,24 @@ GET /my-nlp-index/_search
```
{% include copy-curl.html %}

## Parameters

Each workflow template has a defined schema and a set of APIs with predefined default values for each step. For more information about template parameter defaults, see [Supported workflow templates](#supported-workflow-templates).

### Overriding default values

To override a template's default values, provide the new values in the request body when sending a create workflow request. For example, the following request changes the Cohere model, the name of the `text_embedding` processor output field, and the name of the sparse index of the `semantic_search_with_cohere_embedding` template:

```json
POST /_plugins/_flow_framework/workflow?use_case=semantic_search_with_cohere_embedding
{
"create_connector.model" : "embed-multilingual-v3.0",
"text_embedding.field_map.output": "book_embedding",
"create_index.name": "sparse-book-index"
}
```
{% include copy-curl.html %}

## Viewing workflow resources

The workflow you created provisioned all the necessary resources for semantic search. To view the provisioned resources, call the [Get Workflow Status API]({{site.url}}{{site.baseurl}}/automating-configurations/api/get-workflow-status/) and provide the `workflowID` for your workflow:
Expand Down

0 comments on commit 13f1902

Please sign in to comment.