-
Notifications
You must be signed in to change notification settings - Fork 138
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Browse files
Browse the repository at this point in the history
* add bedrock titan embedding model blueprint Signed-off-by: Yaliang Wu <ylwu@amazon.com> * fix create connector response Signed-off-by: Yaliang Wu <ylwu@amazon.com> * fix step number Signed-off-by: Yaliang Wu <ylwu@amazon.com> --------- Signed-off-by: Yaliang Wu <ylwu@amazon.com> (cherry picked from commit e9e3834) Co-authored-by: Yaliang Wu <ylwu@amazon.com>
- Loading branch information
1 parent
54b198f
commit 693a53c
Showing
2 changed files
with
180 additions
and
2 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
177 changes: 177 additions & 0 deletions
177
docs/remote_inference_blueprints/bedrock_connector_titan_embedding_blueprint.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,177 @@ | ||
| ### Bedrock connector blueprint example | ||
|
|
||
| 1. Add connector endpoint to trusted URLs: | ||
|
|
||
| ```json | ||
| PUT /_cluster/settings | ||
| { | ||
| "persistent": { | ||
| "plugins.ml_commons.trusted_connector_endpoints_regex": [ | ||
| "^https://bedrock-runtime\\..*[a-z0-9-]\\.amazonaws\\.com/.*$" | ||
| ] | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| 2. Create connector for Amazon Bedrock: | ||
|
|
||
| If you are using self-managed Opensearch, you should supply AWS credentials: | ||
|
|
||
| ```json | ||
| POST /_plugins/_ml/connectors/_create | ||
| { | ||
| "name": "Amazon Bedrock Connector: embedding", | ||
| "description": "The connector to bedrock Titan embedding model", | ||
| "version": 1, | ||
| "protocol": "aws_sigv4", | ||
| "parameters": { | ||
| "region": "<PLEASE ADD YOUR AWS REGION HERE>", | ||
| "service_name": "bedrock" | ||
| }, | ||
| "credential": { | ||
| "access_key": "<PLEASE ADD YOUR AWS ACCESS KEY HERE>", | ||
| "secret_key": "<PLEASE ADD YOUR AWS SECRET KEY HERE>", | ||
| "session_token": "<PLEASE ADD YOUR AWS SECURITY TOKEN HERE>" | ||
| }, | ||
| "actions": [ | ||
| { | ||
| "action_type": "predict", | ||
| "method": "POST", | ||
| "url": "https://bedrock-runtime.us-east-1.amazonaws.com/model/amazon.titan-embed-text-v1/invoke", | ||
| "headers": { | ||
| "content-type": "application/json", | ||
| "x-amz-content-sha256": "required" | ||
| }, | ||
| "request_body": "{ \"inputText\": \"${parameters.inputText}\" }", | ||
| "pre_process_function": "\n StringBuilder builder = new StringBuilder();\n builder.append(\"\\\"\");\n String first = params.text_docs[0];\n builder.append(first);\n builder.append(\"\\\"\");\n def parameters = \"{\" +\"\\\"inputText\\\":\" + builder + \"}\";\n return \"{\" +\"\\\"parameters\\\":\" + parameters + \"}\";", | ||
| "post_process_function": "\n def name = \"sentence_embedding\";\n def dataType = \"FLOAT32\";\n if (params.embedding == null || params.embedding.length == 0) {\n return params.message;\n }\n def shape = [params.embedding.length];\n def json = \"{\" +\n \"\\\"name\\\":\\\"\" + name + \"\\\",\" +\n \"\\\"data_type\\\":\\\"\" + dataType + \"\\\",\" +\n \"\\\"shape\\\":\" + shape + \",\" +\n \"\\\"data\\\":\" + params.embedding +\n \"}\";\n return json;\n " | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| If using the AWS Opensearch Service, you can provide an IAM role arn that allows access to the bedrock service. | ||
| Refer to this [AWS doc](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ml-amazon-connector.html) | ||
|
|
||
| ```json | ||
| POST /_plugins/_ml/connectors/_create | ||
| { | ||
| "name": "Amazon Bedrock Connector: embedding", | ||
| "description": "The connector to bedrock Titan embedding model", | ||
| "version": 1, | ||
| "protocol": "aws_sigv4", | ||
| "parameters": { | ||
| "region": "<PLEASE ADD YOUR AWS REGION HERE>", | ||
| "service_name": "bedrock" | ||
| }, | ||
| "credential": { | ||
| "roleArn": "<PLEASE ADD YOUR AWS ROLE ARN HERE>" | ||
| }, | ||
| "actions": [ | ||
| { | ||
| "action_type": "predict", | ||
| "method": "POST", | ||
| "url": "https://bedrock-runtime.us-east-1.amazonaws.com/model/amazon.titan-embed-text-v1/invoke", | ||
| "headers": { | ||
| "content-type": "application/json", | ||
| "x-amz-content-sha256": "required" | ||
| }, | ||
| "request_body": "{ \"inputText\": \"${parameters.inputText}\" }", | ||
| "pre_process_function": "\n StringBuilder builder = new StringBuilder();\n builder.append(\"\\\"\");\n String first = params.text_docs[0];\n builder.append(first);\n builder.append(\"\\\"\");\n def parameters = \"{\" +\"\\\"inputText\\\":\" + builder + \"}\";\n return \"{\" +\"\\\"parameters\\\":\" + parameters + \"}\";", | ||
| "post_process_function": "\n def name = \"sentence_embedding\";\n def dataType = \"FLOAT32\";\n if (params.embedding == null || params.embedding.length == 0) {\n return params.message;\n }\n def shape = [params.embedding.length];\n def json = \"{\" +\n \"\\\"name\\\":\\\"\" + name + \"\\\",\" +\n \"\\\"data_type\\\":\\\"\" + dataType + \"\\\",\" +\n \"\\\"shape\\\":\" + shape + \",\" +\n \"\\\"data\\\":\" + params.embedding +\n \"}\";\n return json;\n " | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| Response: | ||
| ```json | ||
| { | ||
| "connector_id": "nzh9PIsBnGXNcxYpPEcv" | ||
| } | ||
| ``` | ||
|
|
||
| 3. Create model group: | ||
|
|
||
| ```json | ||
| POST /_plugins/_ml/model_groups/_register | ||
| { | ||
| "name": "remote_model_group", | ||
| "description": "This is an example description" | ||
| } | ||
| ``` | ||
|
|
||
| Response: | ||
| ```json | ||
| { | ||
| "model_group_id": "rqR9PIsBQRofe4CScErR", | ||
| "status": "CREATED" | ||
| } | ||
| ``` | ||
|
|
||
| 4. Register model to model group & deploy model: | ||
|
|
||
| ```json | ||
| POST /_plugins/_ml/models/_register | ||
| { | ||
| "name": "Bedrock embedding model", | ||
| "function_name": "remote", | ||
| "model_group_id": "rqR9PIsBQRofe4CScErR", | ||
| "description": "test model", | ||
| "connector_id": "nzh9PIsBnGXNcxYpPEcv" | ||
| } | ||
| ``` | ||
|
|
||
| Response: | ||
| ```json | ||
| { | ||
| "task_id": "r6R9PIsBQRofe4CSlUoG", | ||
| "status": "CREATED" | ||
| } | ||
| ``` | ||
| Get model id from task | ||
| ```json | ||
| GET /_plugins/_ml/tasks/r6R9PIsBQRofe4CSlUoG | ||
| ``` | ||
| Deploy model, in this demo the model id is `sKR9PIsBQRofe4CSlUov` | ||
| ```json | ||
| POST /_plugins/_ml/models/sKR9PIsBQRofe4CSlUov/_deploy | ||
| ``` | ||
|
|
||
| 5. Test model inference | ||
|
|
||
| ```json | ||
| POST /_plugins/_ml/models/sKR9PIsBQRofe4CSlUov/_predict | ||
| { | ||
| "parameters": { | ||
| "inputs": "What is the meaning of life?" | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| Response: | ||
| ```json | ||
| { | ||
| "inference_results": [ | ||
| { | ||
| "output": [ | ||
| { | ||
| "name": "sentence_embedding", | ||
| "data_type": "FLOAT32", | ||
| "shape": [ | ||
| 1536 | ||
| ], | ||
| "data": [ | ||
| 0.5390625, | ||
| -0.46679688, | ||
| -0.125, | ||
| ... | ||
| ] | ||
| } | ||
| ], | ||
| "status_code": 200 | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|