adding blueprint examples for remote inference #1155

dhrubo-os · 2023-07-24T20:11:30Z

Description

[adding blueprint examples for remote inference]

Issues Resolved

[List any issues this PR will resolve]

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>

ylwu-amzn · 2023-07-24T20:21:58Z

docs/remote_inference_blueprints/open_ai_connector_completion_blueprint.md

@@ -0,0 +1,28 @@
+```json
+{
+    "name": "OpenAI Connector",


Suggested change

"name": "OpenAI Connector",

"name": "OpenAI Completions Connector",

Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>

ylwu-amzn · 2023-07-25T00:57:31Z

docs/remote_inference_blueprints/cohere_connector_embedding_blueprint.md

+                -0.52246094,
+                -1.2714844,
+                1.8076172,
+                0.13842773,


Let keep the first five embedding float values, then use ... to repreent the others. It seems not helping too much by showing thousands of floats.

ylwu-amzn · 2023-07-25T01:03:28Z

docs/remote_inference_blueprints/sagemaker_connector_blueprint.md

+      "headers": {
+        "content-type": "application/json"
+      },
+      "url": "https://runtime.sagemaker.${parameters.region}.amazonaws.com/endpoints/lmi-model-2023-06-24-01-35-32-275/invocations",


Delete this URL , this is just our internal Sagemaker endpiont.

Suggested change

"url": "https://runtime.sagemaker.${parameters.region}.amazonaws.com/endpoints/lmi-model-2023-06-24-01-35-32-275/invocations",

"url": "<PLEASE ADD YOUR Sagemaker URL>",

ylwu-amzn · 2023-07-25T01:04:35Z

docs/remote_inference_blueprints/sagemaker_connector_blueprint.md

+        "content-type": "application/json"
+      },
+      "url": "https://runtime.sagemaker.${parameters.region}.amazonaws.com/endpoints/lmi-model-2023-06-24-01-35-32-275/invocations",
+      "request_body": "[\"${parameters.inputs}\"]"


This request body is just from our test sagemaker embedding model. User should add their own body template.

Suggested change

"request_body": "[\"${parameters.inputs}\"]"

"request_body": "<PLEASE ADD YOUR REQUEST BODY>"

ylwu-amzn · 2023-07-25T01:06:05Z

docs/remote_inference_blueprints/sagemaker_connector_blueprint.md

+}
+```
+
+### Corresponding Predict request example:


As we can't predict what's the model deployed to Sagemaker on user side, so we should remove the predict request example and sample response.

ylwu-amzn · 2023-07-25T01:06:53Z

docs/remote_inference_blueprints/sagemaker_connector_blueprint.md

+```json
+POST /_plugins/_ml/connectors/_create
+{
+  "name": "sagemaker: embedding",


Suggested change

"name": "sagemaker: embedding",

"name": "<YOUR MODEL NAME>",

ylwu-amzn · 2023-07-25T01:07:18Z

docs/remote_inference_blueprints/sagemaker_connector_blueprint.md

+POST /_plugins/_ml/connectors/_create
+{
+  "name": "sagemaker: embedding",
+  "description": "Test connector for Sagemaker embedding model",


Suggested change

"description": "Test connector for Sagemaker embedding model",

"description": "<YOUR MODEL DESCRIPTION>",

codecov · 2023-07-25T01:07:44Z

Codecov Report

Merging #1155 (4a5c138) into 2.x (f6530e0) will decrease coverage by 0.04%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##                2.x    #1155      +/-   ##
============================================
- Coverage     78.95%   78.91%   -0.04%     
+ Complexity     2119     2116       -3     
============================================
  Files           167      167              
  Lines          8633     8633              
  Branches        869      869              
============================================
- Hits           6816     6813       -3     
- Misses         1422     1427       +5     
+ Partials        395      393       -2

Flag	Coverage Δ
ml-commons	`78.91% <ø> (-0.04%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 4 files with indirect coverage changes

ylwu-amzn · 2023-07-25T01:09:11Z

docs/remote_inference_blueprints/open_ai_connector_chat_blueprint.md

@@ -0,0 +1,92 @@
+### OpenAI connector blueprint example for chat:


Explain that this blueprint is created from OpenAI doc,
and example is from OpenAI doc https://platform.openai.com/docs/api-reference/chat?

Same for other examples, we should use the exact same example from OpenAI and Cohere official document. And we should put their doc link here so community user can refer to .

Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>

ylwu-amzn · 2023-07-25T17:26:44Z

docs/remote_inference_blueprints/sagemaker_connector_blueprint.md

+        "content-type": "application/json"
+      },
+      "url": "<PLEASE ADD YOUR Sagemaker MODEL ENDPOINT URL>",
+      "request_body": "<PLEASE ADD YOUR REQUEST BODY. Example: [\"${parameters.inputs}\"]>"


How about just "{parameters.input}" ? Add [ and ] looks confusing as that's only for embedding generation , not a general one

You mean: ${parameters.inputs} ?

Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>

ylwu-amzn

LGTM. Thanks a lot

* adding blueprint examples for remote inference Signed-off-by: Dhrubo Saha <dhrubo@amazon.com> * updating blue prints with sample response Signed-off-by: Dhrubo Saha <dhrubo@amazon.com> * addressing comments Signed-off-by: Dhrubo Saha <dhrubo@amazon.com> * addressed comments for sagemaker Signed-off-by: Dhrubo Saha <dhrubo@amazon.com> --------- Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>

adding blueprint examples for remote inference

db930e9

Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>

dhrubo-os temporarily deployed to ml-commons-cicd-env July 24, 2023 20:11 — with GitHub Actions Inactive

dhrubo-os had a problem deploying to ml-commons-cicd-env July 24, 2023 20:11 — with GitHub Actions Error

dhrubo-os had a problem deploying to ml-commons-cicd-env July 24, 2023 20:11 — with GitHub Actions Failure

ylwu-amzn reviewed Jul 24, 2023

View reviewed changes

updating blue prints with sample response

b4f6492

Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>

dhrubo-os temporarily deployed to ml-commons-cicd-env July 25, 2023 00:49 — with GitHub Actions Inactive

dhrubo-os had a problem deploying to ml-commons-cicd-env July 25, 2023 00:49 — with GitHub Actions Failure

dhrubo-os temporarily deployed to ml-commons-cicd-env July 25, 2023 00:49 — with GitHub Actions Inactive

ylwu-amzn reviewed Jul 25, 2023

View reviewed changes

addressing comments

9787f2e

Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>

dhrubo-os temporarily deployed to ml-commons-cicd-env July 25, 2023 17:19 — with GitHub Actions Inactive

ylwu-amzn reviewed Jul 25, 2023

View reviewed changes

addressed comments for sagemaker

4a5c138

Signed-off-by: Dhrubo Saha <dhrubo@amazon.com>

dhrubo-os temporarily deployed to ml-commons-cicd-env July 25, 2023 17:39 — with GitHub Actions Inactive

dhrubo-os had a problem deploying to ml-commons-cicd-env July 25, 2023 17:39 — with GitHub Actions Failure

dhrubo-os had a problem deploying to ml-commons-cicd-env July 25, 2023 17:39 — with GitHub Actions Error

ylwu-amzn approved these changes Jul 25, 2023

View reviewed changes

rbhavna approved these changes Jul 25, 2023

View reviewed changes

dhrubo-os temporarily deployed to ml-commons-cicd-env July 25, 2023 18:01 — with GitHub Actions Inactive

dhrubo-os had a problem deploying to ml-commons-cicd-env July 25, 2023 18:01 — with GitHub Actions Error

dhrubo-os merged commit 8b241a9 into opensearch-project:2.x Jul 25, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

adding blueprint examples for remote inference #1155

adding blueprint examples for remote inference #1155

dhrubo-os commented Jul 24, 2023

ylwu-amzn Jul 24, 2023

ylwu-amzn Jul 25, 2023

ylwu-amzn Jul 25, 2023

ylwu-amzn Jul 25, 2023

ylwu-amzn Jul 25, 2023 •

edited

Loading

ylwu-amzn Jul 25, 2023

ylwu-amzn Jul 25, 2023

codecov bot commented Jul 25, 2023 •

edited

Loading

ylwu-amzn Jul 25, 2023 •

edited

Loading

ylwu-amzn Jul 25, 2023

ylwu-amzn Jul 25, 2023 •

edited

Loading

dhrubo-os Jul 25, 2023

ylwu-amzn left a comment

	"name": "OpenAI Connector",
	"name": "OpenAI Completions Connector",

	"url": "https://runtime.sagemaker.${parameters.region}.amazonaws.com/endpoints/lmi-model-2023-06-24-01-35-32-275/invocations",
	"url": "<PLEASE ADD YOUR Sagemaker URL>",

	"request_body": "[\"${parameters.inputs}\"]"
	"request_body": "<PLEASE ADD YOUR REQUEST BODY>"

	"description": "Test connector for Sagemaker embedding model",
	"description": "<YOUR MODEL DESCRIPTION>",

		@@ -0,0 +1,92 @@
		### OpenAI connector blueprint example for chat:

adding blueprint examples for remote inference #1155

adding blueprint examples for remote inference #1155

Conversation

dhrubo-os commented Jul 24, 2023

Description

Issues Resolved

Check List

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ylwu-amzn Jul 25, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jul 25, 2023 • edited Loading

Codecov Report

ylwu-amzn Jul 25, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ylwu-amzn Jul 25, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ylwu-amzn left a comment

Choose a reason for hiding this comment

ylwu-amzn Jul 25, 2023 •

edited

Loading

codecov bot commented Jul 25, 2023 •

edited

Loading

ylwu-amzn Jul 25, 2023 •

edited

Loading

ylwu-amzn Jul 25, 2023 •

edited

Loading