`BedrockChat` & `GeminiChat` #186809

patrykkopycinski · 2024-06-24T11:31:50Z

Summary

Adopted BedrockChat from @langchain/community package that adds support for tools calling
https://js.langchain.com/v0.2/docs/integrations/chat/bedrock/

Adopted ChatGoogleGenerativeAI from @langchain/google-genai package that adds support for tools calling https://js.langchain.com/v0.2/docs/integrations/chat/google_generativeai

Hidden behind FF: --xpack.securitySolution.enableExperimental=[assistantBedrockChat]

As of this PR integration_assistant is still going to use ActionsClientSimpleChatModel. After the FF will be enabled by default we will switch integration_assistant to use new chat model.

Thank you @stephmilovic a ton 🙇

patrykkopycinski · 2024-06-24T11:52:47Z

/ci

…fix'

patrykkopycinski · 2024-06-24T12:12:58Z

/ci

…nski/kibana into feat/langchain-bedrock-chat

…bedrock-chat

…bedrock-chat # Conflicts: # package.json # x-pack/plugins/integration_assistant/tsconfig.json # yarn.lock

patrykkopycinski · 2024-06-29T21:46:07Z

...elastic_assistant/server/lib/langchain/graphs/default_assistant_graph/nodes/execute_tools.ts

+
+  let out;
+  try {
+    out = await toolExecutor.invoke(agentAction, config);


to catch Received tool input did not match expected schema errors

patrykkopycinski · 2024-06-29T21:56:38Z

/ci

…fix'

…nski/kibana into feat/langchain-bedrock-chat

patrykkopycinski · 2024-06-29T22:27:34Z

/ci

patrykkopycinski · 2024-06-29T23:48:58Z

/ci

patrykkopycinski · 2024-06-30T07:10:27Z

/ci

patrykkopycinski · 2024-06-30T09:47:17Z

/ci

patrykkopycinski · 2024-06-30T17:54:58Z

/ci

elasticmachine · 2024-07-22T20:08:02Z

Pinging @elastic/security-solution (Team: SecuritySolution)

This reverts commit 50d73f1.

stephmilovic

Thanks for these changes @patrykkopycinski. When using the feature flags, I got more accurate and consistent results for Bedrock and Gemini with and without streaming. LGTM

joemcelroy

Checked this PR with Patryk and due to the introduction of safety_settings, this meant some setups (like mine, no monthly billing google setup), it throws an 400 API exception.

Patryk going to discuss what we can do in this situation.

"{\n  \"error\": {\n    \"code\": 400,\n    \"message\": \"User has requested a restricted HarmBlockThreshold setting BLOCK_NONE. You can get access either (a) through an allowlist via your Google account team, or (b) by switching your account type to monthly invoiced billing via this instruction: https://cloud.google.com/billing/docs/how-to/invoiced-billing.\",\n    \"status\": \"INVALID_ARGUMENT\"\n  }\n}\n"

…nski/kibana into feat/langchain-bedrock-chat

stephmilovic · 2024-07-23T14:32:00Z

@joemcelroy I think I fixed the issue. I bubbled up this error so it will reach the user. Additionally, I scaled back the threshold from BLOCK_NONE to BLOCK_ONLY_HIGH. For whatever reason, yesterday i was getting the error 80% of the time when asking the assistant to summarize an alert and now i can't get it at all. Hoping maybe the model was just having a bad day... Regardless, you should no longer experience the error and if you do, you will get a message in the UI:

joemcelroy

Looks good from playground POV

stephmilovic · 2024-07-23T18:34:30Z

@elasticmachine merge upstream

kibana-ci · 2024-07-23T19:19:26Z

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

[job] [logs] Jest Tests #15 / CustomFields renders correctly
[job] [logs] FTR Configs #4 / Screenshots - serverless observability UI response ops docs observability cases Observability case settings case settings screenshots

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`integrationAssistant`	536	539	+3
`securitySolution`	5638	5641	+3
total			+6

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`fleet`	1.8MB	1.8MB	+4.2KB
`integrationAssistant`	933.2KB	937.4KB	+4.2KB
`securitySolution`	17.3MB	17.3MB	+4.5KB
total			+12.8KB

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`@kbn/langchain`	3	1	-2
`elasticAssistant`	1	2	+1
total			-1

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`securitySolution`	86.7KB	86.7KB	+24.0B
`stackConnectors`	53.7KB	53.7KB	+60.0B
`timelines`	109.3KB	113.4KB	+4.2KB
total			+4.3KB

Unknown metric groups

API count

id	before	after	diff
`@kbn/zod`	1224	1254	+30

ESLint disabled line counts

id	before	after	diff
`@kbn/langchain`	0	1	+1

Total ESLint disabled count

id	before	after	diff
`@kbn/langchain`	0	1	+1

History

💔 Build #223266 failed ffe2a37
💔 Build #223232 failed 11c7357
💚 Build #223114 succeeded 0974e88
💔 Build #223092 failed 3923074
💔 Build #223027 failed 8255c11

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @patrykkopycinski

kibanamachine · 2024-07-23T20:21:53Z

💔 All backports failed

Status	Branch	Result
❌	8.15	Backport failed because of merge conflicts You might need to backport the following PRs to 8.15: - Upgrade EUI to v95.3.0 (#187342) - Update dependency @launchdarkly/node-server-sdk to ^9.4.7 (main) (#187807) - Update dependency @redocly/cli to ^1.17.0 (main) (#187765)

Manual backport

To create the backport manually run:

node scripts/backport --pr 186809

Questions ?

Please refer to the Backport tool documentation

stephmilovic · 2024-07-23T20:31:59Z

💚 All backports created successfully

Status	Branch	Result
✅	8.15

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

@stephmilovic

## Summary Adopted `BedrockChat` from `@langchain/community` package that adds support for tools calling https://js.langchain.com/v0.2/docs/integrations/chat/bedrock/ Adopted `ChatGoogleGenerativeAI ` from `@langchain/google-genai` package that adds support for tools calling https://js.langchain.com/v0.2/docs/integrations/chat/google_generativeai Hidden behind FF: `--xpack.securitySolution.enableExperimental=[assistantBedrockChat]` As of this PR `integration_assistant` is still going to use `ActionsClientSimpleChatModel`. After the FF will be enabled by default we will switch `integration_assistant` to use new chat model. Thank you @stephmilovic a ton 🙇 --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Steph Milovic <stephanie.milovic@elastic.co> Co-authored-by: Garrett Spong <spong@users.noreply.github.com> (cherry picked from commit 26dd61e) # Conflicts: # package.json # yarn.lock

@stephmilovic

# Backport This will backport the following commits from `main` to `8.15`: - [`BedrockChat` & `GeminiChat` (#186809)](#186809)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  --------- Co-authored-by: Patryk Kopyciński <contact@patrykkopycinski.com>

* master: (3487 commits) `BedrockChat` & `GeminiChat` (elastic#186809) [ResponseOps] log error when ES Query rules find docs out of time range (elastic#186332) skip flaky suite (elastic#188997) [Security solution][Alert Details] Enable preview feature flag and cypress tests (elastic#188580) [EuiProviders] Warn Developer if EuiProvider is missing (elastic#184608) [Security Solution ] Fixes Timeline infinite loading bug (elastic#188943) Improve SearchSource SearchRequest type (elastic#186862) Deprecate Search Sessions config (elastic#188037) [Synthetics] Add missing monitorType and tag info in cards !! (elastic#188824) [Console Monaco] Resolve uncaught error from tokenizer (elastic#188746) [Data Forge] Add `service.logs` dataset as a data stream (elastic#188786) [Console] Fix failing bulk requests (elastic#188552) Update dependency terser to ^5.31.2 (main) (elastic#188528) [APM][ECO] Telemetry (elastic#188627) [Fleet] Fix uninstall package validation accross space (elastic#188749) Update warning on `xpack.fleet.enableExperimental` (elastic#188917) [DOCS][Cases] Automate more screenshots for cases (elastic#188697) [Fleet] Fix get one agent when feature flag disabled (elastic#188953) chore(investigate): Add investigate-app plugin from poc (elastic#188122) [Monaco Editor] Add Search functionality (elastic#188337) ...

* master: (2400 commits) `BedrockChat` & `GeminiChat` (elastic#186809) [ResponseOps] log error when ES Query rules find docs out of time range (elastic#186332) skip flaky suite (elastic#188997) [Security solution][Alert Details] Enable preview feature flag and cypress tests (elastic#188580) [EuiProviders] Warn Developer if EuiProvider is missing (elastic#184608) [Security Solution ] Fixes Timeline infinite loading bug (elastic#188943) Improve SearchSource SearchRequest type (elastic#186862) Deprecate Search Sessions config (elastic#188037) [Synthetics] Add missing monitorType and tag info in cards !! (elastic#188824) [Console Monaco] Resolve uncaught error from tokenizer (elastic#188746) [Data Forge] Add `service.logs` dataset as a data stream (elastic#188786) [Console] Fix failing bulk requests (elastic#188552) Update dependency terser to ^5.31.2 (main) (elastic#188528) [APM][ECO] Telemetry (elastic#188627) [Fleet] Fix uninstall package validation accross space (elastic#188749) Update warning on `xpack.fleet.enableExperimental` (elastic#188917) [DOCS][Cases] Automate more screenshots for cases (elastic#188697) [Fleet] Fix get one agent when feature flag disabled (elastic#188953) chore(investigate): Add investigate-app plugin from poc (elastic#188122) [Monaco Editor] Add Search functionality (elastic#188337) ...

stephmilovic · 2024-11-21T23:11:40Z

x-pack/packages/kbn-langchain/server/language_models/gemini_chat.ts

+        partialStreamChunk += nextChunk;
+      }
+
+      if (parsedStreamChunk !== null && !parsedStreamChunk.candidates?.[0]?.finishReason) {


Why no finishReason chunks?

BedrockChat

65fdf44

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

538b846

…fix'

patrykkopycinski added 6 commits June 25, 2024 14:22

test

34ff58b

Merge branch 'feat/langchain-bedrock-chat' of github.com:patrykkopyci…

a3aac87

…nski/kibana into feat/langchain-bedrock-chat

Merge branch 'main' of github.com:elastic/kibana into feat/langchain-…

7cc74e7

…bedrock-chat

Merge branch 'main' of github.com:elastic/kibana into feat/langchain-…

19155f0

…bedrock-chat # Conflicts: # package.json # x-pack/plugins/integration_assistant/tsconfig.json # yarn.lock

cleanup

b87106f

add FF

e1434ba

patrykkopycinski commented Jun 29, 2024

View reviewed changes

cleanup

cf3d547

patrykkopycinski self-assigned this Jun 29, 2024

kibanamachine and others added 3 commits June 29, 2024 22:08

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

281f98e

…fix'

fix

10f5de9

Merge branch 'feat/langchain-bedrock-chat' of github.com:patrykkopyci…

2b5332b

…nski/kibana into feat/langchain-bedrock-chat

fix

4cd8f7f

kibanamachine and others added 2 commits June 30, 2024 00:34

[CI] Auto-commit changed files from 'yarn openapi:generate'

1bd50b5

fix

7696942

fix

340097b

patrykkopycinski added 2 commits June 30, 2024 19:48

cleanup

2978871

cleanup

c414a3a

patrykkopycinski added 2 commits June 30, 2024 20:23

basic tests

931745d

cleanup package.json

467cd96

stephmilovic added the ci:cloud-deploy Create or update a Cloud deployment label Jul 22, 2024

stephmilovic added 4 commits July 22, 2024 15:18

fix gemini subaction args

91a3dfa

fix safety settings

8fe746c

Revert "fix weird type thing"

c5cc32e

This reverts commit 50d73f1.

add comment

3923074

stephmilovic approved these changes Jul 22, 2024

View reviewed changes

stephmilovic and others added 3 commits July 22, 2024 18:25

fix jest 1

1133f28

fix jest 2

6e30606

Merge branch 'main' into feat/langchain-bedrock-chat

0974e88

joemcelroy reviewed Jul 23, 2024

View reviewed changes

stephmilovic added 2 commits July 23, 2024 09:29

fix for error

6f50245

Merge branch 'feat/langchain-bedrock-chat' of github.com:patrykkopyci…

11c7357

…nski/kibana into feat/langchain-bedrock-chat

joemcelroy approved these changes Jul 23, 2024

View reviewed changes

mikecote approved these changes Jul 23, 2024

View reviewed changes

update jest

ffe2a37

patrykkopycinski enabled auto-merge (squash) July 23, 2024 16:42

Merge branch 'main' into feat/langchain-bedrock-chat

a0b92df

patrykkopycinski merged commit 26dd61e into elastic:main Jul 23, 2024
44 checks passed

stephmilovic mentioned this pull request Jul 23, 2024

[8.15] BedrockChat & GeminiChat (#186809) #189008

Merged

stephmilovic reviewed Nov 21, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`BedrockChat` & `GeminiChat` #186809

`BedrockChat` & `GeminiChat` #186809

patrykkopycinski commented Jun 24, 2024 •

edited by kibanamachine

Loading

patrykkopycinski commented Jun 24, 2024

patrykkopycinski commented Jun 24, 2024

patrykkopycinski Jun 29, 2024

patrykkopycinski commented Jun 29, 2024

patrykkopycinski commented Jun 29, 2024

patrykkopycinski commented Jun 29, 2024

patrykkopycinski commented Jun 30, 2024

patrykkopycinski commented Jun 30, 2024

patrykkopycinski commented Jun 30, 2024

elasticmachine commented Jul 22, 2024

stephmilovic left a comment

joemcelroy left a comment

stephmilovic commented Jul 23, 2024

joemcelroy left a comment

stephmilovic commented Jul 23, 2024

kibana-ci commented Jul 23, 2024 •

edited

Loading

API count

ESLint disabled line counts

Total ESLint disabled count

kibanamachine commented Jul 23, 2024

stephmilovic commented Jul 23, 2024

stephmilovic Nov 21, 2024

BedrockChat & GeminiChat #186809

BedrockChat & GeminiChat #186809

Conversation

patrykkopycinski commented Jun 24, 2024 • edited by kibanamachine Loading

Summary

patrykkopycinski commented Jun 24, 2024

patrykkopycinski commented Jun 24, 2024

patrykkopycinski Jun 29, 2024

Choose a reason for hiding this comment

patrykkopycinski commented Jun 29, 2024

patrykkopycinski commented Jun 29, 2024

patrykkopycinski commented Jun 29, 2024

patrykkopycinski commented Jun 30, 2024

patrykkopycinski commented Jun 30, 2024

patrykkopycinski commented Jun 30, 2024

elasticmachine commented Jul 22, 2024

stephmilovic left a comment

Choose a reason for hiding this comment

joemcelroy left a comment

Choose a reason for hiding this comment

stephmilovic commented Jul 23, 2024

joemcelroy left a comment

Choose a reason for hiding this comment

stephmilovic commented Jul 23, 2024

kibana-ci commented Jul 23, 2024 • edited Loading

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Module Count

Async chunks

Public APIs missing exports

Page load bundle

API count

ESLint disabled line counts

Total ESLint disabled count

History

kibanamachine commented Jul 23, 2024

💔 All backports failed

Manual backport

Questions ?

stephmilovic commented Jul 23, 2024

💚 All backports created successfully

Questions ?

stephmilovic Nov 21, 2024

Choose a reason for hiding this comment

`BedrockChat` & `GeminiChat` #186809

`BedrockChat` & `GeminiChat` #186809

patrykkopycinski commented Jun 24, 2024 •

edited by kibanamachine

Loading

kibana-ci commented Jul 23, 2024 •

edited

Loading