[Security Assistant] Migrates to LangGraph and adds KB Tools

[spong]

Summary

Migrates our existing RAG pipeline to use LangGraph, and adds tools for Knowledge Base retrieval/storage.

When the assistantKnowledgeBaseByDefault FF is enabled, a new branch, callAssistantGraph(), is taken in postActionsConnectorExecuteRoute that exercises the LangGraph implementation. This is a drop-in replacement for the existing callAgentExecutor() in effort to keep adoption as clean and easy as possible.

The new control flow is as follows:

postActionsConnectorExecuteRoute -> callAssistantGraph() -> getDefaultAssistantGraph() -> isStreamingEnabled ? streamGraph() : invokeGraph()

Graph creation is isolated to getDefaultAssistantGraph(), and execution (streaming or not) has been extracted to streamGraph() and invokeGraph() respectively. Note: Streaming currently only works with ChatOpenAI models, but SimpleChatModelStreaming was de-risked and just need to discuss potential solutions with @stephmilovic. See comment here.

DefaultAssistantGraph

To start with a predictable and piecemeal migration, our existing agentExecutor pipeline has been recreated in LangGraph. It consists of a single agent node, either OpenAIFunctionsAgent, or StructuredChatAgent (depending on the backing LLM), a tool executing node, and a conditional edge that routes between the two nodes until there are no more function calls chosen by the agent. This varies from our initial implementation in that multiple tool calls are now supported, so a user could ask about their alerts AND retrieve additional knowledge base information in the same response.

Note

While chat_history has been plumbed into the graph, after discussing with @YulNaumenko we decided to wait to plumb the rest of persistence into the graph until #184485 is merged. I had already plumbed through the chatTitleGeneration node (here), and so will just need to include initial conversation creation and append/update operations.

Knowledge History & KB Tools

Knowledge History is now always added in the initial prompt for any KB documents marked as required, and two new tools were added for creating and recalling KB entries from within the conversation, KnowledgeBaseWriteTool and KnowledgeBaseRetrievalTool respectively. All three methods of storing and retrieving KB content use the kbDataClient for access, and scopes all requests to the authenticatedUser that made the initial request.

Additional Notes:

• LangChain dependencies have been updated, and a new dependency on LangGraph has been added.

Checklist

Delete any items that are not applicable to this PR.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
  • Feature currently behind a FF, documentation to be added once feature is complete. Tracked in [Request] 8.16 Security Assistant Custom Knowledge Bases security-docs#5337.
• Unit or functional tests were updated or added to match the most common scenarios
  • Test coverage in progress...

[YulNaumenko]

I think we should extend the filter to support empty("shared") KBs, which are available for all users in the space.

[spong]

I'm working through adding the different permutations of kb mock data today for #184974 and will use that to start crafting tests to cover these scenarios.

I want to take a moment and rethink how we're categorizing/namespacing content using kbResource though to see if there will be any issues here. For now I've kept things matching the original KB implementation, and just introduced the user kbResource for all user created entries. We will also need to capture entryType somewhere to differentiate between raw text content and index-backed entries. Would be nice to have a catch-all tags for organization and labeling too.

I will start thinking through more of this early next week as I get back to plumbing through the remainder of the kbDataClient methods and REST API's, but if you have any thoughts here I would love to hear them 🙂

[spong]

@elasticmachine merge upstream

[patrykkopycinski]

Thank you Garrett! 🙇

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: f9505d9

Failed CI Steps

• Jest Tests #1

Test Failures

• [job] [logs] Jest Tests #1 / CustomTimelineDataGridBody should render exactly as snapshots

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
elasticAssistant	32	34	+2

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
elasticAssistant	0	1	+1

Unknown metric groups

API count

id	before	after	diff
elasticAssistant	46	48	+2

History

• 💛 Build #215935 was flaky 269f8aa
• 💔 Build #215926 failed 3d2c140
• 💛 Build #215905 was flaky d38107d
• 💔 Build #215895 failed 5fca287
• 💔 Build #215831 failed 81ec705
• 💔 Build #215641 failed 837c29c

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @spong