[Security Assistant] Adds client hooks and internal routes for managing Knowledge Base Entries #184974

spong · 2024-06-06T22:52:13Z

Summary

This PR adds client hooks and basic REST API's for accessing and mutating Knowledge Base Entries. This is in support of @angorayc building out the new Knowledge Base settings interface.

Change set includes:

Refactors existing KB client hooks from x-pack/packages/kbn-elastic-assistant/impl/knowledge_base to be co-located next to the API methods where we put all our other hooks: x-pack/packages/kbn-elastic-assistant/impl/assistant/api/knowledge_base
Refactors existing KB API calls and associated tests out of kbn-elastic-assistant/impl/assistant/api/index.tsx and into x-pack/packages/kbn-elastic-assistant/impl/assistant/api/knowledge_base/api.tsx
Adds new find_knowledge_base_entries_route.schema.yaml OAS for the supporting /internal/elastic_assistant/knowledge_base/entries/_find route
Refactors SortOrder out of existing OAS's into the shared schemas/common_attributes.schema.yaml

Client Hooks & Routes

Adds new useKnowledgeBaseEntries() hook and corresponding /knowledge_base/entries/_find route for returning paginated KB Entries to populate the KB table in settings. E.g.

    const {
      assistantFeatures: { assistantKnowledgeBaseByDefault: enableKnowledgeBaseByDefault },
      http,
      toasts,
    } = useAssistantContext();
    const { data: kbEntries, isLoading: isLoadingEntries } = useKnowledgeBaseEntries({ http });

Sample Response

{
  "perPage": 20,
  "page": 1,
  "total": 145,
  "data": [
    {
      "timestamp": "2024-06-05T21:19:56.482Z",
      "id": "CtBF6o8BSQy1Bdxt2FHz",
      "createdAt": "2024-06-05T21:19:56.482Z",
      "createdBy": "u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0",
      "updatedAt": "2024-06-05T21:19:56.482Z",
      "updatedBy": "u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0",
      "users": [
        {
          "id": "u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0",
          "name": "elastic"
        }
      ],
      "metadata": {
        "kbResource": "security_labs",
        "source": "/Users/garrettspong/dev/kibana-main/x-pack/plugins/elastic_assistant/server/knowledge_base/security_labs/2022_elastic_global_threat_report_announcement.mdx",
        "required": false
      },
      "namespace": "default",
      "text": "[Source Content Here]",
      "vector": {
        "modelId": ".elser_model_2",
        "tokens": {
          "2": 0.06595266,
          ...
        }
      }
    },
    ...
  ]
}

Response is the full newly created entry. Same format for the entry as above in the _find API, and the KnowledgeBaseEntries cache is invalidated.

Adds new useCreateKnowledgeBaseEntry() hook and corresponding /knowledge_base/entries route for creating new KB Entries

    const entry: KnowledgeBaseEntryCreateProps = {
      metadata: {
        kbResource: 'user',
        required: true,
        source: 'user',
      },
      text: 'Useful information about the user',
    };
    const { mutate: createEntry, isLoading: isCreatingEntry } = useCreateKnowledgeBaseEntry({
      http,
    });
    await createEntry(entry);

Adds new useDeleteKnowledgeBaseEntries() hook and corresponding /knowledge_base/entries/_bulk_action route for deleting existing KB Entries. I left a TODO to plumb through delete_by_query so we can add a filter bar to the table. Need to confirm if we can do pagination with similarity search as well.

    const { mutate: deleteEntries, isLoading: isDeletingEntries } = useDeleteKnowledgeBaseEntries({
      http,
    });
    await deleteEntries({ ids: ['YOE_CZABSQy1BdxtAGbs'] })

See KnowledgeBaseEntryBulkCrudActionResponse for response formats. KnowledgeBaseEntries cache is invalidated upon delete.

Checklist

Delete any items that are not applicable to this PR.

~~Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support~~
Documentation was added for features that require explanation or tutorials
- Feature currently behind feature flag. Documentation to be added before flag is removed. Tracked in [Request] 8.16 Security Assistant Custom Knowledge Bases security-docs#5337
Unit or functional tests were updated or added to match the most common scenarios
- API tests will need to be rounded out as we finalize functionality behind the feature flag

…ooks

spong · 2024-06-11T20:05:57Z

x-pack/packages/kbn-elastic-assistant-common/constants.ts

+export const ELASTIC_AI_ASSISTANT_KNOWLEDGE_BASE_ENTRIES_URL = `${ELASTIC_AI_ASSISTANT_INTERNAL_URL}/knowledge_base/entries`;
+export const ELASTIC_AI_ASSISTANT_KNOWLEDGE_BASE_ENTRIES_URL_FIND = `${ELASTIC_AI_ASSISTANT_KNOWLEDGE_BASE_ENTRIES_URL}/_find`;
+export const ELASTIC_AI_ASSISTANT_KNOWLEDGE_BASE_ENTRIES_URL_BULK_ACTION = `${ELASTIC_AI_ASSISTANT_KNOWLEDGE_BASE_ENTRIES_URL}/_bulk_action`;


Moving to internal routes and actually registering them. Routes contain logic for license/authUser/featureFlagEnabled checks.

spong · 2024-06-11T20:08:02Z

...ic-assistant-common/impl/schemas/anonymization_fields/find_anonymization_fields_route.gen.ts

@@ -17,6 +17,7 @@ import { ArrayFromString } from '@kbn/zod-helpers';
 *   version: 1
 */

+import { SortOrder } from '../common_attributes.gen';


SortOrder was common across kb, conversations, and anonymization, so extracted to common attributes.

spong · 2024-06-11T20:14:11Z

...lastic-assistant-common/impl/schemas/knowledge_base/find_knowledge_base_entries_route.gen.ts

+export type FindKnowledgeBaseEntriesRequestQuery = z.infer<
+  typeof FindKnowledgeBaseEntriesRequestQuery
+>;
+export const FindKnowledgeBaseEntriesRequestQuery = z.object({


I would like to support semantic search in this API as well (already plumbed it out in kbDataClient over in the sec-labs-kb PR). Will probably expose something like search_type:semantic param to use it, and then re-use filter param. If there is a more standard pattern for supporting multiple search types in the same API I can use that.

Not sure how pagination and stable references work with semantic search, so should verify that behavior first.

spong · 2024-06-11T20:17:30Z

x-pack/plugins/elastic_assistant/server/routes/helpers.ts

+/**
+ * Helper to perform checks for authenticated user, capability, and license. Perform all or one
+ * of the checks by providing relevant optional params. Check order is license, authenticated user,
+ * then capability.


Noticed our routes are starting to bloat with common checks like license, authUser or if a feature flag is enabled, so I made this helper to do any or all of those checks in a single block.

spong · 2024-06-11T20:18:31Z

x-pack/plugins/elastic_assistant/server/routes/knowledge_base/entries/create_route.ts

 export const createKnowledgeBaseEntryRoute = (router: ElasticAssistantPluginRouter): void => {
  router.versioned
    .post({
-      access: 'public',
+      access: 'internal',


This route was not previously registered, but it is now as of this PR, so marking as internal until we're ready.

spong · 2024-06-12T05:46:40Z

x-pack/plugins/elastic_assistant/server/lib/langchain/content_loaders/esql_loader.ts

+    const docs = (await docsLoader.load()) as Array<Document<Metadata>>;
+    const languageDocs = (await languageLoader.load()) as Array<Document<Metadata>>;
    const rawExampleQueries = await exampleQueriesLoader.load();


Will be removing castings over in the KB content PR (as loaders become generalized): #184885

spong · 2024-06-12T06:07:33Z

@elasticmachine merge upstream

kibanamachine · 2024-06-14T02:58:00Z

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#6327

[❌] x-pack/test/security_solution_api_integration/test_suites/genai/nlp_cleanup_task/basic_license_essentials_tier/configs/serverless.config.ts: 20/25 tests passed.

see run history

andrew-goldstein · 2024-06-14T03:48:28Z

x-pack/packages/kbn-elastic-assistant/impl/assistant/api/knowledge_base/api.tsx

+  resource,
+  signal,
+}: ReadKnowledgeBaseRequestParams & { http: HttpSetup; signal?: AbortSignal | undefined }): Promise<
+  ReadKnowledgeBaseResponse | IHttpFetchError


Consider allowing errors to propagate as an alternative to returning IHttpFetchError in this function, and the others in this file

This was one of the existing hooks that got moved, so will live as-is for the time being. The new hooks propagate errors and return default data so are easier to work with. Will update these as the the cleanup work around the KB continues.

andrew-goldstein

Thanks @spong for the new KB hooks and routes! 🙏
LGTM 🚀

spong · 2024-06-14T22:16:04Z

@elasticmachine merge upstream

angorayc

Thanks for creating the knowledge base routes @spong !

angorayc · 2024-06-17T09:47:44Z

...bn-elastic-assistant/impl/assistant/api/knowledge_base/entries/use_knowledge_base_entries.ts

+        {
+          method: 'GET',
+          version: API_VERSIONS.internal.v1,
+          query,


In this case we only retrieve 100 knowledge base maximum and no pagination, right?

Yeah, this is leftover from using the conversations API as a template. I will be sure to plumb through pagination to the hook here 👍

angorayc · 2024-06-17T09:49:26Z

...bn-elastic-assistant/impl/assistant/api/knowledge_base/entries/use_knowledge_base_entries.ts

+ * @param {Object} options - The options object.
+ * @param {HttpSetup} options.http - HttpSetup
+ * @param {Function} [options.onFetch] - transformation function for kb entries fetch result
+ * @param {AbortSignal} [options.signal] - AbortSignal


Suggested change

* @param {Object} options - The options object.

* @param {HttpSetup} options.http - HttpSetup

* @param {Function} [options.onFetch] - transformation function for kb entries fetch result

* @param {AbortSignal} [options.signal] - AbortSignal

* @param {HttpSetup} options.http - HttpSetup

* @param {AbortSignal} [options.signal] - AbortSignal

angorayc · 2024-06-17T09:56:41Z

@elasticmachine merge upstream

kibana-ci · 2024-06-19T01:48:08Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 5478087

Failed CI Steps

Test Failures

[job] [logs] x-pack/test/alerting_api_integration/security_and_spaces/group1/config.ts / alerting api integration security and spaces enabled Alerts - Group 1 alerts backfill rule runs find backfill superuser at space1 should handle finding backfill requests with query string appropriately
[job] [logs] FTR Configs #63 / discover/esql discover esql columns should reset columns only if index pattern changes in non-transformational query

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`securitySolution`	5517	5519	+2

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/elastic-assistant-common`	276	286	+10

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	13.6MB	13.6MB	+2.4KB

Unknown metric groups

API count

id	before	after	diff
`@kbn/elastic-assistant-common`	295	305	+10

ESLint disabled line counts

id	before	after	diff
`elasticAssistant`	39	40	+1

Total ESLint disabled count

id	before	after	diff
`elasticAssistant`	39	40	+1

History

💛 Build #216214 was flaky 5a62842
💔 Build #216019 failed 8645635
💛 Build #215950 was flaky 24c5701
💔 Build #215948 failed 0ac5d0f
💚 Build #215936 succeeded af3e139
💚 Build #215375 succeeded 6c36192

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @spong

…o support `IndexEntries` (#186566) ## Summary This is a follow-up to #184974 that updates the KB Entries API's (and underlying schema) to support `IndexEntries` in addition to `DocumentEntries`. `IndexEntries` are entries in the Knowledge Base that are not backed by an embedded raw text source, but rather by an entire Index or Data Stream. The user can set the data source name, the specific field to query (must be ELSER embeddings in this initial implementation), and a description for when the assistant should search this data source for Knowledge Base content. This essentially enables the user to create custom retrieval tools backed by their own data. The changes in this PR, as with the other recent KB enhancements, are behind the following feature flag: ``` xpack.securitySolution.enableExperimental: - 'assistantKnowledgeBaseByDefault' ``` however as code change is required to test the new mappings. For this you can update the `knowledgeBaseDataStream` in `x-pack/plugins/elastic_assistant/server/ai_assistant_service/index.ts` to ```ts this.knowledgeBaseDataStream = this.createDataStream({ resource: 'knowledgeBase', kibanaVersion: options.kibanaVersion, fieldMap: knowledgeBaseFieldMapV2, // Update this to the V2 mapping }); ``` Change set includes: - [X] ES Knowledge Base data stream schema and OAS has been updated to support `IndexEntries`. - [X] OAS schema files have been moved to the `/entries` sub-directory - [ ] Backend KB services have been updated to support `IndexEntries` - [X] Storage methods updated - [ ] Retrieval methods updated (will round out these endpoint when working the UI next) --- With these API changes, I've also introduced a few sample `*.http` files for easier development/testing. These files are supported out of the box in JetBrains IDE's or in VSCode with the [httpyac](https://httpyac.github.io/) (and many other) extensions. Since the configuration for these files includes a `-` in the name, that's why you'll see a few @elastic/kibana-operations files updated. You can read more about `http` files [here](https://www.jetbrains.com/help/webstorm/http-client-in-product-code-editor.html) and for the spec see this repo [here](https://github.com/JetBrains/http-request-in-editor-spec/blob/master/spec.md). If we find these useful, we could add support to our [OpenAPI Generator](https://openapi-generator.tech/docs/generators/jetbrains-http-client) to create these automatically. They currently live co-located next to the OAS and generated schema files here: ``` x-pack/packages/kbn-elastic-assistant-common/impl/schemas/knowledge_base/entries/bulk_crud_knowledge_base_entries_route.http x-pack/packages/kbn-elastic-assistant-common/impl/schemas/knowledge_base/entries/crud_knowledge_base_entries_route.http ``` and the main config here: ``` x-pack/packages/kbn-elastic-assistant-common/env/http-client.env.json ``` The `x-pack/packages/kbn-elastic-assistant-common/.gitignore` has been updated to ignore `http-client.private.env.json` files locally, which is how you can override the config as you'd like. This is helpful to add variables like `basePath` as below: ``` { "dev": { "basePath": "/kbn" } } ``` To use them, just open the corresponding `*.http` for the API you want to test, and click `Send`, and the response will open in another tab. Here is what that looks like for creating one of the new `IndexEntry` KB documents that have been introduced in this PR: <p align="center"> <img width="500" src="https://github.com/user-attachments/assets/c9e70d1a-28d2-4eb3-9853-ab6d8e1c7acf" /> </p> ### Checklist Delete any items that are not applicable to this PR. - [X] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials * Feature currently behind feature flag. Documentation to be added before flag is removed. Tracked in elastic/security-docs#5337 - [X] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

Adds useKnowledgeBaseEntries client hook and co-locates existing kb h…

5ad4bd6

…ooks

spong added release_note:skip Skip the PR/issue when compiling release notes Feature:Security Assistant Security Assistant Team:Security Generative AI Security Generative AI v8.15.0 labels Jun 6, 2024

spong self-assigned this Jun 6, 2024

spong mentioned this pull request Jun 7, 2024

[Security Assistant] Migrates to LangGraph and adds KB Tools #184554

Merged

3 tasks

spong added 4 commits June 7, 2024 13:21

Merge branch 'main' of github.com:elastic/kibana into kb-client-hooks

c798fa0

Adds find route and registers kb entries routes

1c82b9a

Merge branch 'main' of github.com:elastic/kibana into kb-client-hooks

d372a24

Merge branch 'main' of github.com:elastic/kibana into kb-client-hooks

366d056

spong commented Jun 11, 2024

View reviewed changes

spong added 2 commits June 11, 2024 14:25

Schema updates from merge conflict

27d7345

Adds create and delete client hooks and corresponding routes

edaf969

spong changed the title ~~[Security Assistant] Adds client hooks for Knowledge Base Entries~~ [Security Assistant] Adds client hooks and internal routes for managing Knowledge Base Entries Jun 12, 2024

spong commented Jun 12, 2024

View reviewed changes

spong added 2 commits June 11, 2024 23:47

Removes test buttons

6fd07f7

Merge branch 'main' of github.com:elastic/kibana into kb-client-hooks

eacceae

spong marked this pull request as ready for review June 12, 2024 05:47

spong requested a review from a team as a code owner June 12, 2024 05:47

spong requested a review from angorayc June 12, 2024 05:48

kibanamachine and others added 2 commits June 12, 2024 16:07

Merge branch 'main' into kb-client-hooks

6c36192

Merge branch 'main' into kb-client-hooks

af3e139

andrew-goldstein reviewed Jun 14, 2024

View reviewed changes

andrew-goldstein approved these changes Jun 14, 2024

View reviewed changes

Merge branch 'main' of github.com:elastic/kibana into kb-client-hooks

0ac5d0f

Merge branch 'main' into kb-client-hooks

24c5701

angorayc approved these changes Jun 17, 2024

View reviewed changes

kibanamachine and others added 3 commits June 17, 2024 11:56

Merge branch 'main' into kb-client-hooks

8645635

Merge branch 'main' into kb-client-hooks

5a62842

Merge branch 'main' into kb-client-hooks

5478087

spong enabled auto-merge (squash) June 18, 2024 23:33

spong merged commit 1b872fb into elastic:main Jun 19, 2024
42 checks passed

kibanamachine added the backport:skip This commit does not require backporting label Jun 19, 2024

spong mentioned this pull request Jun 20, 2024

[Security Assistant] Updates Knowledge Base ES Schema, OAS, and API to support IndexEntries #186566

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Assistant] Adds client hooks and internal routes for managing Knowledge Base Entries #184974

[Security Assistant] Adds client hooks and internal routes for managing Knowledge Base Entries #184974

spong commented Jun 6, 2024 •

edited

Loading

spong Jun 11, 2024

spong Jun 11, 2024

spong Jun 11, 2024

spong Jun 11, 2024

spong Jun 11, 2024

spong Jun 12, 2024

spong commented Jun 12, 2024

kibanamachine commented Jun 14, 2024

andrew-goldstein Jun 14, 2024

spong Jun 18, 2024

andrew-goldstein left a comment

spong commented Jun 14, 2024

angorayc left a comment

angorayc Jun 17, 2024

spong Jun 17, 2024

angorayc Jun 17, 2024

angorayc commented Jun 17, 2024

kibana-ci commented Jun 19, 2024

API count

ESLint disabled line counts

Total ESLint disabled count

[Security Assistant] Adds client hooks and internal routes for managing Knowledge Base Entries #184974

[Security Assistant] Adds client hooks and internal routes for managing Knowledge Base Entries #184974

Conversation

spong commented Jun 6, 2024 • edited Loading

Summary

Client Hooks & Routes

Sample Response

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spong commented Jun 12, 2024

kibanamachine commented Jun 14, 2024

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#6327

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrew-goldstein left a comment

Choose a reason for hiding this comment

spong commented Jun 14, 2024

angorayc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

angorayc commented Jun 17, 2024

kibana-ci commented Jun 19, 2024

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

API count

ESLint disabled line counts

Total ESLint disabled count

History

spong commented Jun 6, 2024 •

edited

Loading