[Security Assistant] Updates Knowledge Base ES Schema, OAS, and API t…

…o support `IndexEntries` (#186566)

## Summary

This is a follow-up to #184974
that updates the KB Entries API's (and underlying schema) to support
`IndexEntries` in addition to `DocumentEntries`. `IndexEntries` are
entries in the Knowledge Base that are not backed by an embedded raw
text source, but rather by an entire Index or Data Stream. The user can
set the data source name, the specific field to query (must be ELSER
embeddings in this initial implementation), and a description for when
the assistant should search this data source for Knowledge Base content.
This essentially enables the user to create custom retrieval tools
backed by their own data.


The changes in this PR, as with the other recent KB enhancements, are
behind the following feature flag:
```
xpack.securitySolution.enableExperimental:
  - 'assistantKnowledgeBaseByDefault'
```

however as code change is required to test the new mappings. For this
you can update the `knowledgeBaseDataStream` in
`x-pack/plugins/elastic_assistant/server/ai_assistant_service/index.ts`
to

```ts
    this.knowledgeBaseDataStream = this.createDataStream({
      resource: 'knowledgeBase',
      kibanaVersion: options.kibanaVersion,
      fieldMap: knowledgeBaseFieldMapV2, // Update this to the V2 mapping
    });
```

Change set includes:
- [X] ES Knowledge Base data stream schema and OAS has been updated to
support `IndexEntries`.
- [X] OAS schema files have been moved to the `/entries` sub-directory
- [ ] Backend KB services have been updated to support `IndexEntries`
  - [X] Storage methods updated
- [ ] Retrieval methods updated (will round out these endpoint when
working the UI next)

---

With these API changes, I've also introduced a few sample `*.http` files
for easier development/testing. These files are supported out of the box
in JetBrains IDE's or in VSCode with the
[httpyac](https://httpyac.github.io/) (and many other) extensions. Since
the configuration for these files includes a `-` in the name, that's why
you'll see a few @elastic/kibana-operations files updated.

You can read more about `http` files
[here](https://www.jetbrains.com/help/webstorm/http-client-in-product-code-editor.html)
and for the spec see this repo
[here](https://github.com/JetBrains/http-request-in-editor-spec/blob/master/spec.md).
If we find these useful, we could add support to our [OpenAPI
Generator](https://openapi-generator.tech/docs/generators/jetbrains-http-client)
to create these automatically. They currently live co-located next to
the OAS and generated schema files here:

```
x-pack/packages/kbn-elastic-assistant-common/impl/schemas/knowledge_base/entries/bulk_crud_knowledge_base_entries_route.http
x-pack/packages/kbn-elastic-assistant-common/impl/schemas/knowledge_base/entries/crud_knowledge_base_entries_route.http
```

and the main config here:

```
x-pack/packages/kbn-elastic-assistant-common/env/http-client.env.json
```

The `x-pack/packages/kbn-elastic-assistant-common/.gitignore` has been
updated to ignore `http-client.private.env.json` files locally, which is
how you can override the config as you'd like. This is helpful to add
variables like `basePath` as below:

```
{
  "dev": {
    "basePath": "/kbn"
  }
}
```

To use them, just open the corresponding `*.http` for the API you want
to test, and click `Send`, and the response will open in another tab.
Here is what that looks like for creating one of the new `IndexEntry` KB
documents that have been introduced in this PR:

<p align="center">
<img width="500"
src="https://github.com/user-attachments/assets/c9e70d1a-28d2-4eb3-9853-ab6d8e1c7acf"
/>
</p> 





### Checklist

Delete any items that are not applicable to this PR.

- [X] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
* Feature currently behind feature flag. Documentation to be added
before flag is removed. Tracked in
elastic/security-docs#5337
- [X] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

.buildkite/ftr_security_serverless_configs.yml

@@ -77,6 +77,8 @@ enabled:
   - x-pack/test/security_solution_api_integration/test_suites/detections_response/user_roles/trial_license_complete_tier/configs/serverless.config.ts
   - x-pack/test/security_solution_api_integration/test_suites/genai/nlp_cleanup_task/trial_license_complete_tier/configs/serverless.config.ts
   - x-pack/test/security_solution_api_integration/test_suites/genai/nlp_cleanup_task/basic_license_essentials_tier/configs/serverless.config.ts
+  - x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries/trial_license_complete_tier/configs/ess.config.ts
+  - x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries/trial_license_complete_tier/configs/serverless.config.ts
   - x-pack/test/security_solution_api_integration/test_suites/entity_analytics/risk_engine/trial_license_complete_tier/configs/serverless.config.ts
   - x-pack/test/security_solution_api_integration/test_suites/entity_analytics/risk_engine/basic_license_essentials_tier/configs/serverless.config.ts
   - x-pack/test/security_solution_api_integration/test_suites/lists_and_exception_lists/exception_lists_items/trial_license_complete_tier/configs/serverless.config.ts

packages/kbn-cli-dev-mode/src/watcher.ts

@@ -95,7 +95,7 @@ export class Watcher {
         ignore: [
           '**/{node_modules,target,public,coverage,__*__}/**',
           '**/*.{test,spec,story,stories}.*',
-          '**/*.{md,sh,txt}',
+          '**/*.{http,md,sh,txt}',
           '**/debug.log',
         ],
       }

src/dev/build/tasks/copy_legacy_source_task.ts

@@ -38,9 +38,10 @@ export const CopyLegacySource: Task = {
       '!**/jest*',
       '!**/*.{story,stories}.{js,ts}',
       '!**/{test_mocks,stubs}.ts',
-      '!**/*.{scss,console,d.ts,sh,md,mdx,asciidoc,docnav.json}',
+      '!**/*.{scss,console,d.ts,sh,md,mdx,asciidoc,docnav.json,http}',
       '!**/*.{test,test.mocks,mock,mocks,spec}.*',
       '!**/{packages,dev_docs,docs,public,__stories__,storybook,.storybook,ftr_e2e,e2e,scripts,test,tests,test_resources,test_data,__tests__,manual_tests,__jest__,__snapshots__,__mocks__,mock_responses,mocks,fixtures,__fixtures__,cypress,integration_tests}/**',
+      '!**/http-client.env.json',
 
       // explicitly exclude every package directory outside of the root packages dir
       ...getPackages(config.resolveFromRepo('.')).flatMap((p) =>

src/dev/precommit_hook/casing_check_config.js

@@ -83,6 +83,9 @@ export const IGNORE_FILE_GLOBS = [
 
   // ecs templates
   '**/ecs/fields/**/*',
+
+  // Support for including http-client.env.json configurations
+  '**/http-client.env.json',
 ];
 
 /**

x-pack/packages/kbn-elastic-assistant-common/.gitignore

@@ -0,0 +1,2 @@
+# Private configs for .http files
+http-client.private.env.json

x-pack/packages/kbn-elastic-assistant-common/env/README.md

@@ -0,0 +1,33 @@
+With https://github.com/elastic/kibana/pull/186566, we've introduced a few sample `*.http` files for easier development/testing. These files are supported out of the box in JetBrains IDE's or in VSCode with the [httpyac](https://httpyac.github.io/) (and many other) extensions. Since the configuration for these files includes a `-` in the name, a few @elastic/kibana-operations files have been updated to exclude them from checks and being included in the distribution.
+
+You can read more about `http` files [here](https://www.jetbrains.com/help/webstorm/http-client-in-product-code-editor.html) and for the spec see this repo [here](https://github.com/JetBrains/http-request-in-editor-spec/blob/master/spec.md). If we find these useful, we could add support to our [OpenAPI Generator](https://openapi-generator.tech/docs/generators/jetbrains-http-client) to create these automatically. They currently live co-located next to the OAS and generated schema files here:
+
+```
+x-pack/packages/kbn-elastic-assistant-common/impl/schemas/knowledge_base/entries/bulk_crud_knowledge_base_entries_route.http
+x-pack/packages/kbn-elastic-assistant-common/impl/schemas/knowledge_base/entries/crud_knowledge_base_entries_route.http
+```
+
+and the main config here in this directory:
+
+```
+x-pack/packages/kbn-elastic-assistant-common/env/http-client.env.json
+```
+
+The `x-pack/packages/kbn-elastic-assistant-common/.gitignore` has been updated to ignore `http-client.private.env.json` files locally, which is how you can override the config as you'd like. This is helpful to add variables like `basePath` as below:
+
+```
+{
+  "dev": {
+    "basePath": "/kbn"
+  }
+}
+```
+
+To use them, just open the corresponding `*.http` for the API you want to test, and click `Send`, and the response will open in another tab. Here is what that looks like for creating one of the new `IndexEntry` KB documents that have been introduced in the initial PR:
+
+<p align="center">
+  <img width="500" src="https://github.com/user-attachments/assets/c9e70d1a-28d2-4eb3-9853-ab6d8e1c7acf" />
+</p> 
+
+
+For continuing this effort, https://github.com/elastic/kibana/issues/192386 has been created.

x-pack/packages/kbn-elastic-assistant-common/env/http-client.env.json

@@ -0,0 +1,15 @@
+{
+  "dev": {
+    "host": "localhost",
+    "port": "5601",
+    "basePath": "",
+    "elasticApiVersion": "1",
+    "auth": {
+      "username": "elastic",
+      "password": "changeme"
+    },
+    "appContext": {
+      "security": "%7B%22type%22%3A%22application%22%2C%22name%22%3A%22securitySolutionUI%22%2C%22url%22%3A%22%2Fkbn%2Fapp%2Fsecurity%22%7D"
+    }
+  }
+}

x-pack/packages/kbn-elastic-assistant-common/impl/schemas/index.ts

@@ -49,10 +49,10 @@ export * from './actions_connector/post_actions_connector_execute_route.gen';
 
 // Knowledge Base Schemas
 export * from './knowledge_base/crud_kb_route.gen';
-export * from './knowledge_base/bulk_crud_knowledge_base_route.gen';
-export * from './knowledge_base/common_attributes.gen';
-export * from './knowledge_base/crud_knowledge_base_route.gen';
-export * from './knowledge_base/find_knowledge_base_entries_route.gen';
+export * from './knowledge_base/entries/bulk_crud_knowledge_base_entries_route.gen';
+export * from './knowledge_base/entries/common_attributes.gen';
+export * from './knowledge_base/entries/crud_knowledge_base_entries_route.gen';
+export * from './knowledge_base/entries/find_knowledge_base_entries_route.gen';
 
 export * from './prompts/find_prompts_route.gen';
 export { PromptResponse, PromptTypeEnum } from './prompts/bulk_crud_prompts_route.gen';

x-pack/packages/kbn-elastic-assistant-common/impl/schemas/knowledge_base/entries/bulk_crud_knowledge_base_entries_route.http

@@ -0,0 +1,13 @@
+### Empty Bulk Action
+POST http://{{host}}:{{port}}{{basePath}}/internal/elastic_assistant/knowledge_base/entries/_bulk_action
+kbn-xsrf: "true"
+Content-Type: application/json
+Elastic-Api-Version: {{elasticApiVersion}}
+Authorization: Basic {{auth.username}} {{auth.password}}
+X-Kbn-Context: {{appContext.security}}
+
+{
+  "delete": {},
+  "create": [],
+  "update": []
+}