Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ML] Add support for returning pt_tiny_elser from the ml.trainedModelsProvider.getELSER() helper function #193633

Open
spong opened this issue Sep 20, 2024 · 1 comment
Open

[ML] Add support for returning pt_tiny_elser from the ml.trainedModelsProvider.getELSER() helper function #193633

spong opened this issue Sep 20, 2024 · 1 comment
Labels
enhancement New value added to drive a business result Feature:3rd Party Models ML 3rd party models :ml

Comments

@spong
Copy link
Member

spong commented Sep 20, 2024

In plumbing through the Security Assistant Knowledge Base API integration tests (#192665), I ended up having to add support for a modelId override to our API's when interacting with the Knowledge Base as we currently use the ml.trainedModelsProvider.getELSER() helper (which does not return pt_tiny_elser in test environments) to get the modelId for setting up ELSER, ingest pipelines, retrievers, etc. The O11y Assistant folks also have an open issue (#192757) to do the same.

It would be nice if the getELSER() helper function below could either be overridden to return pt_tiny_elser in our test environments, or if our test environments could be updated such that the function returns pt_tiny_elser.

kibana/x-pack/plugins/ml/server/models/model_management/models_provider.ts

Lines 542 to 552 in 68a9244

/**
* Provides an ELSER model name and configuration for download based on the current cluster architecture.
* The current default version is 2. If running on Cloud it returns the Linux x86_64 optimized version.
* If any of the ML nodes run a different OS rather than Linux, or the CPU architecture isn't x86_64,
* a portable version of the model is returned.
*/
async getELSER(
options?: GetModelDownloadConfigOptions
): Promise<ModelDefinitionResponse> | never {
return await this.getCuratedModelConfig('elser', options);
}

Note

Perhaps the need for this will go away or change somewhat when we switch from managing ELSER via the trainedModels API to using a default inference endpoint (#192461).
@botelastic botelastic bot added the needs-team Issues missing a team label label Sep 20, 2024
@spong spong mentioned this issue Sep 20, 2024
Merged
8 tasks
@elasticmachine
Copy link
Contributor

Pinging @elastic/ml-ui (:ml)

@botelastic botelastic bot removed the needs-team Issues missing a team label label Sep 23, 2024
@peteharverson peteharverson added Feature:3rd Party Models ML 3rd party models enhancement New value added to drive a business result labels Sep 23, 2024
spong added a commit that referenced this issue Sep 25, 2024
@spong @kibanamachine @elasticmachine
[Security Assistant] Adds new Knowledge Base Management Settings UI (#…
63730ea 
…192665)

## Summary

This PR updates the Knowledge Base Management Settings page to use the
new `entries` API introduced in
#186566. Many thanks to @angorayc
for her work on the Assistant Management Settings overhaul, and initial
implementation of this new KB Management UI over in
#186847.

<p align="center">
<img width="600"
src="https://github.com/user-attachments/assets/0a82587e-f33c-45f1-9165-1a676d6db5fa"
/>
</p> 



### Feature Flag & Setup
The changes in this PR, as with the other [recent V2 KB
enhancements](#186566), are behind
the following feature flag:
```
xpack.securitySolution.enableExperimental:
  - 'assistantKnowledgeBaseByDefault'
```

~They also require a code change in the `AIAssistantService` to enable
the new mapping (since setup happens on plugin start before FF
registration), so be sure to update `fieldMap` to
`knowledgeBaseFieldMapV2` below before testing:~

This is no longer the case as of
[cdec104](cdec104).
Just changing the above feature flag is now sufficient, just note that
if upgrading and the KB was previously setup, you'll need to manually
delete the data stream (`DELETE
/_data_stream/.kibana-elastic-ai-assistant-knowledge-base-default`) or
the management table will be littered with the old ESQL docs instead of
being a single aggregate entry.

Once configured, the new Knowledge Base Management Settings will become
available in Stack Management. The old settings UI is currently still
available via the Settings Modal, but will soon be removed and replaced
with links to the new interface via the Assistant Settings Context Menu
(replacing the existing `cog`). Please see the designs ([Security
GenAI](https://www.figma.com/design/BMvpY9EhcPIaoOS7LSrkL0/%5B8.15%2C-%5D-GenAI-Security-Settings?node-id=51-25207&node-type=canvas&t=t3vZSPhMxQhScJVt-0)
/ [Unified AI
Assistant](https://www.figma.com/design/xN20zMRNtMlirWB6n9n1xJ/Unified-AI-Assistant-Settings?node-id=0-1&node-type=canvas&t=3RDYE7h2DjLlFlcN-0))
for all changes.

> [!IMPORTANT]
> There are no migrations in place between the legacy and v2 KB
mappings, so be sure to start with a clean ES data directory.

### Testing

To aid with developing the UI, I took the opportunity to start fleshing
out the KB Entries API integration tests. These live in
[x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries](https://github.com/spong/kibana/tree/7ae6be136ad992b2163df13b55118556b01b6cb9/x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries),
and are currently configured to only run on `@ess`, as running
`tiny_elser` in serverless and MKI environments can be tricky (more on
that later).

To start the server and run the tests, from the
`x-pack/test/security_solution_api_integration/` directory run `yarn
genai_kb_entries:server:ess`, and once started, `yarn
genai_kb_entries:runner:ess`.

##### Changes in support of testing

In order to setup the API integration tests for use with the Knowledge
Base, some functional changes needed to be made to the assistant/config:

1. Since ELSER is a heavy model to run in CI, the ML folks have created
`pt_tiny_elser` for use in testing. Unfortunately, the `getELSER()`
helper off the `ml` client that we use to get the `modelld` for
installing ELSER, ingest pipelines, etc, cannot be overridden
([#193633](#193633)), so we must
have some other means of doing that. So to get things working in the
test env, I've plumbed through an optional `modelId` override to the
POST knowledge base route (`/ internal/ elastic_assistant/
knowledge_base/{resource?}?modelId=pt_tiny_elser`). This then overrides
the aiAssistantService `getELSER()` function [when
fetching](https://github.com/elastic/kibana/blob/645b3b863be16d70b8a7130a84b248c19729c340/x-pack/plugins/elastic_assistant/server/ai_assistant_service/index.ts#L334-L354)
a `kbDataClient` using the request, which appears to be the only way to
also trigger a reinitialization of the ingest pipeline (which required
the `modelId`), since that usually only occurs on plugin start. If there
is a cleaner way to perform this reinitialization, please let me know!

2. Turns out
[`getService('ml').importTrainedModel()`](https://github.com/elastic/kibana/blob/f18224c6869ae52228da3764ca9a427106b872fb/x-pack/test/functional/services/ml/api.ts#L1575-L1587)
can't be run in test env's with `ssl:true`, which is the default
security config. You can read more about that issue in
[#193477](#193477), but the
current workaround is to turn off `ssl` for this specific test
configuration, so that's why
[`ess.config.ts`](https://github.com/spong/kibana/blob/cf73d4c7fcd69207a9625046456a94212da833c7/x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries/trial_license_complete_tier/configs/ess.config.ts#L22)
looks a little different. If there's a better way to manage this config,
also please let me know!

##### Additional notes

We don't currently have a `securityAssistant` API client/service to use
in integration tests, so I've just been creating one-off functions using
`supertest` for now. I don't have the bandwidth to work this now, but
perhaps @MadameSheema / @muskangulati-qasource could lend a hand here? I
did need to test multi-user and multi-space scenarios, so I ported over
the same [auth
helpers](https://github.com/elastic/kibana/tree/dc26f1012f35c2445028a87dcc8cb3f063e058b0/x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries/utils/auth)
I saw used in other suites. Would be nice if these were bundled into the
client as well ala how the o11y folks have done it
[here](https://github.com/elastic/kibana/blob/e9f23aa98e3abadd491be61b17e7daa3cc110cdb/x-pack/test/observability_ai_assistant_api_integration/tests/knowledge_base/knowledge_base.spec.ts#L27-L34).
Perhaps this is also on the list of things for @maximpn to generate from
OAS's.... 🙃

### RBAC
In plumbing the UI, I've tried to place `// TODO: KB-RBAC` tags in all
the places I came across that will require an RBAC check/change. This
includes some of the API integration tests, which I currently have
skipped as they would fail without RBAC.

### Other notable changes

* There are now dedicated `legacy` and `v2` helper functions when
managing persistence/retrieval of knowledge base entries. This should
help with tearing out the old KB later, and better readability now.
* I've tried to remove dependency on the `ElasticsearchStore` as much as
possible. The store's only use should now be within tools as a retriever
[here](https://github.com/elastic/kibana/blob/de89153368848397df823c062e907a607d347dff/x-pack/plugins/elastic_assistant/server/routes/helpers.ts#L397-L405),
and in post_evaluate
[here](https://github.com/elastic/kibana/blob/de89153368848397df823c062e907a607d347dff/x-pack/plugins/elastic_assistant/server/routes/evaluate/post_evaluate.ts#L170-L179).
If we adopt the new
[`naturalLanguageToESQL`](#192042)
tool in `8.16` (or update our existing ESQL tool to use the
`kbDataClient` for retrieval), we should be able to get rid of this
entirely.
* Added a
[`spaces_roles_users_data.http`](https://github.com/elastic/kibana/blob/7447394fe39d5e2e098c266c14875d3aa17d3067/x-pack/packages/kbn-elastic-assistant-common/impl/utils/spaces_roles_users_data.http#L1)
file for adding spaces, roles, users, and a sample `slackbot` index for
use with [sample `IndexEntries`
here](https://github.com/elastic/kibana/blob/7447394fe39d5e2e098c266c14875d3aa17d3067/x-pack/packages/kbn-elastic-assistant-common/impl/schemas/knowledge_base/entries/crud_knowledge_base_entries_route.http#L18-L56).

### // TODO
In effort to make incremental progress and facilitate early knowledge
share with @patrykkopycinski, I'm capping this PR where it's at, and so
here are the remaining items to complete full integration of the new
Knowledge Base Management Settings interface:

- [ ] Support `Update` action
- [ ] Move from `EuiInMemoryTable` 
- [ ] Finalize `Setup` UI
- [ ] Cleanup `Save` loaders
- [ ] Plumb through `{{knowledge_history}}` prompt template and include
use's `required` entries

All this work is behind the aforementioned feature flag and required
code change, and this changeset has also been manually upgrade tested to
ensure there are no issues that would impact the regularly scheduled
serverless releases. This is more of a note to reviewers when testing
that full functionality is not present.




### Checklist

- [X] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
* Feature currently behind feature flag. Documentation to be added
before flag is removed. Tracked in
elastic/security-docs#5337
- [X] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
spong added a commit to spong/kibana that referenced this issue Sep 25, 2024
@spong
[Security Assistant] Adds new Knowledge Base Management Settings UI (e…
9c6e7fc 
…lastic#192665)

## Summary

This PR updates the Knowledge Base Management Settings page to use the
new `entries` API introduced in
elastic#186566. Many thanks to @angorayc
for her work on the Assistant Management Settings overhaul, and initial
implementation of this new KB Management UI over in
elastic#186847.

<p align="center">
<img width="600"
src="https://github.com/user-attachments/assets/0a82587e-f33c-45f1-9165-1a676d6db5fa"
/>
</p>

### Feature Flag & Setup
The changes in this PR, as with the other [recent V2 KB
enhancements](elastic#186566), are behind
the following feature flag:
```
xpack.securitySolution.enableExperimental:
  - 'assistantKnowledgeBaseByDefault'
```

~They also require a code change in the `AIAssistantService` to enable
the new mapping (since setup happens on plugin start before FF
registration), so be sure to update `fieldMap` to
`knowledgeBaseFieldMapV2` below before testing:~

This is no longer the case as of
[cdec104](elastic@cdec104).
Just changing the above feature flag is now sufficient, just note that
if upgrading and the KB was previously setup, you'll need to manually
delete the data stream (`DELETE
/_data_stream/.kibana-elastic-ai-assistant-knowledge-base-default`) or
the management table will be littered with the old ESQL docs instead of
being a single aggregate entry.

Once configured, the new Knowledge Base Management Settings will become
available in Stack Management. The old settings UI is currently still
available via the Settings Modal, but will soon be removed and replaced
with links to the new interface via the Assistant Settings Context Menu
(replacing the existing `cog`). Please see the designs ([Security
GenAI](https://www.figma.com/design/BMvpY9EhcPIaoOS7LSrkL0/%5B8.15%2C-%5D-GenAI-Security-Settings?node-id=51-25207&node-type=canvas&t=t3vZSPhMxQhScJVt-0)
/ [Unified AI
Assistant](https://www.figma.com/design/xN20zMRNtMlirWB6n9n1xJ/Unified-AI-Assistant-Settings?node-id=0-1&node-type=canvas&t=3RDYE7h2DjLlFlcN-0))
for all changes.

> [!IMPORTANT]
> There are no migrations in place between the legacy and v2 KB
mappings, so be sure to start with a clean ES data directory.

### Testing

To aid with developing the UI, I took the opportunity to start fleshing
out the KB Entries API integration tests. These live in
[x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries](https://github.com/spong/kibana/tree/7ae6be136ad992b2163df13b55118556b01b6cb9/x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries),
and are currently configured to only run on `@ess`, as running
`tiny_elser` in serverless and MKI environments can be tricky (more on
that later).

To start the server and run the tests, from the
`x-pack/test/security_solution_api_integration/` directory run `yarn
genai_kb_entries:server:ess`, and once started, `yarn
genai_kb_entries:runner:ess`.

##### Changes in support of testing

In order to setup the API integration tests for use with the Knowledge
Base, some functional changes needed to be made to the assistant/config:

1. Since ELSER is a heavy model to run in CI, the ML folks have created
`pt_tiny_elser` for use in testing. Unfortunately, the `getELSER()`
helper off the `ml` client that we use to get the `modelld` for
installing ELSER, ingest pipelines, etc, cannot be overridden
([elastic#193633](elastic#193633)), so we must
have some other means of doing that. So to get things working in the
test env, I've plumbed through an optional `modelId` override to the
POST knowledge base route (`/ internal/ elastic_assistant/
knowledge_base/{resource?}?modelId=pt_tiny_elser`). This then overrides
the aiAssistantService `getELSER()` function [when
fetching](https://github.com/elastic/kibana/blob/645b3b863be16d70b8a7130a84b248c19729c340/x-pack/plugins/elastic_assistant/server/ai_assistant_service/index.ts#L334-L354)
a `kbDataClient` using the request, which appears to be the only way to
also trigger a reinitialization of the ingest pipeline (which required
the `modelId`), since that usually only occurs on plugin start. If there
is a cleaner way to perform this reinitialization, please let me know!

2. Turns out
[`getService('ml').importTrainedModel()`](https://github.com/elastic/kibana/blob/f18224c6869ae52228da3764ca9a427106b872fb/x-pack/test/functional/services/ml/api.ts#L1575-L1587)
can't be run in test env's with `ssl:true`, which is the default
security config. You can read more about that issue in
[elastic#193477](elastic#193477), but the
current workaround is to turn off `ssl` for this specific test
configuration, so that's why
[`ess.config.ts`](https://github.com/spong/kibana/blob/cf73d4c7fcd69207a9625046456a94212da833c7/x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries/trial_license_complete_tier/configs/ess.config.ts#L22)
looks a little different. If there's a better way to manage this config,
also please let me know!

##### Additional notes

We don't currently have a `securityAssistant` API client/service to use
in integration tests, so I've just been creating one-off functions using
`supertest` for now. I don't have the bandwidth to work this now, but
perhaps @MadameSheema / @muskangulati-qasource could lend a hand here? I
did need to test multi-user and multi-space scenarios, so I ported over
the same [auth
helpers](https://github.com/elastic/kibana/tree/dc26f1012f35c2445028a87dcc8cb3f063e058b0/x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries/utils/auth)
I saw used in other suites. Would be nice if these were bundled into the
client as well ala how the o11y folks have done it
[here](https://github.com/elastic/kibana/blob/e9f23aa98e3abadd491be61b17e7daa3cc110cdb/x-pack/test/observability_ai_assistant_api_integration/tests/knowledge_base/knowledge_base.spec.ts#L27-L34).
Perhaps this is also on the list of things for @maximpn to generate from
OAS's.... 🙃

### RBAC
In plumbing the UI, I've tried to place `// TODO: KB-RBAC` tags in all
the places I came across that will require an RBAC check/change. This
includes some of the API integration tests, which I currently have
skipped as they would fail without RBAC.

### Other notable changes

* There are now dedicated `legacy` and `v2` helper functions when
managing persistence/retrieval of knowledge base entries. This should
help with tearing out the old KB later, and better readability now.
* I've tried to remove dependency on the `ElasticsearchStore` as much as
possible. The store's only use should now be within tools as a retriever
[here](https://github.com/elastic/kibana/blob/de89153368848397df823c062e907a607d347dff/x-pack/plugins/elastic_assistant/server/routes/helpers.ts#L397-L405),
and in post_evaluate
[here](https://github.com/elastic/kibana/blob/de89153368848397df823c062e907a607d347dff/x-pack/plugins/elastic_assistant/server/routes/evaluate/post_evaluate.ts#L170-L179).
If we adopt the new
[`naturalLanguageToESQL`](elastic#192042)
tool in `8.16` (or update our existing ESQL tool to use the
`kbDataClient` for retrieval), we should be able to get rid of this
entirely.
* Added a
[`spaces_roles_users_data.http`](https://github.com/elastic/kibana/blob/7447394fe39d5e2e098c266c14875d3aa17d3067/x-pack/packages/kbn-elastic-assistant-common/impl/utils/spaces_roles_users_data.http#L1)
file for adding spaces, roles, users, and a sample `slackbot` index for
use with [sample `IndexEntries`
here](https://github.com/elastic/kibana/blob/7447394fe39d5e2e098c266c14875d3aa17d3067/x-pack/packages/kbn-elastic-assistant-common/impl/schemas/knowledge_base/entries/crud_knowledge_base_entries_route.http#L18-L56).

### // TODO
In effort to make incremental progress and facilitate early knowledge
share with @patrykkopycinski, I'm capping this PR where it's at, and so
here are the remaining items to complete full integration of the new
Knowledge Base Management Settings interface:

- [ ] Support `Update` action
- [ ] Move from `EuiInMemoryTable`
- [ ] Finalize `Setup` UI
- [ ] Cleanup `Save` loaders
- [ ] Plumb through `{{knowledge_history}}` prompt template and include
use's `required` entries

All this work is behind the aforementioned feature flag and required
code change, and this changeset has also been manually upgrade tested to
ensure there are no issues that would impact the regularly scheduled
serverless releases. This is more of a note to reviewers when testing
that full functionality is not present.

### Checklist

- [X] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
* Feature currently behind feature flag. Documentation to be added
before flag is removed. Tracked in
elastic/security-docs#5337
- [X] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
(cherry picked from commit 63730ea)
spong added a commit that referenced this issue Sep 26, 2024
@spong
[8.x] [Security Assistant] Adds new Knowledge Base Management Setting…
acd50da 
…s UI (#192665) (#194074)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Security Assistant] Adds new Knowledge Base Management Settings UI
(#192665)](#192665)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Garrett
Spong","email":"spong@users.noreply.github.com"},"sourceCommit":{"committedDate":"2024-09-25T20:38:18Z","message":"[Security
Assistant] Adds new Knowledge Base Management Settings UI
(#192665)\n\n## Summary\r\n\r\nThis PR updates the Knowledge Base
Management Settings page to use the\r\nnew `entries` API introduced
in\r\nhttps://github.com//pull/186566. Many thanks to
@angorayc\r\nfor her work on the Assistant Management Settings overhaul,
and initial\r\nimplementation of this new KB Management UI over
in\r\nhttps://github.com//pull/186847.\r\n\r\n<p
align=\"center\">\r\n<img
width=\"600\"\r\nsrc=\"https://github.com/user-attachments/assets/0a82587e-f33c-45f1-9165-1a676d6db5fa\"\r\n/>\r\n</p>
\r\n\r\n\r\n\r\n### Feature Flag & Setup\r\nThe changes in this PR, as
with the other [recent V2
KB\r\nenhancements](#186566), are
behind\r\nthe following feature
flag:\r\n```\r\nxpack.securitySolution.enableExperimental:\r\n -
'assistantKnowledgeBaseByDefault'\r\n```\r\n\r\n~They also require a
code change in the `AIAssistantService` to enable\r\nthe new mapping
(since setup happens on plugin start before FF\r\nregistration), so be
sure to update `fieldMap` to\r\n`knowledgeBaseFieldMapV2` below before
testing:~\r\n\r\nThis is no longer the case as
of\r\n[cdec104](https://github.com/elastic/kibana/pull/192665/commits/cdec10402f2e9b889598693f9f415c98ccd9855c).\r\nJust
changing the above feature flag is now sufficient, just note that\r\nif
upgrading and the KB was previously setup, you'll need to
manually\r\ndelete the data stream
(`DELETE\r\n/_data_stream/.kibana-elastic-ai-assistant-knowledge-base-default`)
or\r\nthe management table will be littered with the old ESQL docs
instead of\r\nbeing a single aggregate entry.\r\n\r\nOnce configured,
the new Knowledge Base Management Settings will become\r\navailable in
Stack Management. The old settings UI is currently still\r\navailable
via the Settings Modal, but will soon be removed and replaced\r\nwith
links to the new interface via the Assistant Settings Context
Menu\r\n(replacing the existing `cog`). Please see the designs
([Security\r\nGenAI](https://www.figma.com/design/BMvpY9EhcPIaoOS7LSrkL0/%5B8.15%2C-%5D-GenAI-Security-Settings?node-id=51-25207&node-type=canvas&t=t3vZSPhMxQhScJVt-0)\r\n/
[Unified
AI\r\nAssistant](https://www.figma.com/design/xN20zMRNtMlirWB6n9n1xJ/Unified-AI-Assistant-Settings?node-id=0-1&node-type=canvas&t=3RDYE7h2DjLlFlcN-0))\r\nfor
all changes.\r\n\r\n> [!IMPORTANT]\r\n> There are no migrations in place
between the legacy and v2 KB\r\nmappings, so be sure to start with a
clean ES data directory.\r\n\r\n### Testing\r\n\r\nTo aid with
developing the UI, I took the opportunity to start fleshing\r\nout the
KB Entries API integration tests. These live
in\r\n[x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries](https://github.com/spong/kibana/tree/7ae6be136ad992b2163df13b55118556b01b6cb9/x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries),\r\nand
are currently configured to only run on `@ess`, as
running\r\n`tiny_elser` in serverless and MKI environments can be tricky
(more on\r\nthat later).\r\n\r\nTo start the server and run the tests,
from the\r\n`x-pack/test/security_solution_api_integration/` directory
run `yarn\r\ngenai_kb_entries:server:ess`, and once started,
`yarn\r\ngenai_kb_entries:runner:ess`.\r\n\r\n##### Changes in support
of testing\r\n\r\nIn order to setup the API integration tests for use
with the Knowledge\r\nBase, some functional changes needed to be made to
the assistant/config:\r\n\r\n1. Since ELSER is a heavy model to run in
CI, the ML folks have created\r\n`pt_tiny_elser` for use in testing.
Unfortunately, the `getELSER()`\r\nhelper off the `ml` client that we
use to get the `modelld` for\r\ninstalling ELSER, ingest pipelines, etc,
cannot be
overridden\r\n([#193633](#193633)),
so we must\r\nhave some other means of doing that. So to get things
working in the\r\ntest env, I've plumbed through an optional `modelId`
override to the\r\nPOST knowledge base route (`/ internal/
elastic_assistant/\r\nknowledge_base/{resource?}?modelId=pt_tiny_elser`).
This then overrides\r\nthe aiAssistantService `getELSER()` function
[when\r\nfetching](https://github.com/elastic/kibana/blob/645b3b863be16d70b8a7130a84b248c19729c340/x-pack/plugins/elastic_assistant/server/ai_assistant_service/index.ts#L334-L354)\r\na
`kbDataClient` using the request, which appears to be the only way
to\r\nalso trigger a reinitialization of the ingest pipeline (which
required\r\nthe `modelId`), since that usually only occurs on plugin
start. If there\r\nis a cleaner way to perform this reinitialization,
please let me know!\r\n\r\n2. Turns
out\r\n[`getService('ml').importTrainedModel()`](https://github.com/elastic/kibana/blob/f18224c6869ae52228da3764ca9a427106b872fb/x-pack/test/functional/services/ml/api.ts#L1575-L1587)\r\ncan't
be run in test env's with `ssl:true`, which is the default\r\nsecurity
config. You can read more about that issue
in\r\n[#193477](#193477), but
the\r\ncurrent workaround is to turn off `ssl` for this specific
test\r\nconfiguration, so that's
why\r\n[`ess.config.ts`](https://github.com/spong/kibana/blob/cf73d4c7fcd69207a9625046456a94212da833c7/x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries/trial_license_complete_tier/configs/ess.config.ts#L22)\r\nlooks
a little different. If there's a better way to manage this
config,\r\nalso please let me know!\r\n\r\n##### Additional
notes\r\n\r\nWe don't currently have a `securityAssistant` API
client/service to use\r\nin integration tests, so I've just been
creating one-off functions using\r\n`supertest` for now. I don't have
the bandwidth to work this now, but\r\nperhaps @MadameSheema /
@muskangulati-qasource could lend a hand here? I\r\ndid need to test
multi-user and multi-space scenarios, so I ported over\r\nthe same
[auth\r\nhelpers](https://github.com/elastic/kibana/tree/dc26f1012f35c2445028a87dcc8cb3f063e058b0/x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries/utils/auth)\r\nI
saw used in other suites. Would be nice if these were bundled into
the\r\nclient as well ala how the o11y folks have done
it\r\n[here](https://github.com/elastic/kibana/blob/e9f23aa98e3abadd491be61b17e7daa3cc110cdb/x-pack/test/observability_ai_assistant_api_integration/tests/knowledge_base/knowledge_base.spec.ts#L27-L34).\r\nPerhaps
this is also on the list of things for @maximpn to generate
from\r\nOAS's.... 🙃\r\n\r\n### RBAC\r\nIn plumbing the UI, I've tried to
place `// TODO: KB-RBAC` tags in all\r\nthe places I came across that
will require an RBAC check/change. This\r\nincludes some of the API
integration tests, which I currently have\r\nskipped as they would fail
without RBAC.\r\n\r\n### Other notable changes\r\n\r\n* There are now
dedicated `legacy` and `v2` helper functions when\r\nmanaging
persistence/retrieval of knowledge base entries. This should\r\nhelp
with tearing out the old KB later, and better readability now.\r\n* I've
tried to remove dependency on the `ElasticsearchStore` as much
as\r\npossible. The store's only use should now be within tools as a
retriever\r\n[here](https://github.com/elastic/kibana/blob/de89153368848397df823c062e907a607d347dff/x-pack/plugins/elastic_assistant/server/routes/helpers.ts#L397-L405),\r\nand
in
post_evaluate\r\n[here](https://github.com/elastic/kibana/blob/de89153368848397df823c062e907a607d347dff/x-pack/plugins/elastic_assistant/server/routes/evaluate/post_evaluate.ts#L170-L179).\r\nIf
we adopt the
new\r\n[`naturalLanguageToESQL`](https://github.com/elastic/kibana/pull/192042)\r\ntool
in `8.16` (or update our existing ESQL tool to use the\r\n`kbDataClient`
for retrieval), we should be able to get rid of this\r\nentirely.\r\n*
Added
a\r\n[`spaces_roles_users_data.http`](https://github.com/elastic/kibana/blob/7447394fe39d5e2e098c266c14875d3aa17d3067/x-pack/packages/kbn-elastic-assistant-common/impl/utils/spaces_roles_users_data.http#L1)\r\nfile
for adding spaces, roles, users, and a sample `slackbot` index
for\r\nuse with [sample
`IndexEntries`\r\nhere](https://github.com/elastic/kibana/blob/7447394fe39d5e2e098c266c14875d3aa17d3067/x-pack/packages/kbn-elastic-assistant-common/impl/schemas/knowledge_base/entries/crud_knowledge_base_entries_route.http#L18-L56).\r\n\r\n###
// TODO\r\nIn effort to make incremental progress and facilitate early
knowledge\r\nshare with @patrykkopycinski, I'm capping this PR where
it's at, and so\r\nhere are the remaining items to complete full
integration of the new\r\nKnowledge Base Management Settings
interface:\r\n\r\n- [ ] Support `Update` action\r\n- [ ] Move from
`EuiInMemoryTable` \r\n- [ ] Finalize `Setup` UI\r\n- [ ] Cleanup `Save`
loaders\r\n- [ ] Plumb through `{{knowledge_history}}` prompt template
and include\r\nuse's `required` entries\r\n\r\nAll this work is behind
the aforementioned feature flag and required\r\ncode change, and this
changeset has also been manually upgrade tested to\r\nensure there are
no issues that would impact the regularly scheduled\r\nserverless
releases. This is more of a note to reviewers when testing\r\nthat full
functionality is not present.\r\n\r\n\r\n\r\n\r\n### Checklist\r\n\r\n-
[X] Any text added follows [EUI's
writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\r\nsentence case text and includes
[i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n-
[
]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas
added for features that require explanation or tutorials\r\n* Feature
currently behind feature flag. Documentation to be added\r\nbefore flag
is removed. Tracked
in\r\nhttps://github.com/elastic/security-docs/issues/5337\r\n- [X]
[Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>\r\nCo-authored-by:
Elastic Machine
<elasticmachine@users.noreply.github.com>","sha":"63730ea0c9d9b036a05cb919b25b6d19c2ea8f03","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Feature:Security
Assistant","Team:Security Generative
AI","v8.16.0","backport:version"],"number":192665,"url":"https://github.com/elastic/kibana/pull/192665","mergeCommit":{"message":"[Security
Assistant] Adds new Knowledge Base Management Settings UI
(#192665)\n\n## Summary\r\n\r\nThis PR updates the Knowledge Base
Management Settings page to use the\r\nnew `entries` API introduced
in\r\nhttps://github.com//pull/186566. Many thanks to
@angorayc\r\nfor her work on the Assistant Management Settings overhaul,
and initial\r\nimplementation of this new KB Management UI over
in\r\nhttps://github.com//pull/186847.\r\n\r\n<p
align=\"center\">\r\n<img
width=\"600\"\r\nsrc=\"https://github.com/user-attachments/assets/0a82587e-f33c-45f1-9165-1a676d6db5fa\"\r\n/>\r\n</p>
\r\n\r\n\r\n\r\n### Feature Flag & Setup\r\nThe changes in this PR, as
with the other [recent V2
KB\r\nenhancements](#186566), are
behind\r\nthe following feature
flag:\r\n```\r\nxpack.securitySolution.enableExperimental:\r\n -
'assistantKnowledgeBaseByDefault'\r\n```\r\n\r\n~They also require a
code change in the `AIAssistantService` to enable\r\nthe new mapping
(since setup happens on plugin start before FF\r\nregistration), so be
sure to update `fieldMap` to\r\n`knowledgeBaseFieldMapV2` below before
testing:~\r\n\r\nThis is no longer the case as
of\r\n[cdec104](https://github.com/elastic/kibana/pull/192665/commits/cdec10402f2e9b889598693f9f415c98ccd9855c).\r\nJust
changing the above feature flag is now sufficient, just note that\r\nif
upgrading and the KB was previously setup, you'll need to
manually\r\ndelete the data stream
(`DELETE\r\n/_data_stream/.kibana-elastic-ai-assistant-knowledge-base-default`)
or\r\nthe management table will be littered with the old ESQL docs
instead of\r\nbeing a single aggregate entry.\r\n\r\nOnce configured,
the new Knowledge Base Management Settings will become\r\navailable in
Stack Management. The old settings UI is currently still\r\navailable
via the Settings Modal, but will soon be removed and replaced\r\nwith
links to the new interface via the Assistant Settings Context
Menu\r\n(replacing the existing `cog`). Please see the designs
([Security\r\nGenAI](https://www.figma.com/design/BMvpY9EhcPIaoOS7LSrkL0/%5B8.15%2C-%5D-GenAI-Security-Settings?node-id=51-25207&node-type=canvas&t=t3vZSPhMxQhScJVt-0)\r\n/
[Unified
AI\r\nAssistant](https://www.figma.com/design/xN20zMRNtMlirWB6n9n1xJ/Unified-AI-Assistant-Settings?node-id=0-1&node-type=canvas&t=3RDYE7h2DjLlFlcN-0))\r\nfor
all changes.\r\n\r\n> [!IMPORTANT]\r\n> There are no migrations in place
between the legacy and v2 KB\r\nmappings, so be sure to start with a
clean ES data directory.\r\n\r\n### Testing\r\n\r\nTo aid with
developing the UI, I took the opportunity to start fleshing\r\nout the
KB Entries API integration tests. These live
in\r\n[x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries](https://github.com/spong/kibana/tree/7ae6be136ad992b2163df13b55118556b01b6cb9/x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries),\r\nand
are currently configured to only run on `@ess`, as
running\r\n`tiny_elser` in serverless and MKI environments can be tricky
(more on\r\nthat later).\r\n\r\nTo start the server and run the tests,
from the\r\n`x-pack/test/security_solution_api_integration/` directory
run `yarn\r\ngenai_kb_entries:server:ess`, and once started,
`yarn\r\ngenai_kb_entries:runner:ess`.\r\n\r\n##### Changes in support
of testing\r\n\r\nIn order to setup the API integration tests for use
with the Knowledge\r\nBase, some functional changes needed to be made to
the assistant/config:\r\n\r\n1. Since ELSER is a heavy model to run in
CI, the ML folks have created\r\n`pt_tiny_elser` for use in testing.
Unfortunately, the `getELSER()`\r\nhelper off the `ml` client that we
use to get the `modelld` for\r\ninstalling ELSER, ingest pipelines, etc,
cannot be
overridden\r\n([#193633](#193633)),
so we must\r\nhave some other means of doing that. So to get things
working in the\r\ntest env, I've plumbed through an optional `modelId`
override to the\r\nPOST knowledge base route (`/ internal/
elastic_assistant/\r\nknowledge_base/{resource?}?modelId=pt_tiny_elser`).
This then overrides\r\nthe aiAssistantService `getELSER()` function
[when\r\nfetching](https://github.com/elastic/kibana/blob/645b3b863be16d70b8a7130a84b248c19729c340/x-pack/plugins/elastic_assistant/server/ai_assistant_service/index.ts#L334-L354)\r\na
`kbDataClient` using the request, which appears to be the only way
to\r\nalso trigger a reinitialization of the ingest pipeline (which
required\r\nthe `modelId`), since that usually only occurs on plugin
start. If there\r\nis a cleaner way to perform this reinitialization,
please let me know!\r\n\r\n2. Turns
out\r\n[`getService('ml').importTrainedModel()`](https://github.com/elastic/kibana/blob/f18224c6869ae52228da3764ca9a427106b872fb/x-pack/test/functional/services/ml/api.ts#L1575-L1587)\r\ncan't
be run in test env's with `ssl:true`, which is the default\r\nsecurity
config. You can read more about that issue
in\r\n[#193477](#193477), but
the\r\ncurrent workaround is to turn off `ssl` for this specific
test\r\nconfiguration, so that's
why\r\n[`ess.config.ts`](https://github.com/spong/kibana/blob/cf73d4c7fcd69207a9625046456a94212da833c7/x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries/trial_license_complete_tier/configs/ess.config.ts#L22)\r\nlooks
a little different. If there's a better way to manage this
config,\r\nalso please let me know!\r\n\r\n##### Additional
notes\r\n\r\nWe don't currently have a `securityAssistant` API
client/service to use\r\nin integration tests, so I've just been
creating one-off functions using\r\n`supertest` for now. I don't have
the bandwidth to work this now, but\r\nperhaps @MadameSheema /
@muskangulati-qasource could lend a hand here? I\r\ndid need to test
multi-user and multi-space scenarios, so I ported over\r\nthe same
[auth\r\nhelpers](https://github.com/elastic/kibana/tree/dc26f1012f35c2445028a87dcc8cb3f063e058b0/x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries/utils/auth)\r\nI
saw used in other suites. Would be nice if these were bundled into
the\r\nclient as well ala how the o11y folks have done
it\r\n[here](https://github.com/elastic/kibana/blob/e9f23aa98e3abadd491be61b17e7daa3cc110cdb/x-pack/test/observability_ai_assistant_api_integration/tests/knowledge_base/knowledge_base.spec.ts#L27-L34).\r\nPerhaps
this is also on the list of things for @maximpn to generate
from\r\nOAS's.... 🙃\r\n\r\n### RBAC\r\nIn plumbing the UI, I've tried to
place `// TODO: KB-RBAC` tags in all\r\nthe places I came across that
will require an RBAC check/change. This\r\nincludes some of the API
integration tests, which I currently have\r\nskipped as they would fail
without RBAC.\r\n\r\n### Other notable changes\r\n\r\n* There are now
dedicated `legacy` and `v2` helper functions when\r\nmanaging
persistence/retrieval of knowledge base entries. This should\r\nhelp
with tearing out the old KB later, and better readability now.\r\n* I've
tried to remove dependency on the `ElasticsearchStore` as much
as\r\npossible. The store's only use should now be within tools as a
retriever\r\n[here](https://github.com/elastic/kibana/blob/de89153368848397df823c062e907a607d347dff/x-pack/plugins/elastic_assistant/server/routes/helpers.ts#L397-L405),\r\nand
in
post_evaluate\r\n[here](https://github.com/elastic/kibana/blob/de89153368848397df823c062e907a607d347dff/x-pack/plugins/elastic_assistant/server/routes/evaluate/post_evaluate.ts#L170-L179).\r\nIf
we adopt the
new\r\n[`naturalLanguageToESQL`](https://github.com/elastic/kibana/pull/192042)\r\ntool
in `8.16` (or update our existing ESQL tool to use the\r\n`kbDataClient`
for retrieval), we should be able to get rid of this\r\nentirely.\r\n*
Added
a\r\n[`spaces_roles_users_data.http`](https://github.com/elastic/kibana/blob/7447394fe39d5e2e098c266c14875d3aa17d3067/x-pack/packages/kbn-elastic-assistant-common/impl/utils/spaces_roles_users_data.http#L1)\r\nfile
for adding spaces, roles, users, and a sample `slackbot` index
for\r\nuse with [sample
`IndexEntries`\r\nhere](https://github.com/elastic/kibana/blob/7447394fe39d5e2e098c266c14875d3aa17d3067/x-pack/packages/kbn-elastic-assistant-common/impl/schemas/knowledge_base/entries/crud_knowledge_base_entries_route.http#L18-L56).\r\n\r\n###
// TODO\r\nIn effort to make incremental progress and facilitate early
knowledge\r\nshare with @patrykkopycinski, I'm capping this PR where
it's at, and so\r\nhere are the remaining items to complete full
integration of the new\r\nKnowledge Base Management Settings
interface:\r\n\r\n- [ ] Support `Update` action\r\n- [ ] Move from
`EuiInMemoryTable` \r\n- [ ] Finalize `Setup` UI\r\n- [ ] Cleanup `Save`
loaders\r\n- [ ] Plumb through `{{knowledge_history}}` prompt template
and include\r\nuse's `required` entries\r\n\r\nAll this work is behind
the aforementioned feature flag and required\r\ncode change, and this
changeset has also been manually upgrade tested to\r\nensure there are
no issues that would impact the regularly scheduled\r\nserverless
releases. This is more of a note to reviewers when testing\r\nthat full
functionality is not present.\r\n\r\n\r\n\r\n\r\n### Checklist\r\n\r\n-
[X] Any text added follows [EUI's
writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\r\nsentence case text and includes
[i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n-
[
]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas
added for features that require explanation or tutorials\r\n* Feature
currently behind feature flag. Documentation to be added\r\nbefore flag
is removed. Tracked
in\r\nhttps://github.com/elastic/security-docs/issues/5337\r\n- [X]
[Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>\r\nCo-authored-by:
Elastic Machine
<elasticmachine@users.noreply.github.com>","sha":"63730ea0c9d9b036a05cb919b25b6d19c2ea8f03"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/192665","number":192665,"mergeCommit":{"message":"[Security
Assistant] Adds new Knowledge Base Management Settings UI
(#192665)\n\n## Summary\r\n\r\nThis PR updates the Knowledge Base
Management Settings page to use the\r\nnew `entries` API introduced
in\r\nhttps://github.com//pull/186566. Many thanks to
@angorayc\r\nfor her work on the Assistant Management Settings overhaul,
and initial\r\nimplementation of this new KB Management UI over
in\r\nhttps://github.com//pull/186847.\r\n\r\n<p
align=\"center\">\r\n<img
width=\"600\"\r\nsrc=\"https://github.com/user-attachments/assets/0a82587e-f33c-45f1-9165-1a676d6db5fa\"\r\n/>\r\n</p>
\r\n\r\n\r\n\r\n### Feature Flag & Setup\r\nThe changes in this PR, as
with the other [recent V2
KB\r\nenhancements](#186566), are
behind\r\nthe following feature
flag:\r\n```\r\nxpack.securitySolution.enableExperimental:\r\n -
'assistantKnowledgeBaseByDefault'\r\n```\r\n\r\n~They also require a
code change in the `AIAssistantService` to enable\r\nthe new mapping
(since setup happens on plugin start before FF\r\nregistration), so be
sure to update `fieldMap` to\r\n`knowledgeBaseFieldMapV2` below before
testing:~\r\n\r\nThis is no longer the case as
of\r\n[cdec104](https://github.com/elastic/kibana/pull/192665/commits/cdec10402f2e9b889598693f9f415c98ccd9855c).\r\nJust
changing the above feature flag is now sufficient, just note that\r\nif
upgrading and the KB was previously setup, you'll need to
manually\r\ndelete the data stream
(`DELETE\r\n/_data_stream/.kibana-elastic-ai-assistant-knowledge-base-default`)
or\r\nthe management table will be littered with the old ESQL docs
instead of\r\nbeing a single aggregate entry.\r\n\r\nOnce configured,
the new Knowledge Base Management Settings will become\r\navailable in
Stack Management. The old settings UI is currently still\r\navailable
via the Settings Modal, but will soon be removed and replaced\r\nwith
links to the new interface via the Assistant Settings Context
Menu\r\n(replacing the existing `cog`). Please see the designs
([Security\r\nGenAI](https://www.figma.com/design/BMvpY9EhcPIaoOS7LSrkL0/%5B8.15%2C-%5D-GenAI-Security-Settings?node-id=51-25207&node-type=canvas&t=t3vZSPhMxQhScJVt-0)\r\n/
[Unified
AI\r\nAssistant](https://www.figma.com/design/xN20zMRNtMlirWB6n9n1xJ/Unified-AI-Assistant-Settings?node-id=0-1&node-type=canvas&t=3RDYE7h2DjLlFlcN-0))\r\nfor
all changes.\r\n\r\n> [!IMPORTANT]\r\n> There are no migrations in place
between the legacy and v2 KB\r\nmappings, so be sure to start with a
clean ES data directory.\r\n\r\n### Testing\r\n\r\nTo aid with
developing the UI, I took the opportunity to start fleshing\r\nout the
KB Entries API integration tests. These live
in\r\n[x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries](https://github.com/spong/kibana/tree/7ae6be136ad992b2163df13b55118556b01b6cb9/x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries),\r\nand
are currently configured to only run on `@ess`, as
running\r\n`tiny_elser` in serverless and MKI environments can be tricky
(more on\r\nthat later).\r\n\r\nTo start the server and run the tests,
from the\r\n`x-pack/test/security_solution_api_integration/` directory
run `yarn\r\ngenai_kb_entries:server:ess`, and once started,
`yarn\r\ngenai_kb_entries:runner:ess`.\r\n\r\n##### Changes in support
of testing\r\n\r\nIn order to setup the API integration tests for use
with the Knowledge\r\nBase, some functional changes needed to be made to
the assistant/config:\r\n\r\n1. Since ELSER is a heavy model to run in
CI, the ML folks have created\r\n`pt_tiny_elser` for use in testing.
Unfortunately, the `getELSER()`\r\nhelper off the `ml` client that we
use to get the `modelld` for\r\ninstalling ELSER, ingest pipelines, etc,
cannot be
overridden\r\n([#193633](#193633)),
so we must\r\nhave some other means of doing that. So to get things
working in the\r\ntest env, I've plumbed through an optional `modelId`
override to the\r\nPOST knowledge base route (`/ internal/
elastic_assistant/\r\nknowledge_base/{resource?}?modelId=pt_tiny_elser`).
This then overrides\r\nthe aiAssistantService `getELSER()` function
[when\r\nfetching](https://github.com/elastic/kibana/blob/645b3b863be16d70b8a7130a84b248c19729c340/x-pack/plugins/elastic_assistant/server/ai_assistant_service/index.ts#L334-L354)\r\na
`kbDataClient` using the request, which appears to be the only way
to\r\nalso trigger a reinitialization of the ingest pipeline (which
required\r\nthe `modelId`), since that usually only occurs on plugin
start. If there\r\nis a cleaner way to perform this reinitialization,
please let me know!\r\n\r\n2. Turns
out\r\n[`getService('ml').importTrainedModel()`](https://github.com/elastic/kibana/blob/f18224c6869ae52228da3764ca9a427106b872fb/x-pack/test/functional/services/ml/api.ts#L1575-L1587)\r\ncan't
be run in test env's with `ssl:true`, which is the default\r\nsecurity
config. You can read more about that issue
in\r\n[#193477](#193477), but
the\r\ncurrent workaround is to turn off `ssl` for this specific
test\r\nconfiguration, so that's
why\r\n[`ess.config.ts`](https://github.com/spong/kibana/blob/cf73d4c7fcd69207a9625046456a94212da833c7/x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries/trial_license_complete_tier/configs/ess.config.ts#L22)\r\nlooks
a little different. If there's a better way to manage this
config,\r\nalso please let me know!\r\n\r\n##### Additional
notes\r\n\r\nWe don't currently have a `securityAssistant` API
client/service to use\r\nin integration tests, so I've just been
creating one-off functions using\r\n`supertest` for now. I don't have
the bandwidth to work this now, but\r\nperhaps @MadameSheema /
@muskangulati-qasource could lend a hand here? I\r\ndid need to test
multi-user and multi-space scenarios, so I ported over\r\nthe same
[auth\r\nhelpers](https://github.com/elastic/kibana/tree/dc26f1012f35c2445028a87dcc8cb3f063e058b0/x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries/utils/auth)\r\nI
saw used in other suites. Would be nice if these were bundled into
the\r\nclient as well ala how the o11y folks have done
it\r\n[here](https://github.com/elastic/kibana/blob/e9f23aa98e3abadd491be61b17e7daa3cc110cdb/x-pack/test/observability_ai_assistant_api_integration/tests/knowledge_base/knowledge_base.spec.ts#L27-L34).\r\nPerhaps
this is also on the list of things for @maximpn to generate
from\r\nOAS's.... 🙃\r\n\r\n### RBAC\r\nIn plumbing the UI, I've tried to
place `// TODO: KB-RBAC` tags in all\r\nthe places I came across that
will require an RBAC check/change. This\r\nincludes some of the API
integration tests, which I currently have\r\nskipped as they would fail
without RBAC.\r\n\r\n### Other notable changes\r\n\r\n* There are now
dedicated `legacy` and `v2` helper functions when\r\nmanaging
persistence/retrieval of knowledge base entries. This should\r\nhelp
with tearing out the old KB later, and better readability now.\r\n* I've
tried to remove dependency on the `ElasticsearchStore` as much
as\r\npossible. The store's only use should now be within tools as a
retriever\r\n[here](https://github.com/elastic/kibana/blob/de89153368848397df823c062e907a607d347dff/x-pack/plugins/elastic_assistant/server/routes/helpers.ts#L397-L405),\r\nand
in
post_evaluate\r\n[here](https://github.com/elastic/kibana/blob/de89153368848397df823c062e907a607d347dff/x-pack/plugins/elastic_assistant/server/routes/evaluate/post_evaluate.ts#L170-L179).\r\nIf
we adopt the
new\r\n[`naturalLanguageToESQL`](https://github.com/elastic/kibana/pull/192042)\r\ntool
in `8.16` (or update our existing ESQL tool to use the\r\n`kbDataClient`
for retrieval), we should be able to get rid of this\r\nentirely.\r\n*
Added
a\r\n[`spaces_roles_users_data.http`](https://github.com/elastic/kibana/blob/7447394fe39d5e2e098c266c14875d3aa17d3067/x-pack/packages/kbn-elastic-assistant-common/impl/utils/spaces_roles_users_data.http#L1)\r\nfile
for adding spaces, roles, users, and a sample `slackbot` index
for\r\nuse with [sample
`IndexEntries`\r\nhere](https://github.com/elastic/kibana/blob/7447394fe39d5e2e098c266c14875d3aa17d3067/x-pack/packages/kbn-elastic-assistant-common/impl/schemas/knowledge_base/entries/crud_knowledge_base_entries_route.http#L18-L56).\r\n\r\n###
// TODO\r\nIn effort to make incremental progress and facilitate early
knowledge\r\nshare with @patrykkopycinski, I'm capping this PR where
it's at, and so\r\nhere are the remaining items to complete full
integration of the new\r\nKnowledge Base Management Settings
interface:\r\n\r\n- [ ] Support `Update` action\r\n- [ ] Move from
`EuiInMemoryTable` \r\n- [ ] Finalize `Setup` UI\r\n- [ ] Cleanup `Save`
loaders\r\n- [ ] Plumb through `{{knowledge_history}}` prompt template
and include\r\nuse's `required` entries\r\n\r\nAll this work is behind
the aforementioned feature flag and required\r\ncode change, and this
changeset has also been manually upgrade tested to\r\nensure there are
no issues that would impact the regularly scheduled\r\nserverless
releases. This is more of a note to reviewers when testing\r\nthat full
functionality is not present.\r\n\r\n\r\n\r\n\r\n### Checklist\r\n\r\n-
[X] Any text added follows [EUI's
writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\r\nsentence case text and includes
[i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n-
[
]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas
added for features that require explanation or tutorials\r\n* Feature
currently behind feature flag. Documentation to be added\r\nbefore flag
is removed. Tracked
in\r\nhttps://github.com/elastic/security-docs/issues/5337\r\n- [X]
[Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>\r\nCo-authored-by:
Elastic Machine
<elasticmachine@users.noreply.github.com>","sha":"63730ea0c9d9b036a05cb919b25b6d19c2ea8f03"}},{"branch":"8.x","label":"v8.16.0","labelRegex":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New value added to drive a business result Feature:3rd Party Models ML 3rd party models :ml
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@spong @peteharverson @elasticmachine