[Security Assistant] Adds new Knowledge Base Management Settings UI

[spong]

Summary

This PR updates the Knowledge Base Management Settings page to use the new entries API introduced in #186566. Many thanks to @angorayc for her work on the Assistant Management Settings overhaul, and initial implementation of this new KB Management UI over in #186847.

Feature Flag & Setup

The changes in this PR, as with the other recent V2 KB enhancements, are behind the following feature flag:

xpack.securitySolution.enableExperimental:
  - 'assistantKnowledgeBaseByDefault'

They also require a code change in the AIAssistantService to enable the new mapping (since setup happens on plugin start before FF registration), so be sure to update fieldMap to knowledgeBaseFieldMapV2 below before testing:

This is no longer the case as of cdec104. Just changing the above feature flag is now sufficient, just note that if upgrading and the KB was previously setup, you'll need to manually delete the data stream (DELETE /_data_stream/.kibana-elastic-ai-assistant-knowledge-base-default) or the management table will be littered with the old ESQL docs instead of being a single aggregate entry.

Once configured, the new Knowledge Base Management Settings will become available in Stack Management. The old settings UI is currently still available via the Settings Modal, but will soon be removed and replaced with links to the new interface via the Assistant Settings Context Menu (replacing the existing cog). Please see the designs (Security GenAI / Unified AI Assistant) for all changes.

Important

There are no migrations in place between the legacy and v2 KB mappings, so be sure to start with a clean ES data directory.

Testing

To aid with developing the UI, I took the opportunity to start fleshing out the KB Entries API integration tests. These live in x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries, and are currently configured to only run on @ess, as running tiny_elser in serverless and MKI environments can be tricky (more on that later).

To start the server and run the tests, from the x-pack/test/security_solution_api_integration/ directory run yarn genai_kb_entries:server:ess, and once started, yarn genai_kb_entries:runner:ess.

Changes in support of testing

In order to setup the API integration tests for use with the Knowledge Base, some functional changes needed to be made to the assistant/config:

1. Since ELSER is a heavy model to run in CI, the ML folks have created pt_tiny_elser for use in testing. Unfortunately, the getELSER() helper off the ml client that we use to get the modelld for installing ELSER, ingest pipelines, etc, cannot be overridden (#193633), so we must have some other means of doing that. So to get things working in the test env, I've plumbed through an optional modelId override to the POST knowledge base route (/ internal/ elastic_assistant/ knowledge_base/{resource?}?modelId=pt_tiny_elser). This then overrides the aiAssistantService getELSER() function when fetching a kbDataClient using the request, which appears to be the only way to also trigger a reinitialization of the ingest pipeline (which required the modelId), since that usually only occurs on plugin start. If there is a cleaner way to perform this reinitialization, please let me know!

2. Turns out getService('ml').importTrainedModel() can't be run in test env's with ssl:true, which is the default security config. You can read more about that issue in #193477, but the current workaround is to turn off ssl for this specific test configuration, so that's why ess.config.ts looks a little different. If there's a better way to manage this config, also please let me know!

Additional notes

We don't currently have a securityAssistant API client/service to use in integration tests, so I've just been creating one-off functions using supertest for now. I don't have the bandwidth to work this now, but perhaps @MadameSheema / @muskangulati-qasource could lend a hand here? I did need to test multi-user and multi-space scenarios, so I ported over the same auth helpers I saw used in other suites. Would be nice if these were bundled into the client as well ala how the o11y folks have done it here. Perhaps this is also on the list of things for @maximpn to generate from OAS's.... 🙃

RBAC

In plumbing the UI, I've tried to place // TODO: KB-RBAC tags in all the places I came across that will require an RBAC check/change. This includes some of the API integration tests, which I currently have skipped as they would fail without RBAC.

Other notable changes

• There are now dedicated legacy and v2 helper functions when managing persistence/retrieval of knowledge base entries. This should help with tearing out the old KB later, and better readability now.
• I've tried to remove dependency on the ElasticsearchStore as much as possible. The store's only use should now be within tools as a retriever here, and in post_evaluate here. If we adopt the new naturalLanguageToESQL tool in 8.16 (or update our existing ESQL tool to use the kbDataClient for retrieval), we should be able to get rid of this entirely.
• Added a spaces_roles_users_data.http file for adding spaces, roles, users, and a sample slackbot index for use with sample IndexEntries here.

// TODO

In effort to make incremental progress and facilitate early knowledge share with @patrykkopycinski, I'm capping this PR where it's at, and so here are the remaining items to complete full integration of the new Knowledge Base Management Settings interface:

• Support Update action
• Move from EuiInMemoryTable
• Finalize Setup UI
• Cleanup Save loaders
• Plumb through {{knowledge_history}} prompt template and include use's required entries

All this work is behind the aforementioned feature flag and required code change, and this changeset has also been manually upgrade tested to ensure there are no issues that would impact the regularly scheduled serverless releases. This is more of a note to reviewers when testing that full functionality is not present.

Checklist

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
  • Feature currently behind feature flag. Documentation to be added before flag is removed. Tracked in [Request] 8.16 Security Assistant Custom Knowledge Bases security-docs#5337
• Unit or functional tests were updated or added to match the most common scenarios

[spong]

As noted in the description, we must run with ssl:false for certain ml API's to function. See #193477 for details.

[andrew-goldstein]

If the user is an admin (future RBAC setting enabled), they should see all users entries in KB settings, however their assistant sessions should only ever return their own and global entries. If the user is not an admin, they should only see their own and global entries in both KB settings and assistant sessions

Allowing (only) admin users to view the Private KB entries of other users feels inconsistent with the way conversations work:

• Conversations are scoped to a user + space (this applies to all users)
  • Even users with superuser ES role don't have a CRUD UI to manage other users conversations
• Private KB entries are (as expected) also scoped to a user + space, just like conversations
  • When users with the superuser role create private KB entries, those entries don't effect other users

What is the use case for an admin user changing a Private KB entry on behalf of another user, when those entries only effect the user who created them?

[spong]

What is the use case for an admin user changing a Private KB entry on behalf of another user, when those entries only effect the user who created them?

I was mistaken in my original message, you are correct here @andrew-goldstein -- thank you for questioning this behavior! 🙏

I have confirmed with @jamesspi that RBAC should behave as following:

There will be a new sub-feature added (next to the anonymization sub-feature), that says something like Manage Global Knowledge Base, and when enabled, this feature allows the user to create global entries that everyone in the given space can see, but not edit unless they also have this feature. This feature grants no visibility into the private entries that any individual user has created. So any private entry a user makes can only be seen by that user.

As discussed in our pair session with @patrykkopycinski, I will re-work the _find API to behave accordingly. Till RBAC is put in place, the default behavior will be that all users will have global privileges. So the functional change here is that other users' private entries will not be displayed.

I will update and un-skip this API integration test to ensure this behavior is correct so we have this confidence going forward.

edit: this is done and test un-skipped and new test added in 4d151a5

[andrew-goldstein]

nit: Consider memoizing this non-primitive value, because it's passed as a prop to a component below

[spong]

Will do! I will be extracting the whole sharing field into its own component in the next UI PR as it's used in both entry types.

[andrew-goldstein]

Consider disabling the New button when the KB hasn't been setup yet to avoid the error toaster in the screenshot below:

I'm also wondering if the Setup Knowledge Base CTA positioned at the end of the sentence in the screenshot above should be the same size text as the Learn more link, or perhaps be more prominent like it is in the assistant:

[spong]

Will disable the New button if not available 👍

And there's still some pending UI changes around the setup button. I chatted with @bojanasan and we decided to hide the table and show the same button as in the assistant as you recommend here.

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 4b3d3dd

Failed CI Steps

• Defend Workflows Cypress Tests on Serverless #5

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
integrationAssistant	557	559	+2
securitySolution	5817	5829	+12
total			+14

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/elastic-assistant-common	363	366	+3

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	20.4MB	20.5MB	+19.6KB

Unknown metric groups

API count

id	before	after	diff
@kbn/elastic-assistant-common	390	393	+3

ESLint disabled line counts

id	before	after	diff
@kbn/elastic-assistant	21	20	-1
elasticAssistant	36	35	-1
total			-2

References to deprecated APIs

id	before	after	diff
securitySolution	456	457	+1

Total ESLint disabled count

id	before	after	diff
@kbn/elastic-assistant	22	21	-1
elasticAssistant	36	35	-1
total			-2

History

• 💛 Build #237003 was flaky 44be735
• 💔 Build #236856 failed cdec104
• 💚 Build #236768 succeeded ac494d7
• 💛 Build #236549 was flaky 29decc2

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @spong

[andrew-goldstein]

Thanks @spong for the new settings UI! 🙏
✅ Desk tested locally
LGTM 🚀

[kibanamachine]

💔 All backports failed

Status	Branch	Result
❌	8.x	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 192665

Questions ?

Please refer to the Backport tool documentation

[spong]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation