[Security Assistant] Adds Security Labs Knowledge Base content (elast…

…ic#184885)

## Summary

This PR adds the [Elastic Security
Labs](https://www.elastic.co/security-labs) content to the Security
Assistant Knowledge Base. Content is currently indexed and embedded from
its source `.mdx`, and we're using the same default chunking strategy as
with the ESQL documentation at the moment. Since some of this content is
quite large, we may want to explore other chunking strategies.

Now you can ask to fetch your recent alerts and ask if there's any
Elastic Security Labs content about them 🙂.

LangSmith trace
[here](https://smith.langchain.com/public/0e906acd-ab56-479d-b6d2-9c7538f54d45/r).

Malware alerts courtesy of @jamesspi's
https://github.com/jamesspi/ohmymalware project <img width="16"
src="https://user-images.githubusercontent.com/2946766/141219243-86619f7f-c300-4355-96f5-316d70aa6a0e.png"
/>

<p align="center">
<img width="375"
src="https://github.com/elastic/kibana/assets/2946766/750b6dc5-90b9-444e-be1d-c3df892dae61"
/> <img width="375"
src="https://github.com/elastic/kibana/assets/2946766/e052cea0-ada4-4099-8b97-be05c1379b00"
/>
</p>

<img width="2056" alt="image"
src="https://github.com/user-attachments/assets/9a58a064-deb3-4ad4-a3a9-bb72d056e8b4">

### Checklist

Delete any items that are not applicable to this PR.

- [X] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
* Feature currently behind feature flag. Documentation to be added
before flag is removed. Tracked in
elastic/security-docs#5337
- [X] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

---------

Co-authored-by: Patryk Kopycinski <contact@patrykkopycinski.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
(cherry picked from commit 3499fbb)

x-pack/packages/kbn-elastic-assistant-common/impl/schemas/knowledge_base/crud_kb_route.gen.ts

@@ -81,4 +81,5 @@ export const ReadKnowledgeBaseResponse = z.object({
   is_setup_available: z.boolean().optional(),
   is_setup_in_progress: z.boolean().optional(),
   pipeline_exists: z.boolean().optional(),
+  security_labs_exists: z.boolean().optional(),
 });

x-pack/packages/kbn-elastic-assistant-common/impl/schemas/knowledge_base/crud_kb_route.schema.yaml

@@ -78,6 +78,8 @@ paths:
                     type: boolean
                   pipeline_exists:
                     type: boolean
+                  security_labs_exists:
+                    type: boolean
         400:
           description: Generic Error
           content:

x-pack/packages/kbn-elastic-assistant/impl/assistant/api/knowledge_base/use_knowledge_base_status.tsx

@@ -90,6 +90,7 @@ export const isKnowledgeBaseSetup = (kbStatus: ReadKnowledgeBaseResponse | undef
   return (
     (kbStatus?.elser_exists &&
       kbStatus?.esql_exists &&
+      kbStatus?.security_labs_exists &&
       kbStatus?.index_exists &&
       kbStatus?.pipeline_exists) ??
     false

x-pack/packages/kbn-elastic-assistant/impl/assistant/chat_send/use_chat_send.tsx

@@ -63,7 +63,8 @@ export const useChatSend = ({
     kbStatus?.elser_exists &&
     kbStatus?.index_exists &&
     kbStatus?.pipeline_exists &&
-    kbStatus?.esql_exists;
+    kbStatus?.esql_exists &&
+    kbStatus?.security_labs_exists;
 
   // Handles sending latest user prompt to API
   const handleSendMessage = useCallback(

x-pack/packages/kbn-elastic-assistant/impl/knowledge_base/knowledge_base_settings.tsx

@@ -55,8 +55,13 @@ export const KnowledgeBaseSettings: React.FC<Props> = React.memo(
     // Resource enabled state
     const isElserEnabled = kbStatus?.elser_exists ?? false;
     const isESQLEnabled = kbStatus?.esql_exists ?? false;
+    const isSecurityLabsEnabled = kbStatus?.security_labs_exists ?? false;
     const isKnowledgeBaseSetup =
-      (isElserEnabled && isESQLEnabled && kbStatus?.index_exists && kbStatus?.pipeline_exists) ??
+      (isElserEnabled &&
+        isESQLEnabled &&
+        isSecurityLabsEnabled &&
+        kbStatus?.index_exists &&
+        kbStatus?.pipeline_exists) ??
       false;
     const isSetupInProgress = kbStatus?.is_setup_in_progress ?? false;
     const isSetupAvailable = kbStatus?.is_setup_available ?? false;

x-pack/packages/kbn-elastic-assistant/impl/knowledge_base/knowledge_base_settings_management/helpers.ts

@@ -12,13 +12,15 @@ import {
 } from '@kbn/elastic-assistant-common';
 import { z } from '@kbn/zod';
 
-export const isEsqlSystemEntry = (
+export const isSystemEntry = (
   entry: KnowledgeBaseEntryResponse
 ): entry is KnowledgeBaseEntryResponse & {
   type: DocumentEntryType;
-  kbResource: 'esql';
+  kbResource: 'esql' | 'security_labs';
 } => {
-  return entry.type === DocumentEntryType.value && entry.kbResource === 'esql';
+  return (
+    entry.type === DocumentEntryType.value && ['esql', 'security_labs'].includes(entry.kbResource)
+  );
 };
 
 export const isKnowledgeBaseEntryCreateProps = (

x-pack/packages/kbn-elastic-assistant/impl/knowledge_base/knowledge_base_settings_management/index.tsx

@@ -43,7 +43,7 @@ import { KnowledgeBaseSettings } from '../knowledge_base_settings';
 import { SetupKnowledgeBaseButton } from '../setup_knowledge_base_button';
 import { useDeleteKnowledgeBaseEntries } from '../../assistant/api/knowledge_base/entries/use_delete_knowledge_base_entries';
 import {
-  isEsqlSystemEntry,
+  isSystemEntry,
   isKnowledgeBaseEntryCreateProps,
   isKnowledgeBaseEntryResponse,
 } from './helpers';
@@ -152,13 +152,13 @@ export const KnowledgeBaseSettingsManagement: React.FC = React.memo(() => {
           openFlyout();
         },
         isDeleteEnabled: (entry: KnowledgeBaseEntryResponse) => {
-          return !isEsqlSystemEntry(entry);
+          return !isSystemEntry(entry);
         },
         onDeleteActionClicked: ({ id }: KnowledgeBaseEntryResponse) => {
           deleteEntry({ ids: [id] });
         },
         isEditEnabled: (entry: KnowledgeBaseEntryResponse) => {
-          return !isEsqlSystemEntry(entry);
+          return !isSystemEntry(entry);
         },
         onEditActionClicked: ({ id }: KnowledgeBaseEntryResponse) => {
           const entry = entries.data.find((e) => e.id === id);

x-pack/packages/kbn-elastic-assistant/impl/knowledge_base/knowledge_base_settings_management/use_knowledge_base_table.tsx

@@ -18,7 +18,7 @@ import { useAssistantContext } from '../../..';
 import * as i18n from './translations';
 import { BadgesColumn } from '../../assistant/common/components/assistant_settings_management/badges';
 import { useInlineActions } from '../../assistant/common/components/assistant_settings_management/inline_actions';
-import { isEsqlSystemEntry } from './helpers';
+import { isSystemEntry } from './helpers';
 
 export const useKnowledgeBaseTable = () => {
   const { currentUserAvatar } = useAssistantContext();
@@ -29,7 +29,7 @@ export const useKnowledgeBaseTable = () => {
       if (entry.kbResource === 'user') {
         return 'userAvatar';
       }
-      if (entry.kbResource === 'esql') {
+      if (['esql', 'security_labs'].includes(entry.kbResource)) {
         return 'logoElastic';
       }
       return 'visText';
@@ -83,8 +83,8 @@ export const useKnowledgeBaseTable = () => {
           render: (entry: KnowledgeBaseEntryResponse) => {
             // TODO: Look up user from `createdBy` id if privileges allow
             const userName = entry.users?.[0]?.name ?? 'Unknown';
-            const badgeItem = isEsqlSystemEntry(entry) ? 'Elastic' : userName;
-            const userImage = isEsqlSystemEntry(entry) ? (
+            const badgeItem = isSystemEntry(entry) ? 'Elastic' : userName;
+            const userImage = isSystemEntry(entry) ? (
               <EuiIcon
                 type={'logoElastic'}
                 css={css`
@@ -124,7 +124,7 @@ export const useKnowledgeBaseTable = () => {
         {
           name: i18n.COLUMN_ENTRIES,
           render: (entry: KnowledgeBaseEntryResponse) => {
-            return isEsqlSystemEntry(entry)
+            return isSystemEntry(entry)
               ? entry.text
               : entry.type === DocumentEntryType.value
               ? '1'

x-pack/packages/kbn-elastic-assistant/impl/knowledge_base/setup_knowledge_base_button.tsx

@@ -34,7 +34,8 @@ export const SetupKnowledgeBaseButton: React.FC<Props> = React.memo(({ display }
     kbStatus?.elser_exists &&
     kbStatus?.index_exists &&
     kbStatus?.pipeline_exists &&
-    kbStatus?.esql_exists;
+    kbStatus?.esql_exists &&
+    kbStatus?.security_labs_exists;
 
   const onInstallKnowledgeBase = useCallback(() => {
     setupKB(ESQL_RESOURCE);

x-pack/plugins/elastic_assistant/server/ai_assistant_data_clients/find.ts

@@ -5,7 +5,11 @@
  * 2.0.
  */
 
-import { MappingRuntimeFields, Sort } from '@elastic/elasticsearch/lib/api/types';
+import {
+  AggregationsAggregationContainer,
+  MappingRuntimeFields,
+  Sort,
+} from '@elastic/elasticsearch/lib/api/types';
 import { ElasticsearchClient, Logger } from '@kbn/core/server';
 
 import { estypes } from '@elastic/elasticsearch';
@@ -22,6 +26,7 @@ interface FindOptions {
   index: string;
   runtimeMappings?: MappingRuntimeFields | undefined;
   logger: Logger;
+  aggs?: Record<string, AggregationsAggregationContainer>;
 }
 
 export interface FindResponse<T> {
@@ -41,6 +46,7 @@ export const findDocuments = async <TSearchSchema>({
   fields,
   sortOrder,
   logger,
+  aggs,
 }: FindOptions): Promise<FindResponse<TSearchSchema>> => {
   const query = getQueryFilter({ filter });
   let sort: Sort | undefined;
@@ -67,6 +73,7 @@ export const findDocuments = async <TSearchSchema>({
       index,
       seq_no_primary_term: true,
       size: perPage,
+      aggs,
     });
     return {
       data: response,

x-pack/plugins/elastic_assistant/server/ai_assistant_data_clients/index.ts

@@ -99,13 +99,15 @@ export class AIAssistantDataClient {
     sortOrder,
     filter,
     fields,
+    aggs,
   }: {
     perPage: number;
     page: number;
     sortField?: string;
     sortOrder?: string;
     filter?: string;
     fields?: string[];
+    aggs?: Record<string, estypes.AggregationsAggregationContainer>;
   }): Promise<Promise<FindResponse<TSearchSchema>>> => {
     const esClient = await this.options.elasticsearchClientPromise;
     return findDocuments<TSearchSchema>({
@@ -118,6 +120,7 @@ export class AIAssistantDataClient {
       index: this.indexTemplateAndPattern.alias,
       sortOrder: sortOrder as estypes.SortOrder,
       logger: this.options.logger,
+      aggs,
     });
   };
 }

x-pack/plugins/elastic_assistant/server/ai_assistant_data_clients/knowledge_base/index.ts

@@ -35,13 +35,17 @@ import {
 } from './create_knowledge_base_entry';
 import { EsDocumentEntry, EsIndexEntry, EsKnowledgeBaseEntrySchema } from './types';
 import { transformESSearchToKnowledgeBaseEntry } from './transforms';
-import { ESQL_DOCS_LOADED_QUERY } from '../../routes/knowledge_base/constants';
+import {
+  ESQL_DOCS_LOADED_QUERY,
+  SECURITY_LABS_RESOURCE,
+} from '../../routes/knowledge_base/constants';
 import {
   getKBVectorSearchQuery,
   getStructuredToolForIndexEntry,
   isModelAlreadyExistsError,
 } from './helpers';
 import { getKBUserFilter } from '../../routes/knowledge_base/entries/utils';
+import { loadSecurityLabs } from '../../lib/langchain/content_loaders/security_labs_loader';
 
 /**
  * Params for when creating KbDataClient in Request Context Factory. Useful if needing to modify
@@ -203,9 +207,11 @@ export class AIAssistantKnowledgeBaseDataClient extends AIAssistantDataClient {
   public setupKnowledgeBase = async ({
     soClient,
     installEsqlDocs = true,
+    installSecurityLabsDocs = true,
   }: {
     soClient: SavedObjectsClientContract;
     installEsqlDocs?: boolean;
+    installSecurityLabsDocs?: boolean;
   }): Promise<void> => {
     if (this.options.getIsKBSetupInProgress()) {
       this.options.logger.debug('Knowledge Base setup already in progress');
@@ -257,6 +263,16 @@ export class AIAssistantKnowledgeBaseDataClient extends AIAssistantDataClient {
           this.options.logger.debug(`Knowledge Base docs already loaded!`);
         }
       }
+
+      if (installSecurityLabsDocs) {
+        const labsDocsLoaded = await this.isSecurityLabsDocsLoaded();
+        if (!labsDocsLoaded) {
+          this.options.logger.debug(`Loading Security Labs KB docs...`);
+          await loadSecurityLabs(this, this.options.logger);
+        } else {
+          this.options.logger.debug(`Security Labs Knowledge Base docs already loaded!`);
+        }
+      }
     } catch (e) {
       this.options.setIsKBSetupInProgress(false);
       this.options.logger.error(`Error setting up Knowledge Base: ${e.message}`);
@@ -352,6 +368,18 @@ export class AIAssistantKnowledgeBaseDataClient extends AIAssistantDataClient {
     return esqlDocs.length > 0;
   };
 
+  /**
+   * Returns if Security Labs KB docs have been loaded
+   */
+  public isSecurityLabsDocsLoaded = async (): Promise<boolean> => {
+    const securityLabsDocs = await this.getKnowledgeBaseDocumentEntries({
+      query: '',
+      kbResource: SECURITY_LABS_RESOURCE,
+      required: false,
+    });
+    return securityLabsDocs.length > 0;
+  };
+
   /**
    * Performs similarity search to retrieve LangChain Documents from the knowledge base
    */

x-pack/plugins/elastic_assistant/server/knowledge_base/security_labs/2022_elastic_global_threat_report_announcement.md

@@ -0,0 +1,15 @@
+---
+title: "2022 Elastic Global Threat Report Announcement"
+slug: "2022-elastic-global-threat-report-announcement"
+date: "2022-11-30"
+description: "Discover our latest findings & strategic recommendations to better stay informed of potential directions threat actors may focus on."
+author:
+  - slug: devon-kerr
+image: "gtr-blog-image-720x420.jpg"
+category:
+  - slug: reports
+---
+
+Today Elastic Security Labs celebrates another milestone: launching the [2022 Elastic Global Threat Report](https://www.elastic.co/explore/security-without-limits/global-threat-report), our inaugural summary of threat trends, forecasts, and recommendations. We analyzed millions of telemetry events from sources around the world to share these insights with you; all part of our continued commitment to transparency, and our mission to protect the world's data.
+
+You can find the report [here](https://www.elastic.co/explore/security-without-limits/global-threat-report), we're excited to share it with you.

x-pack/plugins/elastic_assistant/server/knowledge_base/security_labs/2022_elastic_global_threat_report_helping_security_leaders_navigate_todays_threat_landscape.md

@@ -0,0 +1,41 @@
+---
+title: "2022 Elastic Global Threat Report: Helping security leaders navigate today’s threat landscape"
+slug: "2022-elastic-global-threat-report-helping-security-leaders-navigate-todays-threat-landscape"
+date: "2022-11-30"
+description: "A significant percentage of all cyber threats achieve a degree of success against technical, procedural, and human mitigations. So what is a company to do in the face of such unfavorable odds? Find out in this article."
+author:
+  - slug: ken-exner
+image: "gtr-blog-image-720x420.jpg"
+category:
+  - slug: reports
+---
+
+As the threat landscape continues to evolve, cybersecurity stakes are growing exponentially higher for today’s organizations. Between Log4j, geopolitical tension, and increasing ransomware threats, security is not just at the top of the business agenda but also the societal agenda. Meanwhile, threat actors have adopted new capabilities and methods while increasing their cadence of activity.
+
+Threat detection and response has come a long way since the firewall dissolved and the cloud took center stage. AI and machine learning, for example, have been major contributors to the advancement of cybersecurity. Machine learning is being used to identify malicious behavior from bad actors by modeling network behavior and improving overall threat detection.
+
+What’s been difficult is the sea of sameness filled with vendors promising products to mitigate today’s threats while preparing for the next one. As the [2022 Elastic Global Threat Report](https://www.elastic.co/explore/security-without-limits/global-threat-report) outlines, a significant percentage of all threats achieve a degree of success against technical, procedural, and human mitigations. So what is a company to do in the face of such unfavorable odds? At Elastic, we believe there are several ingredients that are critical to managing today’s threat landscape.
+
+## Build a program, not just a tool
+
+Vendors need to start thinking about security products as more than software. They are part of a living, breathing program that takes care and feeding. For Elastic, it’s not just about shipping a solution; it’s about offering a holistic approach to security that happens to come with a great product. It’s sharing insights and best practices and creating a community focused on security data intelligence that extends the value of Elastic Security for customers.
+
+The 2022 Elastic Threat Report is an important part of that program, and we’re excited to share our knowledge with the community. In addition to vital information from the Elastic Security Labs team, the report provides actionable guidance to security practitioners about how to maximize positive outcomes for their organizations.
+
+## It takes an (open) community
+
+The foundation of any good program is a strong community that can support and foster it. Take Elastic’s commitment to open security, for example. The community born from vendors being transparent about their security controls, detection rules, and threat logic can be a force multiplier of best practices across the entire industry.
+
+When vendors engage their experts with experts from across the broader security community about new threats they’ve observed or innovative methods for detecting nuanced attacks, it creates greater scalability of system defenses — not just for the enterprise but also for their customers.
+
+For example, at Elastic we recently opened our Endpoint Security [protections-artifacts repo](https://github.com/elastic/protections-artifacts), adding to our already open [detection-rules repo](https://github.com/elastic/detection-rules/tree/main/rules), to foster further collaboration with our community and be transparent about how we protect users.
+
+## Treat the cause, not the symptom
+
+Despite the ever-growing threat landscape and the risks that it poses, many organizations are still treating security symptoms instead of the cause. Companies can no longer afford to keep the security team siloed and separate from the engineering team. The two functions must work closely to build products and solutions that can withstand the barrage of advancing threats.
+
+At Elastic, we design and build products with security in mind from the start, so it’s baked into every solution we ship to our customers. In fact, we take security so seriously that the office of InfoSec is part of the engineering organization.
+
+We hope that the 2022 Elastic Global Threat Report will help your understanding of the important shifts in the threat landscape, and provide the information you need to make your organization more resilient, prepared, and protected.
+
+[Download the 2022 Elastic Global Threat Report](https://www.elastic.co/explore/security-without-limits/global-threat-report).