Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds web crawler example to Knowledge Base doc #6176

Merged
merged 6 commits into from
Nov 20, 2024

Conversation

benironside
Copy link
Contributor

@benironside benironside commented Nov 18, 2024

Fixes #6157 — Adds a new section to the ESS Knowledge Base doc that shows how to add a threat intelligence feed to Knowledge Base using an Elasticsearch web crawler.

Preview: Knowledge Base (new section is at the bottom)

Copy link

A documentation preview will be available soon.

Request a new doc build by commenting
  • Rebuild this PR: run docs-build
  • Rebuild this PR and all Elastic docs: run docs-build rebuild

run docs-build is much faster than run docs-build rebuild. A rebuild should only be needed in rare situations.

If your PR continues to fail for an unknown reason, the doc build pipeline may be broken. Elastic employees can check the pipeline status here.

@acorretti acorretti force-pushed the 6157-knowledge-base-update branch from 4f74405 to 356789b Compare November 19, 2024 18:23
@acorretti acorretti self-requested a review as a code owner November 19, 2024 18:23
Copy link
Contributor

mergify bot commented Nov 19, 2024

This pull request is now in conflicts. Could you fix it @benironside? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 6157-knowledge-base-update upstream/6157-knowledge-base-update
git merge upstream/main
git push upstream 6157-knowledge-base-update

@acorretti acorretti removed their request for review November 19, 2024 18:54
@benironside benironside force-pushed the 6157-knowledge-base-update branch from 356789b to 5fad163 Compare November 19, 2024 19:31
Copy link
Contributor

@nastasha-solomon nastasha-solomon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a handful of minor edits and suggestions. Looks good overall!

@@ -57,15 +57,15 @@ When Knowledge Base is enabled, AI Assistant receives `open` or `acknowledged` a
To enable Knowledge Base for alerts:

. Ensure that knowledge base is <<enable-knowledge-base, enabled>>.
. Use the slider on the Security AI settings' Knowledge Base tab to select the number of alerts to send to AI Assistant. Click **Save**.
. Use the slider on the **Security AI settings** page's Knowledge Base tab to select the number of alerts to send to AI Assistant. Click **Save**.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that establishing the page first (instead of the slider on the tab) makes this a bit clearer.

Suggested change
. Use the slider on the **Security AI settings** page's Knowledge Base tab to select the number of alerts to send to AI Assistant. Click **Save**.
. On the **Security AI settings** page, go to the **Knowledge Base** tab and use the slider to select the number of alerts to send to AI Assistant. Click **Save**.

. Go to the **Manage Domains** tab. Select the domain associated with your new web crawler, then go the its **Crawl rules** tab and click **Add crawl rule**. For more information, refer to {enterprise-search-ref}/crawler-extraction-rules.html[Web crawler content extraction rules].
.. Under **Policy**, select `Allow`. Under **Rule**, select `Contains`. Under **Path pattern**, enter your path pattern, for example `threat-intelligence`. Click **Save**.
.. Click **Add crawl rule** again. Under **Policy**, select `Disallow`. Under **Rule**, select `Regex`. Under **Path pattern**, enter `.*`. Click **Save**. Make sure this rule appears below the rule created in the previous step on the list.
.. Click **Crawl**, then **Crawl all domains on this index**. A message appears that says "Successfully scheduled a sync, waiting for a connector to pick it up".
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is slightly easier to maintain because you don't have to update the text if/when the copy changes.

Suggested change
.. Click **Crawl**, then **Crawl all domains on this index**. A message appears that says "Successfully scheduled a sync, waiting for a connector to pick it up".
.. Click **Crawl**, then **Crawl all domains on this index**. A success message appears.

.. Under **Policy**, select `Allow`. Under **Rule**, select `Contains`. Under **Path pattern**, enter your path pattern, for example `threat-intelligence`. Click **Save**.
.. Click **Add crawl rule** again. Under **Policy**, select `Disallow`. Under **Rule**, select `Regex`. Under **Path pattern**, enter `.*`. Click **Save**. Make sure this rule appears below the rule created in the previous step on the list.
.. Click **Crawl**, then **Crawl all domains on this index**. A message appears that says "Successfully scheduled a sync, waiting for a connector to pick it up".
. The crawl process will take longer for larger data sources. Once it finishes, your new web crawler's index will contain documents provided by the crawler.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to be the result of an action, not so much a new step. Maybe add this to the end of step 5c, or at the end of step 5?

Copy link
Contributor

@nastasha-solomon nastasha-solomon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look good - thanks!

@benironside benironside merged commit c5b5f65 into main Nov 20, 2024
3 checks passed
mergify bot pushed a commit that referenced this pull request Nov 20, 2024
* Adds web crawler example to Knowledge Base doc

* various minor edits

* fixes note bug

* incorporates James feedback

* incorporates Charles' review

* incorporates review

(cherry picked from commit c5b5f65)
mergify bot pushed a commit that referenced this pull request Nov 20, 2024
* Adds web crawler example to Knowledge Base doc

* various minor edits

* fixes note bug

* incorporates James feedback

* incorporates Charles' review

* incorporates review

(cherry picked from commit c5b5f65)
benironside added a commit that referenced this pull request Nov 20, 2024
* Adds web crawler example to Knowledge Base doc

* various minor edits

* fixes note bug

* incorporates James feedback

* incorporates Charles' review

* incorporates review

(cherry picked from commit c5b5f65)

Co-authored-by: Benjamin Ironside Goldstein <91905639+benironside@users.noreply.github.com>
benironside added a commit that referenced this pull request Nov 20, 2024
* Adds web crawler example to Knowledge Base doc

* various minor edits

* fixes note bug

* incorporates James feedback

* incorporates Charles' review

* incorporates review

(cherry picked from commit c5b5f65)

Co-authored-by: Benjamin Ironside Goldstein <91905639+benironside@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Request] 8.16 Knowledge Base Update
2 participants