Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] Adjust build-release CLI and fix links when generating security docs #3434

Merged
merged 15 commits into from
Feb 12, 2024

Conversation

terrancedejesus
Copy link
Contributor

@terrancedejesus terrancedejesus commented Feb 6, 2024

Issues

Summary

This pull request adjusts the build-release CLI command and the IntegrationRuleDetail.generate() method to account for link formatting issues when converted to asciidoc. Below is a more thorough breakdown of what has changed and why:

  • --add-historical was removed as an argument from build-release command. Since 8.7, historical rules are required for all packages. Since we release current-3 and are at 8.12-8.9, we can always add historical rules by default. Note - this will not effect backporting negatively as we have historical packages for 8.3+.
  • --generate-docs argument has been added to build-release. This is to optionally generate the MDX security docs which was built for a future integration into the improved docs site. It is default to no for now as it is not necessary when running this command at this time.
  • In IntegrationRuleDetail.generate() we have logic to build out the sections of a rule's security doc. These are the attributes of the rule itself. We had not accounted for the setup field prior to this, thus IntegrationRuleDetail.setup_str() was added to add this.
  • Currently in a rule's note or setup field, we add markdown formatted URL links, however, these are not valid in ASCIIDoc an thus break during CI checks in the security-docs repository. As a result, each setup_str() and note_str() methods have been updated to replace [title](url) with url if found. This will format it correctly in the docs so it does not break. Note that investigation guide (note) and setup are code blocks in each rule detail page, thus they do not render Elastic security doc formatting and thus are not compatible with {security-guide}/URI link, which is why we are just giving the URL. - We can focus on reconciliation with doc migration strategy.
  • The regex pattern used should replace ONLY any elastic related URLs in the note or setup fields as these are checked for anchor validation by the CI job, every other URL format is reserved.
[[prebuilt-rule-8-12-3-file-made-immutable-by-chattr]]
=== File made Immutable by Chattr

Detects a file being made immutable using the chattr binary. Making a file immutable means it cannot be deleted or renamed, no link can be created to this file, most of the file's metadata can not be modified, and the file can not be opened in write mode. Threat actors will commonly utilize this to prevent tampering or modification of their malicious files or any system files they have modified for purposes of persistence (e.g .ssh, /etc/passwd, etc.).

*Rule type*: eql

*Rule indices*: 

* auditbeat-*
* logs-endpoint.events.*
* endgame-*

*Severity*: medium

*Risk score*: 47

*Runs every*: 5m

*Searches indices from*: now-9m ({ref}/common-options.html#date-math[Date Math format], see also <<rule-schedule, `Additional look-back time`>>)

*Maximum alerts per execution*: 33

*References*: None

*Tags*: 

* Domain: Endpoint
* OS: Linux
* Use Case: Threat Detection
* Tactic: Defense Evasion
* Data Source: Elastic Endgame
* Data Source: Elastic Defend

*Version*: 110

*Rule authors*: 

* Elastic

*Rule license*: Elastic License v2


==== Setup


[source, markdown]
----------------------------------

This rule requires data coming in from one of the following integrations:
- Elastic Defend
- Auditbeat

### Elastic Defend Integration Setup
Elastic Defend is integrated into the Elastic Agent using Fleet. Upon configuration, the integration allows the Elastic Agent to monitor events on your host and send data to the Elastic Security app.

#### Prerequisite Requirements:
- Fleet is required for Elastic Defend.
- To configure Fleet Server refer to the https://www.elastic.co/guide/en/fleet/current/fleet-server.html.

#### The following steps should be executed in order to add the Elastic Defend integration on a Linux System:
- Go to the Kibana home page and click "Add integrations".
- In the query bar, search for "Elastic Defend" and select the integration to see more details about it.
- Click "Add Elastic Defend".
- Configure the integration name and optionally add a description.
- Select the type of environment you want to protect, either "Traditional Endpoints" or "Cloud Workloads".
- Select a configuration preset. Each preset comes with different default settings for Elastic Agent, you can further customize these later by configuring the Elastic Defend integration policy. https://www.elastic.co/guide/en/security/current/configure-endpoint-integration-policy.html.
- We suggest selecting "Complete EDR (Endpoint Detection and Response)" as a configuration setting, that provides "All events; all preventions"
- Enter a name for the agent policy in "New agent policy name". If other agent policies already exist, you can click the "Existing hosts" tab and select an existing policy instead.
For more details on Elastic Agent configuration settings, refer to the https://www.elastic.co/guide/en/fleet/8.10/agent-policy.html.
- Click "Save and Continue".
- To complete the integration, select "Add Elastic Agent to your hosts" and continue to the next section to install the Elastic Agent on your hosts.
For more details on Elastic Defend refer to the https://www.elastic.co/guide/en/security/current/install-endpoint.html.

### Auditbeat Setup
Auditbeat is a lightweight shipper that you can install on your servers to audit the activities of users and processes on your systems. For example, you can use Auditbeat to collect and centralize audit events from the Linux Audit Framework. You can also use Auditbeat to detect changes to critical files, like binaries and configuration files, and identify potential security policy violations.

#### The following steps should be executed in order to add the Auditbeat on a Linux System:
- Elastic provides repositories available for APT and YUM-based distributions. Note that we provide binary packages, but no source packages.
- To install the APT and YUM repositories follow the setup instructions in this https://www.elastic.co/guide/en/beats/auditbeat/current/setup-repositories.html.
- To run Auditbeat on Docker follow the setup instructions in the https://www.elastic.co/guide/en/beats/auditbeat/current/running-on-docker.html.
- To run Auditbeat on Kubernetes follow the setup instructions in the https://www.elastic.co/guide/en/beats/auditbeat/current/running-on-kubernetes.html.
- For complete “Setup and Run Auditbeat” information refer to the https://www.elastic.co/guide/en/beats/auditbeat/current/setting-up-and-running.html.

#### Custom Ingest Pipeline
For versions <8.2, you need to add a custom ingest pipeline to populate `event.ingested` with @timestamp for non-elastic-agent indexes, like auditbeats/filebeat/winlogbeat etc. For more details to add a custom ingest pipeline refer to the https://www.elastic.co/guide/en/fleet/current/data-streams-pipeline-tutorial.html.


----------------------------------

==== Rule query


[source, js]
----------------------------------
process where host.os.type == "linux" and event.type == "start" and user.id == "0" and
  process.executable : "/usr/bin/chattr" and process.args : ("-*i*", "+*i*") and
  not process.parent.executable: ("/lib/systemd/systemd", "/usr/local/uems_agent/bin/*", "/usr/lib/systemd/systemd") and
  not process.parent.name in ("systemd", "cf-agent", "ntpdate", "xargs", "px", "preinst", "auth")

----------------------------------

*Framework*: MITRE ATT&CK^TM^

* Tactic:
** Name: Defense Evasion
** ID: TA0005
** Reference URL: https://attack.mitre.org/tactics/TA0005/
* Technique:
** Name: File and Directory Permissions Modification
** ID: T1222
** Reference URL: https://attack.mitre.org/techniques/T1222/
* Sub-technique:
** Name: Linux and Mac File and Directory Permissions Modification
** ID: T1222.002
** Reference URL: https://attack.mitre.org/techniques/T1222/002/

Testing

Testing for this involves the detection-rules repository and security-docs repository. Images and links related to successful runs will be found below.

  1. Go to detection rules repository > Actions
  2. Select "Release Docs" Workflow
  3. Enter the following:
    1. run from branch bug-security-doc-generation-broken-links
    2. Target branch: 8.12
    3. message: whatever
    4. pre: 8.12.3
    5. post: 8.12.4
  4. Check https://github.com/elastic/security-docs/pulls
  5. CI should pass as links are valid
Successful Results

Successful Release Docs Workflow Job: https://github.com/elastic/detection-rules/actions/runs/7848748147/job/21420563267

CI-Passing PR to Security Docs without needing to manually fix links: elastic/security-docs#4787
Screenshot 2024-02-09 at 2 16 41 PM
Screenshot 2024-02-09 at 2 59 07 PM

@terrancedejesus terrancedejesus added documentation Improvements or additions to documentation python Internal python for the repository fleet-release Issue tracking rule updates released to (OOB) Fleet integration package labels Feb 6, 2024
@terrancedejesus terrancedejesus self-assigned this Feb 6, 2024
@@ -71,6 +71,7 @@ jobs:
token: ${{ secrets.READ_WRITE_RELEASE_FLEET }}
repository: ${{github.event.inputs.target_repo}}
path: integrations
fetch-depth: 0
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Had an issue with checking out backport branches, so pulling all history.
Tested: elastic/integrations#9114

Copy link
Contributor

@Mikaayenson Mikaayenson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general looks good.

detection_rules/devtools.py Outdated Show resolved Hide resolved
detection_rules/docs.py Show resolved Hide resolved
Copy link
Contributor

@Mikaayenson Mikaayenson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you give this a shot?

detection_rules/docs.py Outdated Show resolved Hide resolved
@terrancedejesus terrancedejesus merged commit 06b97ec into main Feb 12, 2024
13 checks passed
@terrancedejesus terrancedejesus deleted the bug-security-doc-generation-broken-links branch February 12, 2024 15:08
protectionsmachine pushed a commit that referenced this pull request Feb 12, 2024
…ty docs (#3434)

* removed historical argument; added setup string; fixed links

* fixing flake errors

* added types for command arguments

* adjusted get_release_diff to append strings for release tags

* set fetch-depth to 0 for integrations checkout in workflow

* changed the name of the workflow

* removed TODOs

* adjusted release docs workflow to remove prefix for release tags

* adjusted URL replacement only if pointed to docs site

* added elastic website to regex pattern

* add docstrings; adjusted regex; add note for stopgap

* added a note about the regex pattern for elastic URLs

(cherry picked from commit 06b97ec)
protectionsmachine pushed a commit that referenced this pull request Feb 12, 2024
…ty docs (#3434)

* removed historical argument; added setup string; fixed links

* fixing flake errors

* added types for command arguments

* adjusted get_release_diff to append strings for release tags

* set fetch-depth to 0 for integrations checkout in workflow

* changed the name of the workflow

* removed TODOs

* adjusted release docs workflow to remove prefix for release tags

* adjusted URL replacement only if pointed to docs site

* added elastic website to regex pattern

* add docstrings; adjusted regex; add note for stopgap

* added a note about the regex pattern for elastic URLs

(cherry picked from commit 06b97ec)
protectionsmachine pushed a commit that referenced this pull request Feb 12, 2024
…ty docs (#3434)

* removed historical argument; added setup string; fixed links

* fixing flake errors

* added types for command arguments

* adjusted get_release_diff to append strings for release tags

* set fetch-depth to 0 for integrations checkout in workflow

* changed the name of the workflow

* removed TODOs

* adjusted release docs workflow to remove prefix for release tags

* adjusted URL replacement only if pointed to docs site

* added elastic website to regex pattern

* add docstrings; adjusted regex; add note for stopgap

* added a note about the regex pattern for elastic URLs

(cherry picked from commit 06b97ec)
protectionsmachine pushed a commit that referenced this pull request Feb 12, 2024
…ty docs (#3434)

* removed historical argument; added setup string; fixed links

* fixing flake errors

* added types for command arguments

* adjusted get_release_diff to append strings for release tags

* set fetch-depth to 0 for integrations checkout in workflow

* changed the name of the workflow

* removed TODOs

* adjusted release docs workflow to remove prefix for release tags

* adjusted URL replacement only if pointed to docs site

* added elastic website to regex pattern

* add docstrings; adjusted regex; add note for stopgap

* added a note about the regex pattern for elastic URLs

(cherry picked from commit 06b97ec)
protectionsmachine pushed a commit that referenced this pull request Feb 12, 2024
…ty docs (#3434)

* removed historical argument; added setup string; fixed links

* fixing flake errors

* added types for command arguments

* adjusted get_release_diff to append strings for release tags

* set fetch-depth to 0 for integrations checkout in workflow

* changed the name of the workflow

* removed TODOs

* adjusted release docs workflow to remove prefix for release tags

* adjusted URL replacement only if pointed to docs site

* added elastic website to regex pattern

* add docstrings; adjusted regex; add note for stopgap

* added a note about the regex pattern for elastic URLs

(cherry picked from commit 06b97ec)
protectionsmachine pushed a commit that referenced this pull request Feb 12, 2024
…ty docs (#3434)

* removed historical argument; added setup string; fixed links

* fixing flake errors

* added types for command arguments

* adjusted get_release_diff to append strings for release tags

* set fetch-depth to 0 for integrations checkout in workflow

* changed the name of the workflow

* removed TODOs

* adjusted release docs workflow to remove prefix for release tags

* adjusted URL replacement only if pointed to docs site

* added elastic website to regex pattern

* add docstrings; adjusted regex; add note for stopgap

* added a note about the regex pattern for elastic URLs

(cherry picked from commit 06b97ec)
protectionsmachine pushed a commit that referenced this pull request Feb 12, 2024
…ty docs (#3434)

* removed historical argument; added setup string; fixed links

* fixing flake errors

* added types for command arguments

* adjusted get_release_diff to append strings for release tags

* set fetch-depth to 0 for integrations checkout in workflow

* changed the name of the workflow

* removed TODOs

* adjusted release docs workflow to remove prefix for release tags

* adjusted URL replacement only if pointed to docs site

* added elastic website to regex pattern

* add docstrings; adjusted regex; add note for stopgap

* added a note about the regex pattern for elastic URLs

(cherry picked from commit 06b97ec)
protectionsmachine pushed a commit that referenced this pull request Feb 12, 2024
…ty docs (#3434)

* removed historical argument; added setup string; fixed links

* fixing flake errors

* added types for command arguments

* adjusted get_release_diff to append strings for release tags

* set fetch-depth to 0 for integrations checkout in workflow

* changed the name of the workflow

* removed TODOs

* adjusted release docs workflow to remove prefix for release tags

* adjusted URL replacement only if pointed to docs site

* added elastic website to regex pattern

* add docstrings; adjusted regex; add note for stopgap

* added a note about the regex pattern for elastic URLs

(cherry picked from commit 06b97ec)
protectionsmachine pushed a commit that referenced this pull request Feb 12, 2024
…ty docs (#3434)

* removed historical argument; added setup string; fixed links

* fixing flake errors

* added types for command arguments

* adjusted get_release_diff to append strings for release tags

* set fetch-depth to 0 for integrations checkout in workflow

* changed the name of the workflow

* removed TODOs

* adjusted release docs workflow to remove prefix for release tags

* adjusted URL replacement only if pointed to docs site

* added elastic website to regex pattern

* add docstrings; adjusted regex; add note for stopgap

* added a note about the regex pattern for elastic URLs

(cherry picked from commit 06b97ec)
protectionsmachine pushed a commit that referenced this pull request Feb 12, 2024
…ty docs (#3434)

* removed historical argument; added setup string; fixed links

* fixing flake errors

* added types for command arguments

* adjusted get_release_diff to append strings for release tags

* set fetch-depth to 0 for integrations checkout in workflow

* changed the name of the workflow

* removed TODOs

* adjusted release docs workflow to remove prefix for release tags

* adjusted URL replacement only if pointed to docs site

* added elastic website to regex pattern

* add docstrings; adjusted regex; add note for stopgap

* added a note about the regex pattern for elastic URLs

(cherry picked from commit 06b97ec)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport: auto documentation Improvements or additions to documentation fleet-release Issue tracking rule updates released to (OOB) Fleet integration package python Internal python for the repository
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Meta] Update Prebuilt Detection Rules Release Process
3 participants