[Security] Add security detection engine package #797

rw-access · 2021-03-22T19:22:07Z

What does this PR do?

Added an integration that contains security_rule assets to be used by the Detection Engine in Security. The core idea behind this effort is to use Fleet to update rules for the Detection Engine between releases. This way, users can get the latest released rules and package-relevant rules without needing to wait for the next stack release.

See the "related issues" section or ping me for more context. I'll be happy to point to additional (internal) design docs and discussions.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them. (have not tested the package using elastc-package yet)
~~I have verified that all data streams collect metrics or logs.~~
I have added an entry to my package's changelog.yml file.

Author's Checklist

We've had a lot of conversations between teams, so I think we're in alignment. The name of the integration is TBD, and the specifics to install the security_rule asset type are also undecided.

How to test this PR locally

There is a chicken-and-egg issue here. Since it contains a brand new asset type, Kibana doesn't yet know how to install it. So the package will be published first as a pre-release and once this merges and is added to package-storage, then we can start developing the Kibana side to install the package.

There will be more iteration as we go between all of the repositories.

Related issues

*elastic/package-spec#142: Add security_rule asset type

Add NOTICE.txt file at the root level of a package package-spec#151: Add NOTICE.txt file
[snapshot] Create minimal detection_rules package package-storage#843: Original package for this to power
[Security Solution][Detections] Allows rules to be updated from EPR kibana#91949: Now closed, initially pulled in [snapshot] Create minimal detection_rules package package-storage#843, but our design has changed so that EPM will publish the rules as saved objects to an intermediate staging area

Screenshots

None yet

elasticmachine · 2021-03-22T19:30:47Z

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #797 updated
Start Time: 2021-03-30T20:30:21.395+0000
Duration: 45 min 10 sec
Commit: f9f5807

Test stats 🧪

Test	Results
Failed	0
Passed	1842
Skipped	3
Total	1845

Trends 🧪

Steps errors

Expand to view the steps failures

`Boot up the Elastic stack`

Took 3 min 28 sec . View more details on here
Description: ../../build/elastic-package stack up -d -v

Log output

Expand to view the last 100 lines of log output

[2021-03-30T21:15:21.868Z] - kubernetes-382ace30-9d98-11e9-b2ae-49acc4cbcea9 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-3dbf6230-9c20-11e9-9dc8-fd27291d427f (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-3e1e1fd0-9c27-11e9-9dc8-fd27291d427f (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-408fccf0-30d6-11e7-8df8-6d3604a72912 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-44f12b40-2bf4-11e7-859b-f78b612cde28 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-485c8550-9c3a-11e9-9dc8-fd27291d427f (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-58e644f0-30d6-11e7-8df8-6d3604a72912 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-7aac4fd0-30e0-11e7-8df8-6d3604a72912 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-7cbeb750-5794-11e8-afa2-e9067ea62228 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-7d80f790-9d96-11e9-b2ae-49acc4cbcea9 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-826d80c0-9c97-11e9-94fd-c91206cd5249 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-84d9b200-9d98-11e9-b2ae-49acc4cbcea9 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-8a95de50-9c38-11e9-9dc8-fd27291d427f (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-8c6c2690-9bd8-11e9-9dc8-fd27291d427f (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-95595810-9ca8-11e9-94fd-c91206cd5249 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-95a7f110-57a2-11e8-afa2-e9067ea62228 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-a4c9d360-30df-11e7-8df8-6d3604a72912 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-b8a24790-9bf0-11e9-9dc8-fd27291d427f (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-ba7bf750-9bf5-11e9-9dc8-fd27291d427f (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-bcb194a0-9bf8-11e9-9dc8-fd27291d427f (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-cac89fb0-9906-11e9-ba57-b7ab4e2d4b58 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-cd059410-2bfb-11e7-859b-f78b612cde28 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-d6564360-2bfc-11e7-859b-f78b612cde28 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-d86b2da0-9c20-11e9-9dc8-fd27291d427f (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-d9fc1b80-9c9c-11e9-94fd-c91206cd5249 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-da1ff7c0-30ed-11e7-b9e5-2b5b07213ab3 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-e0ddd3e0-98fe-11e9-ba57-b7ab4e2d4b58 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-e1018b90-2bfb-11e7-859b-f78b612cde28 (type: visualization)
[2021-03-30T21:15:21.868Z] - kubernetes-ec360ff0-57a0-11e8-afa2-e9067ea62228 (type: visualization)
[2021-03-30T21:15:21.868Z] - metrics-kubernetes.apiserver (type: index_template)
[2021-03-30T21:15:21.868Z] - metrics-kubernetes.container (type: index_template)
[2021-03-30T21:15:21.868Z] - metrics-kubernetes.controllermanager (type: index_template)
[2021-03-30T21:15:21.868Z] - metrics-kubernetes.event (type: index_template)
[2021-03-30T21:15:21.868Z] - metrics-kubernetes.node (type: index_template)
[2021-03-30T21:15:21.868Z] - metrics-kubernetes.pod (type: index_template)
[2021-03-30T21:15:21.868Z] - metrics-kubernetes.proxy (type: index_template)
[2021-03-30T21:15:21.868Z] - metrics-kubernetes.scheduler (type: index_template)
[2021-03-30T21:15:21.868Z] - metrics-kubernetes.state_container (type: index_template)
[2021-03-30T21:15:21.868Z] - metrics-kubernetes.state_cronjob (type: index_template)
[2021-03-30T21:15:21.868Z] - metrics-kubernetes.state_daemonset (type: index_template)
[2021-03-30T21:15:21.868Z] - metrics-kubernetes.state_deployment (type: index_template)
[2021-03-30T21:15:21.868Z] - metrics-kubernetes.state_node (type: index_template)
[2021-03-30T21:15:21.868Z] - metrics-kubernetes.state_persistentvolume (type: index_template)
[2021-03-30T21:15:21.868Z] - metrics-kubernetes.state_persistentvolumeclaim (type: index_template)
[2021-03-30T21:15:21.868Z] - metrics-kubernetes.state_pod (type: index_template)
[2021-03-30T21:15:21.869Z] - metrics-kubernetes.state_replicaset (type: index_template)
[2021-03-30T21:15:21.869Z] - metrics-kubernetes.state_resourcequota (type: index_template)
[2021-03-30T21:15:21.869Z] - metrics-kubernetes.state_service (type: index_template)
[2021-03-30T21:15:21.869Z] - metrics-kubernetes.state_statefulset (type: index_template)
[2021-03-30T21:15:21.869Z] - metrics-kubernetes.state_storageclass (type: index_template)
[2021-03-30T21:15:21.869Z] - metrics-kubernetes.system (type: index_template)
[2021-03-30T21:15:21.869Z] - metrics-kubernetes.volume (type: index_template)
[2021-03-30T21:15:21.869Z] Done
[2021-03-30T21:15:21.902Z] Running in /var/lib/jenkins/workspace/gest-manager_integrations_PR-797/src/github.com/elastic/integrations
[2021-03-30T21:15:22.199Z] + build/elastic-package stack dump -v --output build/elastic-stack-dump/7.11.2-SNAPSHOT/kubernetes
[2021-03-30T21:15:22.199Z] 2021/03/30 21:15:21 DEBUG Enable verbose logging
[2021-03-30T21:15:22.199Z] 2021/03/30 21:15:21 DEBUG Dump Elastic stack data
[2021-03-30T21:15:22.199Z] 2021/03/30 21:15:21 DEBUG Dump stack logs
[2021-03-30T21:15:22.199Z] 2021/03/30 21:15:21 DEBUG Recreate the output location (path: build/elastic-stack-dump/7.11.2-SNAPSHOT/kubernetes)
[2021-03-30T21:15:22.200Z] 2021/03/30 21:15:21 DEBUG Dump stack logs for elasticsearch
[2021-03-30T21:15:22.200Z] 2021/03/30 21:15:21 DEBUG running command: /usr/local/bin/docker-compose -f /var/lib/jenkins/workspace/gest-manager_integrations_PR-797/.elastic-package/stack/snapshot.yml -p elastic-package-stack logs elasticsearch
[2021-03-30T21:15:23.155Z] 2021/03/30 21:15:23 DEBUG Dump stack logs for elastic-agent
[2021-03-30T21:15:23.155Z] 2021/03/30 21:15:23 DEBUG running command: /usr/local/bin/docker-compose -f /var/lib/jenkins/workspace/gest-manager_integrations_PR-797/.elastic-package/stack/snapshot.yml -p elastic-package-stack logs elastic-agent
[2021-03-30T21:15:24.548Z] 2021/03/30 21:15:24 DEBUG Dump stack logs for kibana
[2021-03-30T21:15:24.548Z] 2021/03/30 21:15:24 DEBUG running command: /usr/local/bin/docker-compose -f /var/lib/jenkins/workspace/gest-manager_integrations_PR-797/.elastic-package/stack/snapshot.yml -p elastic-package-stack logs kibana
[2021-03-30T21:15:25.493Z] 2021/03/30 21:15:25 DEBUG Dump stack logs for package-registry
[2021-03-30T21:15:25.493Z] 2021/03/30 21:15:25 DEBUG running command: /usr/local/bin/docker-compose -f /var/lib/jenkins/workspace/gest-manager_integrations_PR-797/.elastic-package/stack/snapshot.yml -p elastic-package-stack logs package-registry
[2021-03-30T21:15:26.437Z] Path to stack dump: build/elastic-stack-dump/7.11.2-SNAPSHOT/kubernetes
[2021-03-30T21:15:26.437Z] Done
[2021-03-30T21:15:26.459Z] Archiving artifacts
[2021-03-30T21:15:26.769Z] + build/elastic-package stack down -v
[2021-03-30T21:15:26.769Z] 2021/03/30 21:15:26 DEBUG Enable verbose logging
[2021-03-30T21:15:26.769Z] Take down the Elastic stack
[2021-03-30T21:15:26.769Z] 2021/03/30 21:15:26 DEBUG running command: /usr/local/bin/docker-compose -f /var/lib/jenkins/workspace/gest-manager_integrations_PR-797/.elastic-package/stack/snapshot.yml -p elastic-package-stack down
[2021-03-30T21:15:27.340Z] The ELASTICSEARCH_IMAGE_REF variable is not set. Defaulting to a blank string.
[2021-03-30T21:15:27.340Z] The KIBANA_IMAGE_REF variable is not set. Defaulting to a blank string.
[2021-03-30T21:15:27.340Z] The ELASTIC_AGENT_IMAGE_REF variable is not set. Defaulting to a blank string.
[2021-03-30T21:15:27.340Z] Stopping elastic-package-stack_elastic-agent_1    ... 
[2021-03-30T21:15:27.340Z] Stopping elastic-package-stack_kibana_1           ... 
[2021-03-30T21:15:27.340Z] Stopping elastic-package-stack_package-registry_1 ... 
[2021-03-30T21:15:27.340Z] Stopping elastic-package-stack_elasticsearch_1    ... 
[2021-03-30T21:15:29.704Z] 
Stopping elastic-package-stack_elastic-agent_1    ... done

Stopping elastic-package-stack_kibana_1           ... done

Stopping elastic-package-stack_package-registry_1 ... done

Stopping elastic-package-stack_elasticsearch_1    ... done
Removing elastic-package-stack_elastic-agent_is_ready_1    ... 
[2021-03-30T21:15:29.705Z] Removing elastic-package-stack_elastic-agent_1             ... 
[2021-03-30T21:15:29.705Z] Removing elastic-package-stack_kibana_is_ready_1           ... 
[2021-03-30T21:15:29.705Z] Removing elastic-package-stack_elasticsearch_is_ready_1    ... 
[2021-03-30T21:15:29.705Z] Removing elastic-package-stack_kibana_1                    ... 
[2021-03-30T21:15:29.705Z] Removing elastic-package-stack_package-registry_is_ready_1 ... 
[2021-03-30T21:15:29.705Z] Removing elastic-package-stack_package-registry_1          ... 
[2021-03-30T21:15:29.705Z] Removing elastic-package-stack_elasticsearch_1             ... 
[2021-03-30T21:15:29.705Z] 
Removing elastic-package-stack_kibana_is_ready_1           ... done

Removing elastic-package-stack_kibana_1                    ... done

Removing elastic-package-stack_elastic-agent_is_ready_1    ... done

Removing elastic-package-stack_package-registry_is_ready_1 ... done

Removing elastic-package-stack_package-registry_1          ... done

Removing elastic-package-stack_elasticsearch_is_ready_1    ... done

Removing elastic-package-stack_elasticsearch_1             ... done

Removing elastic-package-stack_elastic-agent_1             ... done
Removing network elastic-package-stack_default
[2021-03-30T21:15:30.280Z] Done
[2021-03-30T21:15:30.424Z] Stage "Update Package Storage" skipped due to earlier failure(s)
[2021-03-30T21:15:30.679Z] Running on worker-395930 in /var/lib/jenkins/workspace/gest-manager_integrations_PR-797
[2021-03-30T21:15:30.722Z] [INFO] getVaultSecret: Getting secrets
[2021-03-30T21:15:30.829Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-03-30T21:15:32.866Z] + chmod 755 generate-build-data.sh
[2021-03-30T21:15:32.866Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Ingest-manager/integrations/PR-797/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Ingest-manager/integrations/PR-797/runs/12 FAILURE 2710068
[2021-03-30T21:15:32.867Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Ingest-manager/integrations/PR-797/runs/12/steps/?limit=10000 -o steps-info.json
[2021-03-30T21:15:36.010Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Ingest-manager/integrations/PR-797/runs/12/tests/?status=FAILED -o tests-errors.json
[2021-03-30T21:15:36.711Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Ingest-manager/integrations/PR-797/runs/12/log/ -o pipeline-log.txt

mtojek

I didn't review JSON files, as I don't have domain knowledge here and can't verify them thoroughly.

Please merge the latest master to load the latest dependency on elastic-package (it should start accepting new Kibana types).

Before merging this PR we need to make sure the PR is green.

packages/security_detection_engine/changelog.yml

packages/security_detection_engine/manifest.yml

…-pkg

…ations into add-detection-rules-pkg

go.mod

ycombinator

LGTM.

…-pkg

ruflin · 2021-03-25T07:55:12Z

go.mod

@@ -4,7 +4,7 @@ go 1.12

 require (
 github.com/blang/semver v3.5.1+incompatible
- github.com/elastic/elastic-package v0.0.0-20210323050956-96ac7e929cae
+ github.com/elastic/elastic-package v0.0.0-20210324160257-57698a40f3e3


Could you pull this go mod change into a speparate PR?

I did it here: #804 (spotted also a different issue). Once it's merged, please rebase this PR against master.

Merged in master and confirmed the go changes are gone.
Thanks @mtojek

mtojek · 2021-03-25T08:24:28Z

Hm... I spotted a weird issue:

[2021-03-24T21:22:19.014Z] Error: checking package failed: linting package failed: found 3 validation errors:
[2021-03-24T21:22:19.014Z]    1. item [test-audit.log-config.json] is not allowed in folder [/var/lib/jenkins/workspace/gest-manager_integrations_PR-797/src/github.com/elastic/integrations/packages/gcp/data_stream/audit/_dev/test/pipeline]
[2021-03-24T21:22:19.014Z]    2. item [test-firewall.log-config.json] is not allowed in folder [/var/lib/jenkins/workspace/gest-manager_integrations_PR-797/src/github.com/elastic/integrations/packages/gcp/data_stream/firewall/_dev/test/pipeline]
[2021-03-24T21:22:19.014Z]    3. item [test-vpcflow.log-config.json] is not allowed in folder [/var/lib/jenkins/workspace/gest-manager_integrations_PR-797/src/github.com/elastic/integrations/packages/gcp/data_stream/vpcflow/_dev/test/pipeline]

Will investigate

EDIT:

It seems that there are few files that are not aligned with spec, I will fix them and also bump up the dependency on elastic-package.

mtojek · 2021-03-25T09:53:33Z

@rw-access Please rebase this branch again against the master branch. I merged the update on elastic-package, so above errors should disappear.

…-pkg

Test are passing

rw-access · 2021-03-25T17:17:09Z

@mtojek CI is green now. thanks for pulling the deps into a separate PR.
do i have your 👍 to merge?

Looking into a few more things first. Might need to change the underlying JSON format.

…-pkg

* Add security detection engine package * Fix some lint errors * Format the package * Update version in the changelog * Update dependencies * Change the default owner for the detection engine package * Update go deps * Fix package-* dependencies * Change the security_rule structure to match other Kibana assets

Add security detection engine package

5b70331

rw-access added the enhancement New feature or request label Mar 22, 2021

rw-access added 2 commits March 22, 2021 13:43

Fix some lint errors

8b30ae9

Format the package

98fe8b6

mtojek self-requested a review March 23, 2021 09:23

mtojek previously requested changes Mar 23, 2021

View reviewed changes

packages/security_detection_engine/changelog.yml Outdated Show resolved Hide resolved

packages/security_detection_engine/manifest.yml Outdated Show resolved Hide resolved

mtojek requested a review from ycombinator March 23, 2021 09:33

rw-access added 2 commits March 23, 2021 09:15

Merge remote-tracking branch 'origin/master' into add-detection-rules…

b0590e3

…-pkg

Update version in the changelog

a7ca3a3

rw-access mentioned this pull request Mar 23, 2021

Update the deps to pull in package-spec#153 elastic/elastic-package#294

Merged

rw-access added 2 commits March 23, 2021 11:27

Update dependencies

c474565

Merge branch 'add-detection-rules-pkg' of github.com:rw-access/integr…

8ad404c

…ations into add-detection-rules-pkg

ycombinator reviewed Mar 23, 2021

View reviewed changes

go.mod Outdated Show resolved Hide resolved

rw-access added 3 commits March 24, 2021 14:37

Change the default owner for the detection engine package

3610bcd

Update go deps

7521c68

Fix package-* dependencies

a8aa228

ycombinator approved these changes Mar 24, 2021

View reviewed changes

Merge remote-tracking branch 'origin/master' into add-detection-rules…

527f131

…-pkg

rw-access requested a review from mtojek March 24, 2021 21:36

ruflin reviewed Mar 25, 2021

View reviewed changes

mtojek mentioned this pull request Mar 25, 2021

Fix: adjust configs for pipeline tests (replace json with yml) #804

Merged

Merge remote-tracking branch 'origin/master' into add-detection-rules…

1de305e

…-pkg

rw-access mentioned this pull request Mar 25, 2021

Change the JSON schema for the security_rule Kibana asset elastic/detection-rules#1066

Merged

rw-access added 2 commits March 30, 2021 14:29

Change the security_rule structure to match other Kibana assets

f9f5807

Merge remote-tracking branch 'origin/master' into add-detection-rules…

f78f5ea

…-pkg

rw-access merged commit 092033c into elastic:master Mar 30, 2021

rw-access deleted the add-detection-rules-pkg branch March 30, 2021 21:19

rw-access mentioned this pull request Mar 31, 2021

[Fleet] Install security_rule assets as saved objects elastic/kibana#95885

Merged

2 tasks

rw-access mentioned this pull request Apr 9, 2021

[Security] Adds pre-packaged rule updates through the "Prebuilt Security Detection Rules" Fleet integration elastic/kibana#96698

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security] Add security detection engine package #797

[Security] Add security detection engine package #797

rw-access commented Mar 22, 2021 •

edited

Loading

elasticmachine commented Mar 22, 2021 •

edited

Loading

Build stats

Test stats 🧪

Trends 🧪

`Boot up the Elastic stack`

mtojek left a comment

ycombinator left a comment

ruflin Mar 25, 2021

mtojek Mar 25, 2021

rw-access Mar 25, 2021

mtojek commented Mar 25, 2021 •

edited

Loading

mtojek commented Mar 25, 2021

rw-access commented Mar 25, 2021 •

edited

Loading

[Security] Add security detection engine package #797

[Security] Add security detection engine package #797

Conversation

rw-access commented Mar 22, 2021 • edited Loading

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Mar 22, 2021 • edited Loading

💔 Build Failed

Build stats

Test stats 🧪

Trends 🧪

Steps errors

Boot up the Elastic stack

Log output

mtojek left a comment

Choose a reason for hiding this comment

ycombinator left a comment

Choose a reason for hiding this comment

ruflin Mar 25, 2021

Choose a reason for hiding this comment

mtojek Mar 25, 2021

Choose a reason for hiding this comment

rw-access Mar 25, 2021

Choose a reason for hiding this comment

mtojek commented Mar 25, 2021 • edited Loading

mtojek commented Mar 25, 2021

rw-access commented Mar 25, 2021 • edited Loading

rw-access commented Mar 22, 2021 •

edited

Loading

elasticmachine commented Mar 22, 2021 •

edited

Loading

`Boot up the Elastic stack`

mtojek commented Mar 25, 2021 •

edited

Loading

rw-access commented Mar 25, 2021 •

edited

Loading