Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cleanup rule survey code #1923

Merged
merged 12 commits into from
Sep 6, 2022
Merged

Conversation

brokensound77
Copy link
Contributor

Issues

None

Summary

I was working on a related project using this code and founds some errors due to changes from previous rule object refactoring

@botelastic botelastic bot added the python Internal python for the repository label Apr 13, 2022
@brokensound77
Copy link
Contributor Author

There are a few linting errors to resolve

@Mikaayenson
Copy link
Contributor

Can you link an example of previous rule refactoring that highlights the break?

@Mikaayenson
Copy link
Contributor

@brokensound77 @terrancedejesus Can we setup time together to walk through the changes in detection_rules/eswrap.py?

@Mikaayenson Mikaayenson added the v8.3.0 Rules for 8.3.0 label Apr 19, 2022
@Mikaayenson Mikaayenson added bug Something isn't working v8.2.0 labels Apr 21, 2022
@brokensound77
Copy link
Contributor Author

Can you link an example of previous rule refactoring that highlights the break?

#1029

@Mikaayenson
Copy link
Contributor

Mikaayenson commented Apr 27, 2022

The def search() method in eswrap.py seems to timeout. When setting the --timeout=120 the rule_survey cli takes an long time (>10 min) but completes.

elastic_transport.ConnectionTimeout: Connection timed out
Exception ignored in: <function Kibana.__del__ at 0x110d2edd0>
Traceback (most recent call last):
  File "/Users/stryker/workspace/Community/brokensound77/detection-rules/kibana/connector.py", line 184, in __del__
  File "/Users/stryker/workspace/Community/brokensound77/detection-rules/kibana/connector.py", line 173, in logout
  File "/Users/stryker/workspace/Community/brokensound77/detection-rules/kibana/connector.py", line 103, in get
  File "/Users/stryker/workspace/Community/brokensound77/detection-rules/kibana/connector.py", line 87, in request
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/requests/sessions.py", line 515, in request
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/requests/sessions.py", line 435, in prepare_request
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/requests/cookies.py", line 544, in merge_cookies
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/requests/cookies.py", line 352, in update
  File "/opt/homebrew/Cellar/python@3.10/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/copy.py", line 92, in copy
ImportError: sys.meta_path is None, Python is likely shutting down

@Mikaayenson
Copy link
Contributor

Mikaayenson commented Apr 27, 2022

Here was the output generated:

rule_id name search_count alert_count
a87a4e42-1d82-44bd-b0bf-d9b7f91fb89e Web Application Suspicious Activity: POST Request Declined 0
75ee75d8-c180-481c-ba88-ee50129a6aef Web Application Suspicious Activity: Unauthorized Method 0
43303fd4-4839-4e48-b2b2-803ab060758d Web Application Suspicious Activity: No User Agent 0
d49cc73f-7a16-4def-89ce-9fc7127d7820 Web Application Suspicious Activity: sqlmap User Agent 0
027ff9ea-85e7-42e3-99d2-bbb7069e02eb Potential Cookies Theft via Browser Debugging 0
3115bd2c-0baa-4df0-80ea-45e474b5ef93 Agent Spoofing - Mismatched Agent ID 0
493834ca-f861-414c-8602-150d5505b777 Agent Spoofing - Multiple Hosts Using Same Agent 0
665e7a4f-c58e-4fc6-bc83-87a7572670ac WebServer Access Logs Deleted 0
b0046934-486e-462f-9487-0d4cf9e429c6 Timestomping using Touch Command 0
870aecc0-cea4-4110-af3f-e02e9b373655 Security Software Discovery via Grep 0
c85eb82c-d2c8-485c-a36f-534f914b7663 Virtual Machine Fingerprinting via Grep 0
41824afb-d68c-4d0e-bfee-474dac1fa56e EggShell Backdoor Execution 0
a1a0375f-22c2-48c0-81a4-7c2d11cc6856 Potential Reverse Shell Activity via Terminal 0
8acb7614-1d92-4359-bfcf-478b6d9de150 Suspicious JAVA Child Process 0
c3f5e1d8-910e-43b4-8d44-d748e498ca86 Potential JAVA/JNDI Exploitation Attempt 0
9c260313-c811-4ec8-ab89-8f6530e0246c Hosts File Modified 0
58ac2aa5-6718-427c-a845-5f3ac5af00ba Zoom Meeting with no Passcode 0
93f47b6f-5728-4004-ba00-625083b3dcb0 Modification of Standard Authentication Module or Configuration 0
e6c1a552-7776-44ad-ae0f-8746cc07773c Bash Shell Profile Modification 0
2215b8bd-1759-4ffa-8ab8-55c8e6b32e7f SSH Authorized Keys File Modification 0
76152ca1-71d0-4003-9e37-0983e12832da Potential Privilege Escalation via Sudoers File Modification 0
8a1b0278-0f9a-487d-96bd-d4833298e87a Setuid / Setgid Bit Set via chmod -1
f37f3054-d40b-49ac-aa9b-a786c74c58b8 Sudo Heap-Based Buffer Overflow Attempt 0
931e25a5-0f5e-4ae0-ba0d-9e94eff7e3a4 Sudoers File Modification 0
699e9fdb-b77c-4c01-995c-1c15019b9c43 Threat Intel Filebeat Module (v8.x) Indicator Match 0
0c9a14d9-d65d-486f-9b5b-91e4e6b22bd0 Threat Intel Indicator Match 0
594e0cbf-86cc-45aa-9ff7-ff27db27d3ed AWS CloudTrail Log Created 0
ea248a02-bc47-4043-8e94-2885b19b2636 AWS IAM Brute Force of Assume Role Policy 0
333de828-8190-4cf5-8d7c-7575846f6fe0 AWS IAM User Addition to Group 0
4d50a94f-2844-43fa-8395-6afbd5e1c5ef AWS Management Console Brute Force of Root User Identity 0
a00681e3-9ed6-447c-ab2c-be648821c622 AWS Access Secret in Secrets Manager 0
7024e2a0-315d-4334-bb1a-441c593e16ab AWS CloudTrail Log Deleted 0
1aa8fa52-44a7-4dae-b058-f3333b91c8d7 AWS CloudTrail Log Suspended 0
f772ec8a-e182-483c-91d2-72058f76a44c AWS CloudWatch Alarm Deletion 0
7024e2a0-315d-4334-bb1a-552d604f27bc AWS Config Service Tampering 0
fbd44836-0d69-4004-a0b4-03c20370c435 AWS Configuration Recorder Stopped 0
9395fd2c-9947-4472-86ef-4aceb2f7e872 AWS EC2 Flow Log Deletion 0
8623535c-1e17-44e1-aa97-7a0699c3037d AWS EC2 Network Access Control List Deletion 0
7b3da11a-60a2-412e-8aa7-011e1eb9ed47 AWS ElastiCache Security Group Created 0
1ba5160d-f5a2-4624-b0ff-6a1dc55d2516 AWS ElastiCache Security Group Modified or Deleted 0
523116c0-d89d-4d7c-82c2-39e6845a78ef AWS GuardDuty Detector Deletion 0
227dc608-e558-43d9-b521-150772250bae AWS S3 Bucket Configuration Deletion 0
91d04cd4-47a9-4334-ab14-084abe274d49 AWS WAF Access Control List Deletion 0
5beaebc1-cc13-4bfc-9949-776f9e0dc318 AWS WAF Rule or Rule Group Deletion 0
c1812764-0788-470f-8e74-eb4a14d47573 AWS EC2 Full Network Packet Capture Detected 0
98fd7407-0bd5-5817-cda0-3fcc33113a56 AWS EC2 Snapshot Activity 0
e919611d-6b6f-493b-8314-7ed6ac2e413b AWS EC2 VM Export Failure 0
119c8877-8613-416d-a98a-96b6664ee73a AWS RDS Snapshot Export 0
bf1073bf-ce26-4607-b405-ba1ed8e9e204 AWS RDS Snapshot Restored 0
87594192-4539-4bc4-8543-23bc3d5bd2b4 AWS EventBridge Rule Disabled or Deleted 0
3e002465-876f-4f04-b016-84ef48ce7e5d AWS CloudTrail Log Updated 0
68a7a5a5-a2fc-4a76-ba9f-26849de881b4 AWS CloudWatch Log Group Deletion 0
d624f0ae-3dd1-4856-9aad-ccfe4d4bfa17 AWS CloudWatch Log Stream Deletion 0
bb9b13b2-1700-48a8-a750-b43b0a72ab69 AWS EC2 Encryption Disabled 0
536997f7-ae73-447d-a12d-bff1e8f5f0a0 AWS EFS File System or Mount Deleted 0
d8fc1cca-93ed-43c1-bbb6-c0dd3eff2958 AWS IAM Deactivation of MFA Device 0
867616ec-41e5-4edc-ada2-ab13ab45de8a AWS IAM Group Deletion 0
863cdf31-7fd3-41cf-a185-681237ea277b AWS RDS Security Group Deletion 0
9055ece6-2689-4224-a0e0-b04881e1f8ad AWS Deletion of RDS Instance or Cluster 0
ecf2b32c-e221-4bd4-aa3b-c7d59b3bc01d AWS RDS Instance/Cluster Stoppage 0
e2a67480-3b79-403d-96e3-fdd2992c50ef AWS Management Console Root Login 0
69c420e8-6c9e-4d28-86c0-8a2be2d1e78c AWS IAM Password Recovery Requested 0
37b211e8-4e2f-440f-86d8-06cc8f158cfa AWS Execution via System Manager 0
39144f38-5284-4f8e-a2ae-e3fd628d90b0 AWS EC2 Network Access Control List Creation 0
29052c19-ff3e-42fd-8363-7be14d7c5469 AWS Security Group Configuration Change Detection 0
169f3a93-efc7-4df2-94d6-0d9438c310d1 AWS IAM Group Creation 0
e14c5fd7-fdd7-49c2-9e5b-ec49d817bc8d AWS RDS Cluster Creation 0
378f9024-8a0c-46a5-aa08-ce147ac73a4e AWS RDS Security Group Creation 0
f30f3443-4fbb-4c27-ab89-c3ad49d62315 AWS RDS Instance Creation 0
12051077-0124-4394-9522-8f4f4db1d674 AWS Route 53 Domain Transfer Lock Disabled 0
2045567e-b0af-444a-8c0b-0b6e2dae9e13 AWS Route 53 Domain Transferred to Another Account 0
e3c27562-709a-42bd-82f2-3ed926cced19 AWS Route53 private hosted zone associated with a VPC 0
e12c0318-99b1-44f2-830c-3a38a43207ca AWS Route Table Created 0
e7cd5982-17c8-4959-874c-633acde7d426 AWS Route Table Modified or Deleted 0
979729e7-0c52-4c4c-b71e-88103304a79f AWS SAML Activity 0
bc0c6f0d-dab0-47a3-b135-0925f0a333bc AWS Root Login Without MFA 0
93075852-b0f5-4b8b-89c3-a226efae5726 AWS Security Token Service (STS) AssumeRole Usage 0
b45ab1d2-712f-4f01-a751-df3826969807 AWS STS GetSessionToken Abuse 0
a60326d7-dca7-4fb7-93eb-1ca03a1febbd AWS IAM Assume Role Policy Update 0
b6dce542-2b75-4ffb-b7d6-38787298ba9d Azure Event Hub Authorization Rule Created or Updated 0
3ad77ed4-4dcf-4c51-8bfc-e3f7ce316b2f Azure Full Network Packet Capture Detected 0
792dd7a6-7e00-4a0a-8a9a-a7c24720b5ec Azure Key Vault Modified 0
1e0b832e-957e-43ae-b319-db82d228c908 Azure Storage Account Key Regenerated 0
1a36cace-11a7-43a8-9a10-b497c5a02cd3 Azure Application Credential Modification 0
d79c4b2a-6134-4edd-86e6-564a92a933f9 Azure Blob Permissions Modification 0
5370d4cd-2bb3-4d71-abf5-1e1d0ff5a2de Azure Diagnostic Settings Deletion 0
60b6b72f-0fbc-47e7-9895-9ba7627a8b50 Azure Service Principal Addition 0
e0f36de1-0342-453d-95a9-a068b257b053 Azure Event Hub Deletion 0
e02bd3ea-72c6-4181-ac2b-0f83d17ad969 Azure Firewall Policy Deletion 0
09d028a5-dcde-409f-8ae0-557cef1b7082 Azure Frontdoor Web Application Firewall (WAF) Policy Deleted 0
8b64d36a-1307-4b2e-a77b-a0027e4d27c8 Azure Kubernetes Events Deleted 0
323cb487-279d-4218-bcbd-a568efe930c6 Azure Network Watcher Deletion 0
f0bc081a-2346-4744-a6a4-81514817e888 Azure Alert Suppression Rule Created or Modified 0
2636aa6c-88b5-4337-9c31-8d0192a8ef45 Azure Blob Container Access Level Modification 0
60884af6-f553-4a6c-af13-300047455491 Azure Command Execution on Virtual Machine 0
8ddab73b-3d15-4e5d-9413-47f05553c1d7 Azure Automation Runbook Deleted 0
f766ffaf-9568-4909-b734-75d19b35cbf4 Azure Service Principal Credentials Added 0
83a1931d-8136-46fc-b7b9-2db4f639e014 Azure Kubernetes Pods Deleted 0
bb4fe8d2-7ae2-475c-8b5d-55b449e4264f Azure Resource Group Deletion 0
573f6e7a-7acf-4bcd-ad42-c4969124d3c0 Azure Virtual Network Device Modified or Deleted 0
37994bca-0611-4500-ab67-5588afe73b77 Azure Active Directory High Risk Sign-in 0
26edba02-6979-4bce-920a-70b080a7be81 Azure Active Directory High Risk User Sign-in Heuristic 0
a605c51a-73ad-406d-bf3a-f24cc41d5c97 Azure Active Directory PowerShell Sign-in 0
1c6a8c7a-5cb6-4a82-ba27-d5a5b8a40a38 Possible Consent Grant Attack via Azure-Registered Application 0
141e9b3a-ff37-4756-989d-05d7cbf35b0e Azure External Guest User Invitation 0
df26fd74-1baa-4479-b42e-48da84642330 Azure Automation Account Created 0
16280f1e-57e6-4242-aa21-bb4d16f13b2f Azure Automation Runbook Created or Modified 0
e9ff9c1c-fe36-4d0d-b3fd-9e0bf4853a62 Azure Automation Webhook Created 0
bc48bba7-4a23-4232-b551-eca3ca1e3f20 Azure Conditional Access Policy Modified 0
04c5a96f-19c5-44fd-9571-a0b033f9086f Azure AD Global Administrator Role Assigned 0
ed9ecd27-e3e6-4fd9-8586-7754803f7fc8 Azure Global Administrator Role Addition to PIM User 0
7882cebf-6cf1-4de3-9662-213aa13e8b80 Azure Privilege Identity Management Role Modified 0
dafa3235-76dc-40e2-9f71-1773b96d24cf Multi-Factor Authentication Disabled for an Azure User 0
774f5e28-7b75-4a58-b94e-41bf060fdd86 User Added as Owner for Azure Application 0
38e5acdd-5f20-4d99-8fe4-f0a1a592077f User Added as Owner for Azure Service Principal 0
1c966416-60c1-436b-bfd0-e002fddbfd89 Azure Kubernetes Rolebindings Created 0
3f0e5410-a4bf-4e8c-bcfc-79d67a285c54 CyberArk Privileged Access Security Error 0
c5f81243-56e0-47f9-b5bb-55a5ed89ba57 CyberArk Privileged Access Security Recommended Monitor 0
9a1a2dae-0b5f-4c3d-8305-a268d404c306 Endpoint Security 0
d62b64a8-a7c9-43e5-aee3-15a725a794e7 GCP Pub/Sub Subscription Creation 0
a10d3d9d-0f65-48f1-8b25-af175e2594f5 GCP Pub/Sub Topic Creation 0
30562697-9859-4ae0-a8c5-dab45d664170 GCP Firewall Rule Creation 0
ff9b571e-61d6-4f6c-9561-eb4cca3bafe1 GCP Firewall Rule Deletion 0
2783d84f-5091-4d7d-9319-9fceda8fa71b GCP Firewall Rule Modification 0
5663b693-0dea-4f2e-8275-f1ae5ff2de8e GCP Logging Bucket Deletion 0
51859fa0-d86b-4214-bf48-ebb30ed91305 GCP Logging Sink Deletion 0
cc89312d-6f47-48e4-a87c-4977bd4633c3 GCP Pub/Sub Subscription Deletion 0
3202e172-01b1-4738-a932-d024c514ba72 GCP Pub/Sub Topic Deletion 0
97359fd8-757d-4b1d-9af1-ef29e4a8680e GCP Storage Bucket Configuration Modification 0
2326d1b2-9acf-4dee-bd21-867ea7378b4d GCP Storage Bucket Permissions Modification 0
184dfe52-2999-42d9-b9d1-d1ca54495a61 GCP Logging Sink Modification 0
e2fb5b18-e33c-4270-851e-c3d675c9afcd GCP IAM Role Deletion 0
8fb75dda-c47a-4e34-8ecd-34facf7aad13 GCP Service Account Deletion 0
bca7d28e-4a48-47b1-adb7-5074310e9a61 GCP Service Account Disabled 0
bc0f2d83-32b8-4ae2-b0e6-6a45772e9331 GCP Storage Bucket Deletion 0
c58c3081-2e1d-4497-8491-e73a45d1a6d6 GCP Virtual Private Cloud Network Deletion 0
9180ffdf-f3d0-4db3-bf66-7a14bcff71b8 GCP Virtual Private Cloud Route Creation 0
a17bcc91-297b-459b-b5ce-bc7460d8f82a GCP Virtual Private Cloud Route Deletion 0
aa8007f0-d1df-49ef-8520-407857594827 GCP IAM Custom Role Creation 0
9890ee61-d061-403d-9bf6-64934c51f638 GCP IAM Service Account Key Deletion 0
0e5acaae-6a64-4bbc-adb8-27649c03f7e1 GCP Service Account Key Creation 0
7ceb2216-47dd-4e64-9433-cddc99727623 GCP Service Account Creation 0
2f0bae2d-bf20-4465-be86-1311addebaa3 GCP Kubernetes Rolebindings Created or Patched 0
785a404b-75aa-4ffd-8be5-3334a5a544dd Application Added to Google Workspace Domain 0
cf549724-c577-4fd6-8f9b-d1b8ec519ec0 Domain Added to Google Workspace Trusted Domains 0
93e63c3e-4154-4fc6-9f86-b411e0987bbf Google Workspace Admin Role Deletion 0
cad4500a-abd7-4ef3-b5d3-95524de7cfe1 Google Workspace MFA Enforcement Disabled 0
a99f82f5-8e77-4f8b-b3ce-10c0f6afbc73 Google Workspace Password Policy Modified 0
e555105c-ba6d-481f-82bb-9b633e7b4827 MFA Disabled for Google Workspace Organization 0
68994a6c-c7ba-4e82-b476-26a26877adf6 Google Workspace Admin Role Assigned to a User 0
acbc8bb9-2486-49a8-8779-45fb5f9a93ee Google Workspace API Access Granted via Domain-Wide Delegation of Authority 0
ad3f2807-2b3e-47d7-b282-f84acbbe14be Google Workspace Custom Admin Role Created 0
6f435062-b7fc-4af9-acea-5b1ead65c5a5 Google Workspace Role Modified 0
ec8efb0c-604d-42fa-ac46-ed1cfbc38f78 Microsoft 365 Inbox Forwarding Rule Created 0
26f68dba-ce29-497b-8e13-b4fde1db5a2d Attempts to Brute Force a Microsoft 365 User Account 0
3efee4f0-182a-40a8-a835-102c68a4175d Potential Password Spraying of Microsoft 365 User Accounts 0
2de10e77-c144-4e69-afb7-344e7127abd0 O365 Excessive Single Sign-On Logon Errors 0
60f3adec-1df9-4104-9c75-b97d9f078b25 Microsoft 365 Exchange DLP Policy Removed 0
d743ff2a-203e-4a46-a3e3-40512cfe8fbb Microsoft 365 Exchange Malware Filter Policy Deletion 0
ca79768e-40e1-4e45-a097-0e5fbc876ac2 Microsoft 365 Exchange Malware Filter Rule Modification 0
03024bd9-d23f-4ec1-8674-3cf1a21e130b Microsoft 365 Exchange Safe Attachment Rule Disabled 0
675239ea-c1bc-4467-a6d3-b9e2cc7f676d O365 Mailbox Audit Logging Bypass 0
ff4dd44a-0ac6-44c4-8609-3f81bc820f02 Microsoft 365 Exchange Transport Rule Creation 0
272a6484-2663-46db-a532-ef734bf9a796 Microsoft 365 Exchange Transport Rule Modification 0
721999d0-7ab2-44bf-b328-6e63367b9b29 Microsoft 365 Potential ransomware activity 0
b2951150-658f-4a60-832f-a00d1e6c6745 Microsoft 365 Unusual Volume of File Deletion 0
d68eb1b5-5f1c-4b6d-9e63-5b6b145cd4aa Microsoft 365 Exchange Anti-Phish Policy Deletion 0
97314185-2568-4561-ae81-f3e480e5e695 Microsoft 365 Exchange Anti-Phish Rule Modification 0
a989fa1b-9a11-4dd8-a3e9-f0de9c6eb5f2 Microsoft 365 Exchange Safe Link Policy Disabled 0
0136b315-b566-482f-866c-1d8e2477ba16 Microsoft 365 User Restricted from Sending Email 0
5930658c-2107-4afc-91af-e0e55b7f7184 O365 Email Reported by User as Malware or Phish 0
bba1b212-b85c-41c6-9b28-be0e5cdfc9b1 OneDrive Malware File Upload 0
0e52157a-8e96-4a95-a6e3-5faae5081a74 SharePoint Malware File Upload 0
514121ce-c7b6-474a-8237-68ff71672379 Microsoft 365 Exchange DKIM Signing Configuration Disabled 0
bbd1a775-8267-41fa-9232-20e5582596ac Microsoft 365 Teams Custom Application Interaction Allowed 0
0ce6487d-8069-4888-9ddd-61b52490cebc O365 Exchange Suspicious Mailbox Right Delegation 0
98995807-5b09-4e37-8a54-5cae5dc932d7 Microsoft 365 Exchange Management Group Role Assignment 0
88671231-6626-4e1b-abb7-6e361a171fbb Microsoft 365 Global Administrator Role Assigned 0
27f7c15a-91f8-4c3d-8b9e-1f99cc030a51 Microsoft 365 Teams External Access Enabled 0
5e552599-ddec-4e14-bad1-28aa42404388 Microsoft 365 Teams Guest Access Enabled 0
684554fc-0777-47ce-8c9b-3d01f198d7f8 New or Modified Federation Domain 0
8a5c1e5f-ad63-481e-b53a-ef959230f7f1 Attempt to Deactivate an Okta Network Zone 0
c749e367-a069-4a73-b1f2-43a3798153ad Attempt to Delete an Okta Network Zone 0
3805c3dc-f82c-4f8d-891e-63c24d3102b0 Attempted Bypass of Okta MFA 0
e08ccd49-0380-4b2b-8d71-8000377d6e49 Attempts to Brute Force an Okta User Account 0
97a8e584-fd3b-421f-9b9d-9c9d9e57e9d7 Potential Abuse of Repeated MFA Push Notifications 0
42bf698b-4738-445b-8231-c834ddefd8a0 Okta Brute Force or Password Spraying Attack 0
cdbebdc1-dc97-43c6-a538-f26a20c0a911 Okta User Session Impersonation 0
e90ee3af-45fc-432e-a850-4a58cf14a457 High Number of Okta User Password Reset or Unlock Attempts 0
676cff2b-450b-4cf1-8ed2-c0c58a4a2dd7 Attempt to Revoke Okta API Token 0
e6e3ecff-03dd-48ec-acbd-54a04de10c68 Possible Okta DoS Attack 0
4edd3e1a-3aa0-499b-8147-4d2ea43b1613 Unauthorized Access to an Okta Application 0
f994964f-6fce-4d75-8e79-e16ccc412588 Suspicious Activity Reported by Okta User 0
edb91186-1c7e-4db8-b53e-bfa33a1a0a8a Attempt to Deactivate an Okta Application 0
b719a170-3bdb-4141-b0e3-13e3cf627bfe Attempt to Deactivate an Okta Policy 0
cc92c835-da92-45c9-9f29-b4992ad621a0 Attempt to Deactivate an Okta Policy Rule 0
d48e1c13-4aca-4d1f-a7b1-a9161c0ad86f Attempt to Delete an Okta Application 0
b4bb1440-0fcb-4ed1-87e5-b06d58efc5e9 Attempt to Delete an Okta Policy 0
d5d86bf5-cf0c-4c06-b688-53fdc072fdfd Attempt to Delete an Okta Policy Rule 0
c74fd275-ab2c-4d49-8890-e2943fa65c09 Attempt to Modify an Okta Application 0
e48236ca-b67a-4b4e-840c-fdc7782bc0c3 Attempt to Modify an Okta Network Zone 0
6731fbf2-8f28-49ed-9ab9-9a918ceb5a45 Attempt to Modify an Okta Policy 0
000047bb-b27a-47ec-8b62-ef1a5d2c9e19 Attempt to Modify an Okta Policy Rule 0
cd16fb10-0261-46e8-9932-a0336278cdbe Modification or Removal of an Okta Application Sign-On Policy 0
6885d2ae-e008-4762-b98a-e8e1cd3a81e9 Threat Detected by Okta ThreatInsight 0
b8075894-0b62-46e5-977c-31275da34419 Administrator Privileges Assigned to an Okta Group 0
f06414a6-f2a4-466d-8eba-10f85e8abf71 Administrator Role Assigned to an Okta User 0
96b9f4ea-0e8c-435b-8d53-2096e75fcac5 Attempt to Create Okta API Token 0
cd89602e-9db0-48e3-9391-ae3bf241acd8 Attempt to Deactivate MFA for an Okta User Account 0
729aa18d-06a6-41c7-b175-b65b739b1181 Attempt to Reset MFA Factors for an Okta User Account 0
9f1c4ca3-44b5-481d-ba42-32dc215a2769 Potential Protocol Tunneling via EarthWorm 0
6b84d470-9036-4cc0-a27c-6d90bbfe81ab Sensitive Files Compression 0
f28e2be4-6eca-4349-bdd9-381573730c22 Potential OpenSSH Backdoor Logging Activity 0
125417b8-d3df-479f-8418-12d7e034fee3 Attempt to Disable IPTables or Firewall 0
2f8a1226-5720-437d-9c20-e0029deb6194 Attempt to Disable Syslog Service 0
debff20a-46bc-4a4d-bae5-5cdd14222795 Base16 or Base32 Encoding/Decoding Activity 0
7bcbb3ac-e533-41ad-a612-d6c3bf666aba Tampering of Bash Command-Line History 0
eb9eb8ba-a983-41d9-9c93-a1c05112ca5e Potential Disabling of SELinux 0
a1329140-8de3-4445-9f87-908fb6d824f4 File Deletion via Shred 0
9f9a2a82-93a8-4b1a-8778-1780895626d4 File Permission Modification in Writable Directory 0
b9666521-4742-49ce-9ddc-b8e84c35acae Creation of Hidden Files and Directories 0
cd66a5af-e34b-4bb0-8931-57d0a043f2ef Kernel Module Removal 0
aa895aea-b69c-4411-b110-8d7599634b30 System Log File Deletion 0
2d8043ed-5bda-4caf-801c-c1feb7410504 Enumeration of Kernel Modules 0
5b03c9fb-9945-4d2f-9568-fd690fee3fba Virtual Machine Fingerprinting 0
8fed8450-847e-43bd-874c-3bbf0cd425f3 Linux Restricted Shell Breakout via apt/apt-get Changelog Escape 0
10754992-28c7-4472-be5b-f3770fd04f2d Linux Restricted Shell Breakout via awk Commands 0
e9b4a3c7-24fc-49fd-a00f-9c938031eef1 Linux Restricted Shell Breakout via busybox Shell Evasion 0
1859ce38-6a50-422b-a5e8-636e231ea0cd Linux Restricted Shell Breakout via c89/c99 Shell evasion 0
0968cfbd-40f0-4b1c-b7b1-a60736c7b241 Linux Restricted Shell Breakout via cpulimit Shell Evasion 0
ee619805-54d7-4c56-ba6f-7717282ddd73 Linux Restricted Shell Breakout via crash Shell evasion 0
72d33577-f155-457d-aad3-379f9b750c97 Linux Restricted Shell Breakout via env Shell Evasion 0
fd3fc25e-7c7c-4613-8209-97942ac609f6 Linux Restricted Shell Breakout via the expect command 0
6f683345-bb10-47a7-86a7-71e9c24fb358 Linux Restricted Shell Breakout via the find command 0
f52362cd-baf1-4b6d-84be-064efc826461 Linux Restricted Shell Breakout via flock Shell evasion 0
da986d2c-ffbf-4fd6-af96-a88dbf68f386 Linux Restricted Shell Breakout via the gcc command 0
83b2c6e5-e0b2-42d7-8542-8f3af86a1acb Linux Restricted Shell Breakout via the mysql command 0
05e5a668-7b51-4a67-93ab-e9af405c9ef3 Interactive Terminal Spawned via Perl 0
d76b02ef-fc95-4001-9297-01cb7412232f Interactive Terminal Spawned via Python 0
97da359b-2b61-4a40-b2e4-8fc48cf7a294 Linux Restricted Shell Breakout via the ssh command 0
89583d1b-3c2e-4606-8b74-0a9fd2248e88 Linux Restricted Shell Breakout via the vi command 0
fb9937ce-7e21-46bf-831d-1ad96eac674d Auditd Max Failed Login Attempts 0
cab4f01c-793f-4a54-a03e-e5d85b96d7af Auditd Login from Forbidden Location 0
20dc4620-3b68-4269-8124-ca5091e00ea8 Auditd Max Login Sessions 0
90e28af7-1d96-4582-bf11-9a1eff21d0e5 Auditd Login Attempt at Forbidden Time 0
e19e64ee-130e-4c07-961f-8a339f0b8362 Connection to External Network via Telnet 0
1b21abcc-4d9f-4b08-a7f5-316f5f94b973 Connection to Internal Network via Telnet 0
90169566-2260-4824-b8e4-8615c3b4ed52 Hping Process Activity 0
041d4d41-9589-43e2-ba13-5680af75ebc2 Potential DNS Tunneling via Iodine 0
adb961e0-cb74-42a0-af9e-29fc41f88f5f Netcat Network Activity 0
0d69150b-96f8-467c-a86d-a67a3378ce77 Nping Process Activity 0
df959768-b0c9-4d45-988c-5606a2be8e5a Unusual Process Execution - Temp 0
d6450d4e-81c6-46a3-bd94-079886318ed5 Strace Process Activity 0
0415f22a-2336-45fa-ba07-618a5942e22c Modification of OpenSSH Binaries 0
e3e904b3-0a8e-4e68-86a8-977a163e21d3 Persistence via KDE AutoStart Script or Desktop File Modification 0
231876e7-4d1f-4d63-a47c-47dd1acdc1cb Potential Shell via Web Server 0
717f82c2-7741-4f9b-85b8-d06aeb853f4f Modification of Dynamic Linker Preload Shared Object 0
8da41fc9-7735-4b24-9cc6-c78dfc9fc9c9 Potential Privilege Escalation via PKEXEC 0
20457e4f-d1de-4b92-ae69-142e27a4342a Access of Stored Browser Credentials 0
96e90768-c3b7-4df6-b5d9-6237f8bc36a8 Access to Keychain Credentials Directories 0
02ea4563-ec10-4974-b7de-12e65aa4f9b3 Dumping Account Hashes via Built-In Commands 0
565d6ca5-75ba-4c82-9b13-add25353471c Dumping of Keychain Content via Security Command 0
ad88231f-e2ab-491c-8fc6-64746da26cfe Kerberos Cached Credentials Dumping 0
9092cd6c-650f-4fa3-8a8a-28256c7489c9 Keychain Password Retrieval via Command Line 0
10a500bb-a28f-418e-ba29-ca4c8d1a9f2f WebProxy Settings Modification 0
ace1e989-a541-44df-93a8-a8b0591b63c0 Potential SSH Brute Force Detected 0
38948d29-3d5d-42e3-8aec-be832aaaf8eb Prompt for Credentials with OSASCRIPT 0
d75991f2-b989-419d-b797-ac1e54ec2d61 SystemKey Access via Command Line 0
f683dcdf-a018-4801-b066-193d4ae6c8e5 SoftwareUpdate Preferences Modification 0
f0b48bbc-549e-4bcf-8ee0-a7a72586c6a7 Attempt to Remove File Quarantine Attribute 0
4da13d6e-904f-4636-81d8-6ab14b4e6ae9 Attempt to Disable Gatekeeper 0
bc1eeacf-2972-434f-b782-3a532b100d67 Attempt to Install Root Certificate 0
7453e19e-3dbf-4e4e-9ae0-33d6c6ed15e1 Modification of Environment Variable via Launchctl 0
eea82229-b002-470e-a9e1-00be38b14d32 Potential Privacy Control Bypass via TCCDB Modification 0
c02c8b9f-5e1d-463c-a1b0-04edcdfe1a3d Potential Privacy Control Bypass via Localhost Secure Copy 0
6482255d-f468-45ea-a5b3-d3a7de1331ae Modification of Safari Settings via Defaults Command 0
d22a85c6-d2ad-4cc4-bf7b-54787473669a Potential Microsoft Office Sandbox Evasion 0
b00bcd89-000c-4425-b94c-716ef67762f6 TCC Bypass via Mounted APFS Snapshot Access 0
70fa1af4-27fd-4f26-bd03-50b6af6b9e24 Attempt to Unload Elastic Endpoint Security Kernel Extension 0
6e9b351e-a531-4bdc-b73e-7034d6eed7ff Enumeration of Users or Groups via Built-in Commands 0
35330ba2-c859-4c98-8b7f-c19159ea0e58 Execution via Electron Child Process Node.js Module 0
080bc66a-5d56-4d1f-8071-817671716db9 Suspicious Browser Child Process 0
99239e7d-b0d4-46e3-8609-acafcf99f68c macOS Installer Spawns Network Event 0
5d9f8cfc-0d03-443e-a167-2b0597ce0965 Suspicious Automator Workflows Execution 0
47f76567-d58a-4fed-b32b-21f571e28910 Apple Script Execution followed by Network Connection 0
d461fac0-43e8-49e2-85ea-3a58fe120b4f Shell Execution via Apple Scripting 0
66da12b1-ac83-40eb-814c-07ed1d82b7b9 Suspicious macOS MS Office Child Process 0
16904215-2c95-4ac8-bf5c-12354e047192 Potential Kerberos Attack via Bifrost 0
661545b4-1a90-4f45-85ce-2ebd7c6a15d0 Attempt to Mount SMB Share via Command Line 0
5ae4e6f8-d1bf-40fa-96ba-e29645e1e4dc Remote SSH Login Enabled via systemsetup Command 0
15dacaa0-5b90-466b-acab-63435a59701a Virtual Private Network Connection Attempt 0
41b638a1-8ab6-4f8e-86d9-466317ef2db5 Potential Hidden Local User Account Creation 0
082e3f8c-6f80-485c-91eb-5b112cb79b28 Launch Agent Creation or Modification and Immediate Loading 0
f24bcae1-8980-4b30-b5dd-f851b055c9e7 Creation of Hidden Login Item via Apple Script 0
9d19ece6-c20e-481a-90c5-ccca596537de LaunchDaemon Creation or Modification and Immediate Loading 0
e6c98d38-633d-4b3e-9387-42112cd5ac10 Authorization Plugin Modification 0
083fa162-e790-4d85-9aeb-4fea04188adb Suspicious Hidden Child Process of Launchd 0
89fa6cb7-6b53-4de2-b604-648488841ab8 Persistence via DirectoryService Plugin Modification 0
c81cefcb-82b9-4408-a533-3c3df549e62d Persistence via Docker Shortcut Modification 0
a6bf4dd4-743e-4da8-8c03-3ebd753a6c90 Emond Rules Creation or Modification 0
3e3d15c6-1509-479a-b125-21718372157e Suspicious Emond Child Process 0
cc2fd2d0-ba3a-4939-b87f-2901764ed036 Attempt to Enable the Root Account 0
092b068f-84ac-485d-8a55-7dd9e006715f Creation of Hidden Launch Agent or Daemon 0
37f638ea-909d-4f94-9248-edd21e4a9906 Finder Sync Plugin Registered and Enabled 0
c292fa52-4115-408a-b897-e14f684b3cb7 Persistence via Folder Action Script 0
5d0265bf-dea9-41a9-92ad-48a8dcd05080 Persistence via Login or Logout Hook 0
ac412404-57a5-476f-858f-4e8fbb4f48d8 Potential Persistence via Login Hook 0
88817a33-60d3-411f-ba79-7c905d865b2a Sublime Plugin or Application Script Modification 0
48ec9452-e1fd-4513-a376-10a1a26d2c83 Potential Persistence via Periodic Tasks 0
48d7f54d-c29e-4430-93a9-9db6b5892270 Unexpected Child Process of macOS Screensaver Engine 0
e6e8912f-283f-4d0d-8442-e0dcaf49944b Screensaver Plist File Modified by Unexpected Process 0
cb71aa62-55c8-42f0-b0dd-afb0bb0b1f51 Suspicious Calendar File Modification 0
b4449455-f986-4b5a-82ed-e36b129331f7 Potential Persistence via Atom Init Script Modification 0
827f8d8f-4117-4ae4-b551-f56d54b9da6b Apple Scripting Execution with Administrator Privileges 0
f0eb70e9-71e9-40cd-813f-bf8e8c812cb1 Execution with Explicit Credentials via Scripting 0
f85ce03f-d8a8-4c83-acdc-5c8cd0592be7 Suspicious Child Process of Adobe Acrobat Reader Update Service 0
565c2b44-7a21-4818-955f-8d4737967d2e Potential Admin Group Account Addition 0
0ff84c42-873d-41a2-a4ed-08d74d352d01 Privilege Escalation via Root Crontab File Modification 0
cf53f532-9cc9-445a-9ae7-fced307ec53c Cobalt Strike Command and Control Beacon -1
e7075e8d-a966-458e-a183-85cd331af255 Default Cobalt Strike Team Server Certificate 0
6ea71ff0-9e95-475b-9506-2580d1ce6154 DNS Activity to the Internet 0
ff013cb4-274d-434a-96bb-fe15ddd3ae92 Roshal Archive (RAR) or PowerShell File Downloaded from the Internet 0
4a4e23cf-78a2-449c-bac3-701924c269d3 Possible FIN7 DGA Command and Control Behavior -1
2e580225-2a58-48ef-938b-572933be06fe Halfbaked Command and Control Beacon -1
a9cb3641-ff4b-4cdc-a063-b4b8d02a67c7 IPSEC NAT Traversal Port Activity 0
d7e62693-aab9-4f66-a21a-3d79ecdd603d SMTP on Port 26/TCP 0
8c1bdde8-4204-45c0-9e0c-c85ca3902488 RDP (Remote Desktop Protocol) from the Internet 0
34fde489-94b0-4500-a76f-b8a157cf9269 Telnet Port Activity 0
5700cb81-df44-46aa-a5d7-337798f53eb8 VNC (Virtual Network Computing) from the Internet 0
3ad49c61-7adc-42c1-b788-732eda2f5abf VNC (Virtual Network Computing) to the Internet 0
143cb236-0956-4f42-a706-814bcaa0cf5a RPC (Remote Procedure Call) from the Internet 0
32923416-763a-4531-bb35-f33b9232ecdb RPC (Remote Procedure Call) to the Internet 0
c82b2bd8-d701-420c-ba43-f11a155b681a SMB (Windows File Sharing) Activity to the Internet 0
31295df3-277b-4c56-a1fb-84e31b4222a9 Inbound Connection to an Unsecure Elasticsearch Node -1
77a3c3df-8ec4-4da4-b758-878f551dee69 Adversary Behavior - Detected - Elastic Endgame 0
571afc56-5ed9-465d-a2a9-045f099f6e7e Credential Dumping - Detected - Elastic Endgame 0
db8c33a8-03cd-4988-9e2c-d0a4863adb13 Credential Dumping - Prevented - Elastic Endgame 0
c0be5f31-e180-48ed-aa08-96b36899d48f Credential Manipulation - Detected - Elastic Endgame 0
c9e38e64-3f4c-4bf3-ad48-0e61a60ea1fa Credential Manipulation - Prevented - Elastic Endgame 0
2003cdc8-8d83-4aa5-b132-1f9a8eb48514 Exploit - Detected - Elastic Endgame 0
2863ffeb-bf77-44dd-b7a5-93ef94b72036 Exploit - Prevented - Elastic Endgame 0
0a97b20f-4144-49ea-be32-b540ecc445de Malware - Detected - Elastic Endgame 0
3b382770-efbb-44f4-beed-f5e0a051b895 Malware - Prevented - Elastic Endgame 0
c3167e1b-f73c-41be-b60b-87f4df707fe3 Permission Theft - Detected - Elastic Endgame 0
453f659e-0429-40b1-bfdb-b6957286e04b Permission Theft - Prevented - Elastic Endgame 0
80c52164-c82a-402c-9964-852533d58be1 Process Injection - Detected - Elastic Endgame 0
990838aa-a953-4f3e-b3cb-6ddf7584de9e Process Injection - Prevented - Elastic Endgame 0
8cb4f625-7743-4dfb-ae1b-ad92be9df7bd Ransomware - Detected - Elastic Endgame 0
e3c5d5cb-41d5-4206-805c-f30561eae3ac Ransomware - Prevented - Elastic Endgame 0
eb079c62-4481-4d6e-9643-3ca499df7aaa External Alerts 0
6aace640-e631-4870-ba8e-5fdda09325db Exporting Exchange Mailbox via PowerShell 0
2f2f4939-0b34-40c2-a0a3-844eb7889f43 PowerShell Suspicious Script with Audio Capture Capabilities 0
bd2c86a0-8b61-4457-ab38-96943984e889 PowerShell Keylogging Script 0
959a7353-1129-4aa7-9084-30746b256a70 PowerShell Suspicious Script with Screenshot Capabilities 0
45d273fb-1dca-457d-9855-bcb302180c21 Encrypting Files with WinRar or 7z 0
3838e0e3-1850-4850-a411-2e8c5ba40ba8 Network Connection via Certutil 0
66883649-f908-4a5b-a1e0-54090a1d3a32 Connection to Commonly Abused Web Services 0
3a59fc81-99d3-47ea-8cd6-d48d561fca20 Potential DNS Tunneling via NsLookup 0
e3cf38fa-d5b8-46cc-87f9-4a7513e4281d Connection to Commonly Abused Free SSL Certificate Providers 0
acd611f3-2b93-47b3-a0a3-7723bcc46f6d Potential Command and Control via Internet Explorer 0
3535c8bb-3bd5-40f4-ae32-b7cd589d5372 Port Forwarding Rule Addition 0
76fd43b7-3480-4dd9-8ad7-8bd36bfad92f Potential Remote Desktop Tunneling Detected 0
15c0b7a7-9c34-4869-b25b-fa6518414899 Remote File Download via Desktopimgdownldr Utility 0
c6453e73-90eb-4fe7-a98c-cde7bbfc504a Remote File Download via MpCmdRun 0
33f306e8-417c-411b-965c-c2812d6d3f4d Remote File Download via PowerShell 0
1d276579-3380-4095-ad38-e596a01bc64f Remote File Download via Script Interpreter 0
22599847-5d13-48cb-8872-5796fee8692b SUNBURST Command and Control Activity 0
b25a7df2-120a-4db2-bd3f-3e4b86b24bee Remote File Copy via TeamViewer 0
00140285-b827-4aee-aa09-8113f58a08f3 Potential Credential Access via Windows Utilities 0
3bc6deaa-fbd4-433a-ae21-3e892f95624f NTDS or SAM Database File Copied 0
9d110cb3-5f4b-4c9a-b9f5-53f0a1707ae5 Microsoft Build Engine Loading Windows Credential Libraries 0
9f962927-1a4f-45f3-a57b-287f2c7029c1 Potential Credential Access via DCSync 0
e514d8cd-ed15-4011-84e2-d15147e059f1 Kerberos Pre-authentication Disabled for User 0
b83a7e96-2eb3-4edf-8346-427b6858d3bd Creation or Modification of Domain Backup DPAPI private key 0
a7e7bfa3-088e-4f13-b29e-3986e0e756b8 Credential Acquisition via Registry Hive Dumping 0
0564fb9d-90b9-4234-a411-82a546dc1343 Microsoft IIS Service Account Password Dumped 0
c25e9c87-95e1-4368-bfab-9fd34cf867ec Microsoft IIS Connection Strings Decryption 0
897dc6b5-b39f-432a-8d75-d3730d50c782 Kerberos Traffic from Unusual Process 0
f2f46686-6f3c-4724-bd7d-24e31c70f98f LSASS Memory Dump Creation 0
208dbe77-01ed-4954-8d44-1e5751cb20de LSASS Memory Dump Handle Access 0
ebb200e8-adf0-43f8-a0bb-4ee5b5d852c6 Mimikatz Memssp Log File Detected 0
d703a5af-d5b0-43bd-8ddb-7a5d500b7da5 Modification of WDigest Security Provider 0
a4c7473a-5cb4-4bc1-9d06-e4a75adbc494 Windows Registry File Creation in SMB Share 0
54c3d186-0461-4dc3-9b33-2dc5c7473936 Network Logon Provider Registry Modification 0
577ec21e-56fe-4065-91d8-45eb8224fe77 PowerShell MiniDump Script 0
eb610e70-f9e6-4949-82b9-f1c5bcd37c39 PowerShell Kerberos Ticket Request 0
02a4576a-7480-4284-9327-548a806b5e48 Potential Credential Access via DuplicateHandle in LSASS -1
850d901a-2a3c-46c6-8b22-55398a01aad8 Potential Remote Credential Access via Registry 0
be8afaed-4bcd-4e0a-b5f9-5562003dde81 Searching for Saved Credentials via VaultCmd 0
f494c678-3c33-43aa-b169-bb3d5198c41d Sensitive Privilege SeEnableDelegationPrivilege assigned to a User 0
79f97b31-480e-4e63-a7f4-ede42bf2c6de Potential Shadow Credentials added to AD Object 0
0b2f3da5-b5ec-47d1-908b-6ebb74814289 User account exposed to Kerberoasting 0
c5c9f591-d111-4cf8-baec-c26a39bc31ef Potential Credential Access via Renamed COM+ Services DLL 0
9960432d-9b26-409f-972b-839a959e79e2 Potential Credential Access via LSASS Memory Dump -1
0f93cb9a-1931-48c2-8cd0-f173fd3e5283 Potential LSASS Memory Dump via PssCaptureSnapShot 0
47e22836-4a16-4b35-beee-98f6c4ee9bf2 Suspicious Remote Registry Access via SeBackupPrivilege -1
d117cbb4-7d56-41b4-b999-bdf8c25648a0 Symbolic Link to Shadow Copy Created 0
a16612dd-b30e-4d41-86a0-ebe70974ec00 Potential LSASS Clone Creation via PssCaptureSnapShot 0
4630d948-40d4-4cef-ac69-4002e29bc3db Adding Hidden File Attribute via Attrib 0
f874315d-5188-4b4a-8521-d1c73093a7e4 Modification of AmsiEnable Registry Key 0
b5877334-677f-4fb9-86d5-a9721274223b Clearing Windows Console History 0
d331bbe2-6db4-4941-80a5-8270db72eb61 Clearing Windows Event Logs 0
45ac4800-840f-414c-b221-53dd36a5aaf7 Windows Event Logs Cleared 0
28896382-7d4f-4d50-9b72-67091901fd26 Suspicious Process from Conhost 0
203ab79b-239b-4aa5-8e54-fc50623ee8e4 Creation or Modification of Root Certificate 0
56557cde-d923-4b88-adee-c61b3f3b5dc3 Windows CryptoAPI Spoofing Vulnerability (CVE-2020-0601 - CurveBall) 0
2ffa1f1e-b6db-47fa-994b-1512743847eb Windows Defender Disabled via Registry Modification 0
2c17e5d7-08b9-43b2-b58a-0270d65ac85b Windows Defender Exclusions Added via PowerShell 0
f675872f-6d85-40a3-b502-c0d2ef101e92 Delete Volume USN Journal with Fsutil 0
818e23e6-2094-4f0e-8c01-22d30f3506c6 PowerShell Script Block Logging Disabled 0
4b438734-3793-4fda-bd42-ceeada0be8f9 Disable Windows Firewall Rules via Netsh 0
c8cccb06-faf2-4cd5-886e-2c9636cfcb87 Disabling Windows Defender Security Settings via PowerShell 0
4de76544-f0e5-486a-8f84-eae0b6063cdc Disable Windows Event and Security Logs Using Built-in Tools 0
a22a09c2-2162-4df0-a356-9aacbeb56a04 DNS-over-HTTPS Enabled via Registry 0
201200f1-a99b-43fb-88ed-f65a45c4972c Suspicious .NET Code Compilation 0
074464f9-f30d-4029-8c03-0ed237fffec7 Remote Desktop Enabled in Windows Firewall by Netsh 0
8b4f0816-6a65-4630-86a6-c21c179c0d09 Enable Host Network Discovery via Netsh 0
416697ae-e468-4093-a93d-59661fa619ec Control Panel Process with Unusual Arguments 0
edf8ee23-5ea7-4123-ba19-56b41e424ae3 ImageLoad via Windows Update Auto Update Client 0
c5dc3223-13a2-44a2-946c-e9dc0aa0449c Microsoft Build Engine Started by an Office Application 0
9d110cb3-5f4b-4c9a-b9f5-53f0a1707ae2 Microsoft Build Engine Started by a Script Process 0
9d110cb3-5f4b-4c9a-b9f5-53f0a1707ae3 Microsoft Build Engine Started by a System Process 0
9d110cb3-5f4b-4c9a-b9f5-53f0a1707ae4 Microsoft Build Engine Using an Alternate Name 0
9d110cb3-5f4b-4c9a-b9f5-53f0a1707ae6 Microsoft Build Engine Started an Unusual Process 0
1160dcdb-0a0a-4a79-91d8-9b84616edebd Potential DLL SideLoading via Trusted Microsoft Programs 0
053a0387-f3b5-4ba5-8245-8002cca2bd08 Potential DLL Side-Loading via Microsoft Antimalware Service Executable 0
8b2b3a62-a598-4293-bc14-3d5fa22bb98f Executable File Creation with Multiple Extensions 0
93c1ce76-494c-4f01-8167-35edfb52f7b1 Encoded Executable Stored in the Registry 0
ebf1adea-ccf2-4943-8b96-7ab11ca173a5 IIS HTTP Logging Disabled 0
9d110cb3-5f4b-4c9a-b9f5-53f0a1707ae9 Process Injection by the Microsoft Build Engine 0
a13167f1-eec2-4015-9631-1fee60406dcf InstallUtil Process Making Network Connections 0
b41a13c6-ba45-4bab-a534-df53d0cfed6a Suspicious Endpoint Security Parent Process 0
2e1e835d-01e5-48ca-b9fc-7a61f7f11902 Renamed AutoIt Scripts Interpreter 0
ac5012b8-8da8-440b-aaaf-aedafdea2dff Suspicious WerFault Child Process 0
32c5cf9c-2ef8-4e87-819e-5ccb7cd18b14 Program Files Directory Masquerading 0
6ea41894-66c3-4df7-ad6b-2c5074eb3df8 Potential Windows Error Manager Masquerading 0
fe794edd-487f-4a90-b285-3ee54f2af2d3 Microsoft Windows Defender Tampering 0
63e65ec3-43b1-45b0-8f2d-45b34291dc44 Network Connection via Signed Binary 0
feeed87c-5e95-4339-aef1-47fd79bcfbe3 MS Office Macro Security Registry Modifications 0
0e79980b-4250-4a50-a509-69294c14e84b MsBuild Making Network Connections 0
c2d90150-0133-451c-a783-533e736c12d7 Mshta Making Network Connections 0
b86afe07-0d98-4738-b15d-8d7465f95ff5 Network Connection via MsXsl 0
1fe3b299-fbb5-4657-a937-1d746f2c711a Unusual Network Activity from a Windows System Binary 0
c88d4bd0-5649-4c52-87ea-9be59dbfbcf2 Parent Process PID Spoofing 0
e26f042e-c590-4e82-8e05-41e81bd822ad Suspicious .NET Reflection via PowerShell 0
81fe9dc6-a2d7-4192-a2d8-eed98afc766a PowerShell Suspicious Payload Encoded and Compressed 0
2e29e96a-b67c-455a-afe4-de6183431d0d Potential Process Injection via PowerShell 0
ccc55af4-9882-4c67-87b4-449a7ae8079c Potential Process Herpaderping Attempt 0
f63c8e3c-d396-404f-b2ea-0379d3942d73 Windows Firewall Disabled via PowerShell 0
09443c92-46b3-45a4-8f25-383b028b258d Process Termination followed by Deletion 0
f036953a-4615-4707-a1ca-dc53bf69dcd5 Unusual Child Processes of RunDLL32 0
9aa0e1f6-52ce-42e1-abb3-09657cee2698 Scheduled Tasks AT Command Enabled 0
5aee924b-6ceb-4633-980e-1bde8cdb40c5 Potential Secure File Deletion via SDelete Utility 0
f2c7b914-eda3-40c2-96ac-d23ef91776ca SIP Provider Modification 0
b9960fef-82c6-4816-befa-44745030e917 SolarWinds Process Disabling Services via Registry 0
fd70c98a-c410-42dc-a2e3-761c71848acf Suspicious CertUtil Commands 0
8a1d4831-3ce6-4859-9891-28931fa6101d Suspicious Execution from a Mounted Device 0
acf738b5-b5b2-4acc-bad9-1e18ee234f40 Suspicious Managed Code Hosting Process 0
2dd480be-1263-4d9c-8672-172928f6789a Suspicious Process Access via Direct System Call -1
3ed032b2-45d8-4406-bc79-7ad1eabb2c72 Suspicious Process Creation CallTrace -1
4ed678a9-3a4f-41fb-9fea-f85a6e0a0dff Suspicious Script Object Execution 0
7f370d54-c0eb-4270-ac5a-9a6020585dc6 Suspicious WMIC XSL Script Execution 0
97aba1ef-6034-4bd3-8c1a-1e0996b27afa Suspicious Zoom Child Process 0
e94262f2-c1e9-4d3f-a907-aeab16712e1a Unusual Executable File Creation by a System Critical Process 0
71bccb61-e19b-452f-b104-79a60e546a95 Unusual File Creation - Alternate Data Stream 0
4bd1c1af-79d4-4d37-9efa-6e0240640242 Unusual Process Execution Path - Alternate Data Stream 0
c7894234-7814-44c2-92a9-f7d851ea246a Unusual Network Connection via DllHost 0
52aaab7b-b51c-441a-89ce-4387b3aea886 Unusual Network Connection via RunDLL32 0
610949a1-312f-4e04-bb55-3a79b8c95267 Unusual Process Network Connection 0
de9bd7e0-49e9-4e92-a64d-53ade2e66af1 Unusual Child Process from a System Virtual Process 0
06dceabf-adca-48af-ac79-ffdf4c3b1e9a Potential Evasion via Filter Manager 0
e0dacebe-4311-4d50-9387-b17e89c2e7fd Whitespace Padding in Process Command Line 0
ad0d2742-9a49-11ec-8d6b-acde48001122 Signed Proxy Execution via MS WorkFolders 0
eda499b8-a073-4e35-9733-22ec71f57f3a AdFind Command Activity 0
871ea072-1b71-4def-b016-6278b505138d Enumeration of Administrator Accounts 0
7b08314d-47a0-4b71-ae4e-16544176924f File and Directory Discovery 0
2856446a-34e6-435b-9fb5-f8f040bfa7ed Account Discovery Command via SYSTEM Account 0
7b8bfc26-81d2-435e-965c-d722ee397ef1 Windows Network Enumeration 0
0c7ca5c2-728d-4ad9-b1c5-bbba83ecb1f4 Peripheral Device Discovery 0
61ac3638-40a3-44b2-855a-985636ca985e PowerShell Suspicious Discovery Related Windows API Functions 0
1d72d014-e2ab-4707-b056-9b96abe7b511 External IP Lookup from Non-Browser Process 0
291a0de9-937a-4189-94c0-3e847c8b13e4 Enumeration of Privileged Local Groups Membership 0
0635c542-1b96-4335-9b47-126582d2c19a Remote System Discovery Commands 0
6ea55c81-e2ba-42f2-a134-bccf857ba922 Security Software Discovery using WMIC 0
ef862985-3f13-4262-a686-5f357bbb9bc2 Whoami Process Activity 0
d72e33fc-6e91-42ff-ac8b-e573268c5a87 Command Execution via SolarWinds Process 0
93b22c0a-06a0-4131-b830-b10d5e166ff4 Suspicious SolarWinds Child Process 0
1a6075b0-7479-450e-8fe7-b8b8438ac570 Execution of COM object via Xwizard 0
89f9a4b0-9f8f-4ee0-8823-c4751a6d6696 Command Prompt Network Connection 0
fd7a6052-58fa-4397-93c3-4795249ccfa2 Svchost spawning Cmd 0
3b47900d-e793-49e8-968f-c90dc3526aa1 Unusual Parent Process for cmd.exe 0
9ccf3ce0-0057-440a-91f5-870c6ad39093 Command Shell Activity Started via RunDLL32 0
770e0c4d-b998-41e5-a62e-c7901fd7f470 Enumeration Command Spawned via WMIPrvSE 0
ebfe1448-7fac-4d59-acea-181bd89b1f7f Process Execution from an Unusual Directory 0
cff92c41-2225-4763-b4ce-6f71e5bda5e6 Execution from Unusual Directory - Command Line 0
b29ee2be-bf99-446c-ab1a-2dc0183394b8 Network Connection via Compiled HTML File 0
0d8ad79f-9025-45d8-80c1-4f0cd3c5e8e5 Execution of File Written or Modified by Microsoft Office 0
1defdd62-cd8d-426e-a246-81a37751bb2b Execution of File Written or Modified by PDF Reader 0
ad84d445-b1ce-4377-82d9-7c633f28bf9a Suspicious Portable Executable Encoded in Powershell Script 0
56f2e9b5-4803-4e44-a0a4-a52dc79d57fe PowerShell PSReflect Script 0
55d551c6-333b-4665-ab7e-5d14a59715ce PsExec Network Connection 0
fb02b8d3-71ee-4af1-bacd-215d23f17efa Network Connection via Registration Utility 0
5cd55388-a19c-47c7-8ec4-f41656c2fded Outbound Scheduled Task Activity via PowerShell 0
a3ea12f3-0d4e-4667-8b44-4230c63f3c75 Execution via local SxS Shared Module 0
12f07955-1674-44f7-86b5-c35da0a6f41a Suspicious Cmd Execution via WMI 0
891cb88e-441a-4c3e-be2d-120d99fe7b0d Suspicious WMI Image Load from MS Office 0
53a26770-9cbd-40c5-8b57-61d01a325e14 Suspicious PDF Reader Child Process 0
852c1f19-68e8-43a6-9dce-340771fe1be3 Suspicious PowerShell Engine ImageLoad 0
e2f9fdf5-8076-45ad-9427-41e0e03dc9c2 Suspicious Process Execution via Renamed PsExec Executable 0
17c7f6a5-5bc9-4e1f-92bf-13632d24384d Suspicious Execution - Short Program Name 0
e3343ab9-4245-4715-b344-e11c56b0a47f Process Activity via Compiled HTML File 0
05b358de-aa6d-4f6c-89e6-78f74018b43b Conhost Spawned By Suspicious Parent Process 0
4ed493fc-d637-4a36-80ff-ac84937e5461 Execution via MSSQL xp_cmdshell Stored Procedure 0
11ea6bec-ebde-4d71-a8e9-784948f8e3e9 Third-party Backup Files Deleted via Unexpected Process 0
581add16-df76-42bb-af8e-c979bfb39a59 Deleting Backup Catalogs with Wbadmin 0
69c251fb-a5d6-4035-b5ec-40438bd829ff Modification of Boot Configuration 0
035889c4-2686-4583-a7df-67f89c292f2c High Number of Process and/or Service Terminations 0
b5ea4bfe-a1b2-421f-9d47-22a75a6f2921 Volume Shadow Copy Deleted or Resized via VssAdmin 0
d99a037b-c8e2-47a5-97b9-170d076827c4 Volume Shadow Copy Deletion via PowerShell 0
dc9c1f74-dac3-48e3-b47f-eb79db358f57 Volume Shadow Copy Deletion via WMIC 0
f545ff26-3c94-4fd0-bd33-3c7f95a3a0fc Windows Script Executing PowerShell 0
b64b183e-1a76-422d-9179-7b389513e74d Windows Script Interpreter Executing Process via WMI 0
6cd1779c-560f-4b68-a8f1-11009b27fe63 Microsoft Exchange Server UM Writing Suspicious Files 0
483c4daf-b0c6-49e0-adf3-0bfa93231d6b Microsoft Exchange Server UM Spawning Suspicious Processes 0
f81ee52c-297e-46d9-9205-07e66931df26 Microsoft Exchange Worker Spawning Suspicious Processes 0
a624863f-a70d-417f-a7d2-7a404638d47f Suspicious MS Office Child Process 0
32f4675e-6c49-4ace-80f9-97c9259dca2e Suspicious MS Outlook Child Process 0
8c37dc0e-e3ac-4c97-8aa0-cf6a9122de45 Unusual Child Process of dns.exe 0
c7ce36c0-32ff-4f9a-bfc2-dcb242bf99f9 Unusual File Modification by dns.exe 0
9a5b4e31-6cde-4295-9ff7-6be1b8567e1b Suspicious Explorer Child Process 0
d61cbcf8-1bc1-4cff-85ba-e7b21c5beedc Service Command Lateral Movement 0
622ecb68-fa81-4601-90b5-f8cd661e4520 Incoming DCOM Lateral Movement via MSHTA 0
51ce96fb-9e52-4dad-b0ba-99b54440fc9a Incoming DCOM Lateral Movement with MMC 0
8f919d4b-a5af-47ca-a594-6be59cd924a4 Incoming DCOM Lateral Movement with ShellBrowserWindow or ShellWindows 0
ddab1f5f-7089-44f5-9fda-de5b11322e77 NullSessionPipe Registry Modification 0
c82c7d8f-fb9e-4874-a4bd-fd9e3f9becf1 Direct Outbound SMB Connection 0
11013227-0301-4a8c-b150-4db924484475 Abnormally Large DNS Response 0
c57f8579-e2a5-4804-847f-f2732edc5156 Potential Remote Desktop Shadowing Activity 0
58bc134c-e8d2-4291-a552-b4b3e537c60b Lateral Tool Transfer 0
4fe9d835-40e1-452d-8230-17c147cafad8 Execution via TSClient Mountpoint 0
ab75c24b-2502-43a0-bf7c-e60e662c811e Remote Execution via File Shares 0
1cd01db9-be24-4bef-8e7c-e923f0ff78ab Incoming Execution via WinRM Remote Shell 0
f3475224-b179-4f78-8877-c2bd64c26b88 WMI Incoming Lateral Movement 0
c4210e1c-64f2-4f48-b67e-b5a8ffe3aa14 Mounting Hidden or WebDav Remote Shares 0
2772264c-6fb9-4d9d-9014-b416eed21254 Incoming Execution via PowerShell Remoting 0
58aa72ca-d968-4f34-b9f7-bea51d75eb50 RDP Enabled via Registry 0
8c81e506-6e82-4884-9b9a-75d3d252f967 Potential SharpRDP Behavior 0
fa01341d-6662-426b-9d0c-6d81e33c8a9d Remote File Copy to a Hidden Share 0
aa9a274d-6b53-424d-ac5e-cb8ca4251650 Remotely Started Services via RPC 0
954ee7c8-5437-49ae-b2d6-2960883898e9 Remote Scheduled Task Creation 0
e8571d5f-bea1-46c2-9f56-998de2d3ed95 Service Control Spawned via Script Interpreter 0
71c5cb27-eca5-4151-bb47-64bc3f883270 Suspicious RDP ActiveX Client Loaded 0
25224a80-5a4a-4b8a-991e-6ab390465c4f Lateral Movement via Startup Folder 0
6e9130a5-9be6-48e5-943a-9628bfc74b18 AdminSDHolder Backdoor 0
2bf78aa2-9c56-48de-b139-f169bf99cf86 Adobe Hijack Persistence 0
c5ce48a6-7f57-4ee8-9313-3d0024caee10 Installation of Custom Shim Databases 0
513f0ffd-b317-4b9c-9494-92ce861f22c7 Registry Persistence via AppCert DLL 0
d0e159cf-73e9-40d1-a9ed-077e3158a855 Registry Persistence via AppInit DLL 0
62a70f6f-3c37-43df-a556-f64fa475fba2 Account configured with never Expiring Password 0
2edc8076-291e-41e9-81e4-e3fcbc97ae5e Creation of a Hidden Local User Account 0
6839c821-011d-43bd-bd5b-acff00257226 Image File Execution Options Injection 0
c8b150f0-0164-475b-a75e-74b47800a9ff Suspicious Startup Shell Folder Modification 0
c0429aa8-9974-42da-bfb6-53a0a515a145 Creation or Modification of a new GPO Scheduled Task or Service 0
1327384f-00f3-44d5-9a8c-2373ba071e92 Persistence via Scheduled Job Creation 0
afcce5ad-65de-4ed2-8516-5e093d3ac99a Local Scheduled Task Creation 0
689b9d57-e4d5-4357-ad17-9c334609d79a Scheduled Task Created by a Windows Script 0
f44fa4b6-524c-4e87-8d9e-a32599e4fb7c Persistence via Microsoft Office AddIns 0
397945f3-d39a-4e6f-8bcb-9656c2031438 Persistence via Microsoft Outlook VBA 0
e052c845-48d0-4f46-8a13-7d0aba05df82 KRBTGT Delegation Backdoor 0
ce64d965-6cb0-466d-b74f-8d2c76f47f05 New ActiveSyncAllowedDeviceID Added via PowerShell 0
7405ddf1-6c8e-41ce-818f-48bea6bcaed8 Potential Modification of Accessibility Binaries 0
54902e45-3467-49a4-8abc-529f2c8cfb80 Uncommon Registry Persistence Change 0
2820c9c2-bcd7-4d6e-9eba-faf3891ba450 Account Password Reset Remotely 0
97fc44d3-8dae-4019-ae83-298c3015600f Startup or Run Key Registry Modification 0
e7125cea-9fe1-42a5-9a05-b0792cf86f5a Execution of Persistent Suspicious Program 0
61d29caf-6c15-4d1e-9ccb-7ad12ccc0bc7 AdminSDHolder SDProp Exclusion Added -1
403ef0d3-8259-40c9-a5b6-d48354712e49 Unusual Persistence via Services Registry 0
440e2db4-bc7f-4c96-a068-65b78da59bde Startup Persistence by a Suspicious Process 0
2fba96c0-ade5-4bce-b92f-a5df2509da3f Startup Folder Persistence via Unsigned Process 0
f7c4dc5a-a58d-491d-9f14-9b66507121c0 Persistent Scripts in the Startup Directory 0
16a52c14-7883-47af-8745-9357803f0d4c Component Object Model Hijacking 0
baa5d22c-5e1c-4f33-bfc9-efa73bb53022 Suspicious Image Load (taskschd.dll) from MS Office 0
5d1d6907-0747-4d5d-9b24-e4a18853dc0a Suspicious Execution via Scheduled Task 0
36a8e048-d888-4f61-a8b9-0f9e2e40f317 Suspicious ImagePath Service Creation 0
0022d47d-39c7-4f69-a232-4fe9dc7a3acd System Shells via Services 0
14ed1aa9-ebfd-4cf9-a463-0ac59ec55204 Potential Persistence via Time Provider Modification 0
5cd8e1f7-0050-4afc-b2df-904e40b2f5ae User Added to Privileged Group in Active Directory 0
1aa9181a-492b-4c01-8b16-fa0735786b2b User Account Creation 0
fd4a992d-6130-4802-9ff8-829b89ae801f Potential Application Shimming via Sdbinst 0
c3b915e0-22f3-4bf7-991d-b643513c722f Persistence via BITS Job Notify Cmdline 0
a9b05c3b-b304-4bf9-970d-acdfaef2944c Persistence via Hidden Run Key Detected 0
e86da94d-e54b-4fb5-b96c-cecff87e8787 Installation of Security Support Provider 0
68921d85-d0dc-48b3-865f-43291ca2c4f2 Persistence via TelemetryController Scheduled Task Hijack 0
265db8f5-fc73-4d0d-b434-6483b56372e2 Persistence via Update Orchestrator Service Hijack 0
9b6813a1-daf1-457e-b0e6-0bb4e55b8a4c Persistence via WMI Event Subscription 0
70d12c9c-0dbd-4a1a-bc44-1467502c9cf6 Persistence via WMI Standard Registry Provider 0
2917d495-59bd-4250-b395-c29409b76086 Webshell Detection: Script Process Child of Common Web Processes 0
d31f183a-e5b1-451b-8534-ba62bca0b404 Disabling User Account Control via Registry Modification 0
16fac1a1-21ee-4ca6-b720-458e3855d046 Startup/Logon Script added to Group Policy Object 0
b9554892-5e0e-424b-83a0-5aef95aa43bf Group Policy Abuse for Privilege Addition 0
15a8ba77-1c13-4274-88fe-6bd14133861e Scheduled Task Execution at Scale via GPO 0
58c6d58b-a0d3-412d-b3b8-0981a9400607 Potential Privilege Escalation via InstallerFileTakeOver 0
e9abe69b-1deb-4e19-ac4a-5d5ac00f72eb Potential LSA Authentication Package Abuse 0
3ecbdc9e-e4f2-43fa-8cca-63802125e582 Privilege Escalation via Named Pipe Impersonation 0
bfeaf89b-a2a7-48a3-817f-e41829dc61ee Suspicious DLL Loaded for Persistence or Privilege Escalation 0
8f3e91c7-d791-4704-80a1-42c160d7aa27 Potential Port Monitor or Print Processor Registration Abuse 0
bd7eefee-f671-494e-98df-f01daf9e5f17 Suspicious Print Spooler Point and Print DLL 0
5bb4a95d-5a08-48eb-80db-4c3a63ec78a8 Suspicious PrintSpooler Service Executable File Creation 0
c4818812-d44f-47be-aaef-4cfb2f9cc799 Suspicious Print Spooler File Deletion 0
a7ccae7b-9d2c-44b2-a061-98e5946971fa Suspicious PrintSpooler SPL File Created 0
d563aaba-2e72-462b-8658-3e5ea22db3a6 Privilege Escalation via Windir Environment Variable 0
bdcf646b-08d4-492c-870a-6c04e3700034 Potential Privileged Escalation via SamAccountName Spoofing 0
b90cdde7-7e0d-4359-8bf0-2c112ce2008a UAC Bypass Attempt with IEditionUpgradeManager Elevated COM Interface 0
fc7c0fa4-8f03-4b3e-8336-c5feab0be022 UAC Bypass Attempt via Elevated COM Internet Explorer Add-On Installer 0
68d56fdc-7ffa-4419-8e95-81641bd6f845 UAC Bypass via ICMLuaUtil Elevated COM Interface 0
1dcc51f6-ba26-49e7-9ef4-2655abb2361e UAC Bypass via DiskCleanup Scheduled Task Hijack 0
5a14d01d-7ac8-4545-914c-b687c2cf66b3 UAC Bypass Attempt via Privileged IFileOperation COM Interface 0
31b4c719-f2b4-41f6-a9bd-fce93c2eaf62 Bypass UAC via Event Viewer 0
290aca65-e94d-403b-ba0f-62f320e63f51 UAC Bypass Attempt via Windows Directory Masquerading 0
1178ae09-5aff-460a-9f2f-455cd0ac4d8e UAC Bypass via Windows Firewall Snap-In Hijack 0
35df0dd8-092d-4a83-88c1-5151a804f31b Unusual Parent-Child Relationship 0
ee5300a7-7e31-4a72-a258-250abb8b3aa1 Unusual Print Spooler Child Process 0
6a8ab9cc-4023-4d17-b5df-1a3e16882ce7 Unusual Service Host Child Process - Childless Service 0
76ddb638-abf7-42d5-be22-4a70b0bf7241 Privilege Escalation via Rogue Named Pipe Impersonation 0
55c2bf58-2a39-4c58-a384-c8b1978153c2 Windows Service Installed via an Unusual Client 0

@Mikaayenson
Copy link
Contributor

The json file indicates some errors, which should probably be confirmed.

e.g.

  "a16612dd-b30e-4d41-86a0-ebe70974ec00": {
    "error": "Found 1 problem\nline 3:3: Unknown column [process.parent.executable], did you mean [process.executable]?",
    "error_retrieving_results": true,
    "name": "Potential LSASS Clone Creation via PssCaptureSnapShot",
    "rule_id": "a16612dd-b30e-4d41-86a0-ebe70974ec00",
    "search_count": 0
  },

20220427T142841L.json.zip

Copy link
Contributor

@Mikaayenson Mikaayenson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are a couple suggestions and comments that need to be address first.

detection_rules/devtools.py Show resolved Hide resolved
detection_rules/devtools.py Show resolved Hide resolved
detection_rules/devtools.py Show resolved Hide resolved
detection_rules/eswrap.py Show resolved Hide resolved
Copy link
Contributor

@terrancedejesus terrancedejesus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed together with @Mikaayenson during debug session. Several changes were made to run the commands/code OOTB that may need addressed moving forward for this to be operable.

@brokensound77
Copy link
Contributor Author

The def search() method in eswrap.py seems to timeout. When setting the --timeout=120 the rule_survey cli takes an long time (>10 min) but completes.

elastic_transport.ConnectionTimeout: Connection timed out
Exception ignored in: <function Kibana.__del__ at 0x110d2edd0>
Traceback (most recent call last):
  File "/Users/stryker/workspace/Community/brokensound77/detection-rules/kibana/connector.py", line 184, in __del__
  File "/Users/stryker/workspace/Community/brokensound77/detection-rules/kibana/connector.py", line 173, in logout
  File "/Users/stryker/workspace/Community/brokensound77/detection-rules/kibana/connector.py", line 103, in get
  File "/Users/stryker/workspace/Community/brokensound77/detection-rules/kibana/connector.py", line 87, in request
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/requests/sessions.py", line 515, in request
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/requests/sessions.py", line 435, in prepare_request
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/requests/cookies.py", line 544, in merge_cookies
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/requests/cookies.py", line 352, in update
  File "/opt/homebrew/Cellar/python@3.10/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/copy.py", line 92, in copy
ImportError: sys.meta_path is None, Python is likely shutting down

Is this only occuring when you authenticate with a cookie? What about with user/pass? I have not encountered this - does the stack have a lot of data? If so, this can also be controlled by tightening the search window with this

@brokensound77
Copy link
Contributor Author

The json file indicates some errors, which should probably be confirmed.

e.g.

  "a16612dd-b30e-4d41-86a0-ebe70974ec00": {
    "error": "Found 1 problem\nline 3:3: Unknown column [process.parent.executable], did you mean [process.executable]?",
    "error_retrieving_results": true,
    "name": "Potential LSASS Clone Creation via PssCaptureSnapShot",
    "rule_id": "a16612dd-b30e-4d41-86a0-ebe70974ec00",
    "search_count": 0
  },

20220427T142841L.json.zip

This is a good thing - rule error checking for free

@brokensound77
Copy link
Contributor Author

If this tests good for you, this looks to address all stated issues IINM

@Mikaayenson
Copy link
Contributor

The def search() method in eswrap.py seems to timeout. When setting the --timeout=120 the rule_survey cli takes an long time (>10 min) but completes.

elastic_transport.ConnectionTimeout: Connection timed out
Exception ignored in: <function Kibana.__del__ at 0x110d2edd0>
Traceback (most recent call last):
  File "/Users/stryker/workspace/Community/brokensound77/detection-rules/kibana/connector.py", line 184, in __del__
  File "/Users/stryker/workspace/Community/brokensound77/detection-rules/kibana/connector.py", line 173, in logout
  File "/Users/stryker/workspace/Community/brokensound77/detection-rules/kibana/connector.py", line 103, in get
  File "/Users/stryker/workspace/Community/brokensound77/detection-rules/kibana/connector.py", line 87, in request
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/requests/sessions.py", line 515, in request
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/requests/sessions.py", line 435, in prepare_request
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/requests/cookies.py", line 544, in merge_cookies
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/requests/cookies.py", line 352, in update
  File "/opt/homebrew/Cellar/python@3.10/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/copy.py", line 92, in copy
ImportError: sys.meta_path is None, Python is likely shutting down

Is this only occuring when you authenticate with a cookie? What about with user/pass? I have not encountered this - does the stack have a lot of data? If so, this can also be controlled by tightening the search window with this

This was with user/pass not with a cookie. I haven't touched the stack in a while so I don't believe there's a lot of data.

@Mikaayenson
Copy link
Contributor

The json file indicates some errors, which should probably be confirmed.
e.g.

  "a16612dd-b30e-4d41-86a0-ebe70974ec00": {
    "error": "Found 1 problem\nline 3:3: Unknown column [process.parent.executable], did you mean [process.executable]?",
    "error_retrieving_results": true,
    "name": "Potential LSASS Clone Creation via PssCaptureSnapShot",
    "rule_id": "a16612dd-b30e-4d41-86a0-ebe70974ec00",
    "search_count": 0
  },

20220427T142841L.json.zip

This is a good thing - rule error checking for free

Do you know why that example is actually throwing an error? It seems like the fields are correct.

@brokensound77 brokensound77 removed v8.2.0 v8.3.0 Rules for 8.3.0 labels Jun 10, 2022
@Mikaayenson
Copy link
Contributor

Mikaayenson commented Aug 19, 2022

@brokensound77 @terrancedejesus I'm not sure when I'm doing wrong, but I just retested this PR with the default command and no optional params and it doesn't work. I have a .detection-rules-cfg.json with the following params set.

{"cloud_id": "my cloud id",
"es_user": "elastic",
"es_password": "my password",
"kibana_user": "elastic",
"kibana_password": "my password",
}
 python -m detection_rules kibana search-alerts
(dtest) ➜  detection-rules git:(cleanup-survey-code) python -m detection_rules kibana search-alerts
Loaded config file: /Users/stryker/workspace/ElasticGitHub/community/bksound/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Traceback (most recent call last):
  File "/Users/stryker/.virtualenvs/dtest/lib/python3.8/site-packages/requests/models.py", line 434, in prepare_url
    scheme, auth, host, port, path, query, fragment = parse_url(url)
  File "/Users/stryker/.virtualenvs/dtest/lib/python3.8/site-packages/urllib3/util/url.py", line 397, in parse_url
    return six.raise_from(LocationParseError(source_url), None)
  File "<string>", line 3, in raise_from
urllib3.exceptions.LocationParseError: Failed to parse: https://<truncated url>:443:9243/internal/security/login

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/stryker/workspace/ElasticGitHub/community/bksound/detection-rules/detection_rules/__main__.py", line 34, in <module>
    main()
  File "/Users/stryker/workspace/ElasticGitHub/community/bksound/detection-rules/detection_rules/__main__.py", line 31, in main
    root(prog_name="detection_rules")
  File "/Users/stryker/.virtualenvs/dtest/lib/python3.8/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/Users/stryker/.virtualenvs/dtest/lib/python3.8/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/stryker/.virtualenvs/dtest/lib/python3.8/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/stryker/.virtualenvs/dtest/lib/python3.8/site-packages/click/core.py", line 1134, in invoke
    Command.invoke(self, ctx)
  File "/Users/stryker/.virtualenvs/dtest/lib/python3.8/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/stryker/.virtualenvs/dtest/lib/python3.8/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/stryker/.virtualenvs/dtest/lib/python3.8/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/stryker/workspace/ElasticGitHub/community/bksound/detection-rules/detection_rules/kbwrap.py", line 33, in kibana_group
    ctx.obj['kibana'] = get_kibana_client(**kibana_kwargs)
  File "/Users/stryker/workspace/ElasticGitHub/community/bksound/detection-rules/detection_rules/misc.py", line 322, in get_kibana_client
    kibana.login(kibana_user, kibana_password, provider_type=provider_type, provider_name=provider_name)
  File "/Users/stryker/workspace/ElasticGitHub/community/bksound/detection-rules/kibana/connector.py", line 127, in login
    self.post(path, data=payload, error=True, verbose=False)
  File "/Users/stryker/workspace/ElasticGitHub/community/bksound/detection-rules/kibana/connector.py", line 111, in post
    return self.request('POST', uri, params=params, data=data, error=error, **kwargs)
  File "/Users/stryker/workspace/ElasticGitHub/community/bksound/detection-rules/kibana/connector.py", line 87, in request
    response = self.session.request(method, url, params=params, data=body, **kwargs)
  File "/Users/stryker/.virtualenvs/dtest/lib/python3.8/site-packages/requests/sessions.py", line 573, in request
    prep = self.prepare_request(req)
  File "/Users/stryker/.virtualenvs/dtest/lib/python3.8/site-packages/requests/sessions.py", line 484, in prepare_request
    p.prepare(
  File "/Users/stryker/.virtualenvs/dtest/lib/python3.8/site-packages/requests/models.py", line 368, in prepare
    self.prepare_url(url, params)
  File "/Users/stryker/.virtualenvs/dtest/lib/python3.8/site-packages/requests/models.py", line 436, in prepare_url
    raise InvalidURL(*e.args)
requests.exceptions.InvalidURL: Failed to parse: https://<truncated url>:443:9243/internal/security/login

When I tried without the config file, it still did not work.

(dtest) ➜  detection-rules git:(cleanup-survey-code) python -m detection_rules kibana search-alerts --kibana-url <my kibana url> -ku elastic -kp <password> --cloud-id <my cloud id>

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

CLI Error (): Missing required --cloud-id or --kibana-url

@brokensound77
Copy link
Contributor Author

Traceback (most recent call last):
File "/Users/stryker/.virtualenvs/dtest/lib/python3.8/site-packages/requests/models.py", line 434, in prepare_url
scheme, auth, host, port, path, query, fragment = parse_url(url)
File "/Users/stryker/.virtualenvs/dtest/lib/python3.8/site-packages/urllib3/util/url.py", line 397, in parse_url
return six.raise_from(LocationParseError(source_url), None)
File "", line 3, in raise_from
urllib3.exceptions.LocationParseError: Failed to parse: https://:443:9243/internal/security/login

It looks like as of 8.0 the domain from cloud included the port which created an invalid kibana url with double ports

resolved in 5fe2720

@brokensound77
Copy link
Contributor Author

also, I am updating the table for alerts since the structure changed in 8.0

@botelastic botelastic bot added the schema label Aug 19, 2022
@Mikaayenson
Copy link
Contributor

also, I am updating the table for alerts since the structure changed in 8.0

@brokensound77 Awesome. I think this is almost done. I was able to get the command to finally work with the config file.


(dtest) ➜  detection-rules git:(cleanup-survey-code) python -m detection_rules kibana search-alerts
Loaded config file: /Users/stryker/workspace/ElasticGitHub/community/bksound/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

==========================================================================================================================
                                                                    kibana
                                                                    alert
 host                rule
 hostname            name                                                               status   original_time
==========================================================================================================================
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-13T22:23:47.618Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-13T22:23:47.643Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-13T22:23:49.379Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-13T22:23:49.426Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-13T22:23:49.838Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-13T22:23:50.936Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-13T22:23:50.969Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-13T22:23:50.997Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-13T22:23:51.042Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-13T22:23:51.247Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-13T22:23:51.247Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-13T22:23:51.247Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-13T22:23:51.247Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-13T22:23:51.353Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-13T22:23:51.397Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-13T21:13:13.889Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-13T21:13:14.760Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-13T21:13:15.889Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-13T21:13:15.917Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-13T21:13:15.945Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-13T21:13:17.288Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-13T21:13:17.329Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-13T21:13:17.391Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-13T21:13:17.423Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-13T21:13:17.781Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-13T21:13:18.220Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-13T21:13:18.220Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-13T21:13:18.220Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-13T21:13:18.220Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-13T21:13:18.220Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-20T14:34:41.452Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-20T14:34:43.577Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-20T14:34:44.456Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-20T14:34:52.431Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-20T14:34:52.695Z
 Stryker-ml2.local   Enumeration of Users or Groups via Built-in Commands [Duplicate]   active   2022-08-20T14:34:54.626Z

However it still does not work with the command line parameters. Can you try testing the command without using the config? Or perhaps I'm passing the CLI args wrong, in which case the CLI.md should be updated.

(dtest) ➜  detection-rules git:(cleanup-survey-code) python -m detection_rules kibana search-alerts --kibana-url <kibana url> -ku elastic -kp <kibana password> --cloud-id <cloud id>

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

CLI Error (): Missing required --cloud-id or --kibana-url
(dtest) ➜  detection-rules git:(cleanup-survey-code)

@brokensound77
Copy link
Contributor Author

However it still does not work with the command line parameters. Can you try testing the command without using the config? Or perhaps I'm passing the CLI args wrong, in which case the CLI.md should be updated.

(dtest) ➜  detection-rules git:(cleanup-survey-code) python -m detection_rules kibana search-alerts --kibana-url <kibana url> -ku elastic -kp <kibana password> --cloud-id <cloud id>

You're not calling the command correctly. Kibana args should come after kibana not the command

@Mikaayenson
Copy link
Contributor

Mikaayenson commented Aug 21, 2022

Yup . You're right, when I passed the kibana creds, it was in the wrong order. The search-alerts command is working after your last two commits.

The final thing is the rule-survey command.

python -m detection_rules dev test rule-survey --cloud-id=<cloudid> --es-user=elastic --es-password=<password>--kibana-user=elastic --kibana-password=<password>

It seems to timeout. I think as a default we should limit the search window so that it at least returns something and doesn't timeout with an error. I tried the command with and without the timeout flag. It probably shouldn't crash when just using the default params. I timed it and it crashed after running for 21 min the first time and 17 min the second time.

▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Running survey against 679 rules
Saving detailed dump to: /Users/stryker/workspace/Community/brokensound77/detection-rules/surveys/20220821T075755L.json
/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py:277: ElasticsearchWarning: this request accesses system indices: [.apm-agent-configuration, .apm-custom-link, .async-search, .fleet-agents-7, .fleet-artifacts-7, .fleet-enrollment-api-keys-7, .fleet-policies-7, .fleet-policies-leader-7, .fleet-servers-7, .kibana_7.16.2_001, .kibana_task_manager_7.16.2_001, .ml-config, .security-7, .security-tokens-7, .transform-internal-007], but in a future major version, direct access to system indices will be prevented by default
  return self.client.count(body=formatted_dsl, index=index_str, q=lucene_query, allow_no_indices=True,
/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py:183: ElasticsearchWarning: this request accesses system indices: [.apm-agent-configuration, .apm-custom-link, .async-search, .fleet-agents-7, .fleet-artifacts-7, .fleet-enrollment-api-keys-7, .fleet-policies-7, .fleet-policies-leader-7, .fleet-servers-7, .kibana_7.16.2_001, .kibana_task_manager_7.16.2_001, .ml-config, .security-7, .security-tokens-7, .transform-internal-007], but in a future major version, direct access to system indices will be prevented by default
  results = self.client.eql.search(body=formatted_dsl, index=index_str, **kwargs)['hits']
/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py:183: ElasticsearchWarning: this request accesses system indices: [.apm-custom-link], but in a future major version, direct access to system indices will be prevented by default
  results = self.client.eql.search(body=formatted_dsl, index=index_str, **kwargs)['hits']
/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py:183: ElasticsearchWarning: this request accesses system indices: [.fleet-policies-leader-7], but in a future major version, direct access to system indices will be prevented by default
  results = self.client.eql.search(body=formatted_dsl, index=index_str, **kwargs)['hits']
/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py:183: ElasticsearchWarning: this request accesses system indices: [.kibana_task_manager_7.16.2_001], but in a future major version, direct access to system indices will be prevented by default
  results = self.client.eql.search(body=formatted_dsl, index=index_str, **kwargs)['hits']
/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py:183: ElasticsearchWarning: this request accesses system indices: [.fleet-agents-7], but in a future major version, direct access to system indices will be prevented by default
  results = self.client.eql.search(body=formatted_dsl, index=index_str, **kwargs)['hits']
/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py:183: ElasticsearchWarning: this request accesses system indices: [.fleet-artifacts-7], but in a future major version, direct access to system indices will be prevented by default
  results = self.client.eql.search(body=formatted_dsl, index=index_str, **kwargs)['hits']
/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py:183: ElasticsearchWarning: this request accesses system indices: [.fleet-policies-7], but in a future major version, direct access to system indices will be prevented by default
  results = self.client.eql.search(body=formatted_dsl, index=index_str, **kwargs)['hits']
/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py:183: ElasticsearchWarning: this request accesses system indices: [.fleet-servers-7], but in a future major version, direct access to system indices will be prevented by default
  results = self.client.eql.search(body=formatted_dsl, index=index_str, **kwargs)['hits']
/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py:183: ElasticsearchWarning: this request accesses system indices: [.security-tokens-7], but in a future major version, direct access to system indices will be prevented by default
  results = self.client.eql.search(body=formatted_dsl, index=index_str, **kwargs)['hits']
/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py:183: ElasticsearchWarning: this request accesses system indices: [.transform-internal-007], but in a future major version, direct access to system indices will be prevented by default
  results = self.client.eql.search(body=formatted_dsl, index=index_str, **kwargs)['hits']
/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py:183: ElasticsearchWarning: this request accesses system indices: [.fleet-enrollment-api-keys-7], but in a future major version, direct access to system indices will be prevented by default
  results = self.client.eql.search(body=formatted_dsl, index=index_str, **kwargs)['hits']
/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py:183: ElasticsearchWarning: this request accesses system indices: [.kibana_7.16.2_001], but in a future major version, direct access to system indices will be prevented by default
  results = self.client.eql.search(body=formatted_dsl, index=index_str, **kwargs)['hits']
/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py:183: ElasticsearchWarning: this request accesses system indices: [.security-7], but in a future major version, direct access to system indices will be prevented by default
  results = self.client.eql.search(body=formatted_dsl, index=index_str, **kwargs)['hits']
/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py:183: ElasticsearchWarning: this request accesses system indices: [.async-search], but in a future major version, direct access to system indices will be prevented by default
  results = self.client.eql.search(body=formatted_dsl, index=index_str, **kwargs)['hits']
/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py:183: ElasticsearchWarning: this request accesses system indices: [.apm-agent-configuration], but in a future major version, direct access to system indices will be prevented by default
  results = self.client.eql.search(body=formatted_dsl, index=index_str, **kwargs)['hits']
/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py:183: ElasticsearchWarning: this request accesses system indices: [.ml-config], but in a future major version, direct access to system indices will be prevented by default
  results = self.client.eql.search(body=formatted_dsl, index=index_str, **kwargs)['hits']
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.10/3.10.5/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/homebrew/Cellar/python@3.10/3.10.5/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/__main__.py", line 34, in <module>
    main()
  File "/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/__main__.py", line 31, in main
    root(prog_name="detection_rules")
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/misc.py", line 444, in _wrapped
    return func(*args, **kwargs)
  File "/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/devtools.py", line 1045, in rule_survey
    counts = collector.count_from_rule(rules, start_time=start_time, end_time=end_time)
  File "/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py", line 291, in count_from_rule
    rule_results['search_count'] = self.count(query=rule.contents.data.query,
  File "/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py", line 273, in count
    results = self.search(query=query, language=language, index=index, start_time=start_time, end_time=end_time,
  File "/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py", line 183, in search
    results = self.client.eql.search(body=formatted_dsl, index=index_str, **kwargs)['hits']
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/elasticsearch/_sync/client/utils.py", line 414, in wrapped
    return api(*args, **kwargs)
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/elasticsearch/_sync/client/eql.py", line 303, in search
    return self.perform_request(  # type: ignore[return-value]
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/elasticsearch/_sync/client/_base.py", line 390, in perform_request
    return self._client.perform_request(
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/elasticsearch/_sync/client/_base.py", line 286, in perform_request
    meta, resp_body = self.transport.perform_request(
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/elastic_transport/_transport.py", line 329, in perform_request
    meta, raw_data = node.perform_request(
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/elastic_transport/_node/_http_urllib3.py", line 199, in perform_request
    raise err from None
elastic_transport.ConnectionTimeout: Connection timed out
python -m detection_rules dev test rule-survey  --es-user=elastic     17.13s user 1.29s system 1% cpu 17:05.22 total

@brokensound77 when you run it, does this happen?

I also tried running the command with the date-range flag, and I don't think it's being used.


(detection_dev) ➜  detection-rules git:(cleanup-survey-code) time python -m detection_rules dev test rule-survey --cloud-id=<cloud id> --es-user=elastic --es-password=<password> --kibana-user=elastic --kibana-password=<password> -d now-10s now
Loaded config file: /Users/stryker/workspace/Community/brokensound77/detection-rules/.detection-rules-cfg.yml

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Running survey against 679 rules
Saving detailed dump to: /Users/stryker/workspace/Community/brokensound77/detection-rules/surveys/20220823T131816L.json
/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py:277: ElasticsearchWarning: this request accesses system indices: [.apm-agent-configuration, .apm-custom-link, .async-search, .fleet-agents-7, .fleet-artifacts-7, .fleet-enrollment-api-keys-7, .fleet-policies-7, .fleet-policies-leader-7, .fleet-servers-7, .kibana_7.16.2_001, .kibana_task_manager_7.16.2_001, .ml-config, .security-7, .security-tokens-7, .transform-internal-007], but in a future major version, direct access to system indices will be prevented by default
  return self.client.count(body=formatted_dsl, index=index_str, q=lucene_query, allow_no_indices=True,
/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py:183: ElasticsearchWarning: this request accesses system indices: [.apm-agent-configuration, .apm-custom-link, .async-search, .fleet-agents-7, .fleet-artifacts-7, .fleet-enrollment-api-keys-7, .fleet-policies-7, .fleet-policies-leader-7, .fleet-servers-7, .kibana_7.16.2_001, .kibana_task_manager_7.16.2_001, .ml-config, .security-7, .security-tokens-7, .transform-internal-007], but in a future major version, direct access to system indices will be prevented by default
  results = self.client.eql.search(body=formatted_dsl, index=index_str, **kwargs)['hits']
/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py:183: ElasticsearchWarning: this request accesses system indices: [.apm-custom-link], but in a future major version, direct access to system indices will be prevented by default
  results = self.client.eql.search(body=formatted_dsl, index=index_str, **kwargs)['hits']
/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py:183: ElasticsearchWarning: this request accesses system indices: [.fleet-policies-leader-7], but in a future major version, direct access to system indices will be prevented by default
  results = self.client.eql.search(body=formatted_dsl, index=index_str, **kwargs)['hits']
/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py:183: ElasticsearchWarning: this request accesses system indices: [.kibana_task_manager_7.16.2_001], but in a future major version, direct access to system indices will be prevented by default
  results = self.client.eql.search(body=formatted_dsl, index=index_str, **kwargs)['hits']
/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py:183: ElasticsearchWarning: this request accesses system indices: [.fleet-agents-7], but in a future major version, direct access to system indices will be prevented by default
  results = self.client.eql.search(body=formatted_dsl, index=index_str, **kwargs)['hits']
/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py:183: ElasticsearchWarning: this request accesses system indices: [.fleet-artifacts-7], but in a future major version, direct access to system indices will be prevented by default
  results = self.client.eql.search(body=formatted_dsl, index=index_str, **kwargs)['hits']
/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py:183: ElasticsearchWarning: this request accesses system indices: [.fleet-policies-7], but in a future major version, direct access to system indices will be prevented by default
  results = self.client.eql.search(body=formatted_dsl, index=index_str, **kwargs)['hits']
/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py:183: ElasticsearchWarning: this request accesses system indices: [.fleet-servers-7], but in a future major version, direct access to system indices will be prevented by default
  results = self.client.eql.search(body=formatted_dsl, index=index_str, **kwargs)['hits']
/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py:183: ElasticsearchWarning: this request accesses system indices: [.security-tokens-7], but in a future major version, direct access to system indices will be prevented by default
  results = self.client.eql.search(body=formatted_dsl, index=index_str, **kwargs)['hits']
/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py:183: ElasticsearchWarning: this request accesses system indices: [.transform-internal-007], but in a future major version, direct access to system indices will be prevented by default
  results = self.client.eql.search(body=formatted_dsl, index=index_str, **kwargs)['hits']
/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py:183: ElasticsearchWarning: this request accesses system indices: [.fleet-enrollment-api-keys-7], but in a future major version, direct access to system indices will be prevented by default
  results = self.client.eql.search(body=formatted_dsl, index=index_str, **kwargs)['hits']
/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py:183: ElasticsearchWarning: this request accesses system indices: [.kibana_7.16.2_001], but in a future major version, direct access to system indices will be prevented by default
  results = self.client.eql.search(body=formatted_dsl, index=index_str, **kwargs)['hits']
/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py:183: ElasticsearchWarning: this request accesses system indices: [.security-7], but in a future major version, direct access to system indices will be prevented by default
  results = self.client.eql.search(body=formatted_dsl, index=index_str, **kwargs)['hits']
/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py:183: ElasticsearchWarning: this request accesses system indices: [.async-search], but in a future major version, direct access to system indices will be prevented by default
  results = self.client.eql.search(body=formatted_dsl, index=index_str, **kwargs)['hits']
/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py:183: ElasticsearchWarning: this request accesses system indices: [.apm-agent-configuration], but in a future major version, direct access to system indices will be prevented by default
  results = self.client.eql.search(body=formatted_dsl, index=index_str, **kwargs)['hits']
/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py:183: ElasticsearchWarning: this request accesses system indices: [.ml-config], but in a future major version, direct access to system indices will be prevented by default
  results = self.client.eql.search(body=formatted_dsl, index=index_str, **kwargs)['hits']
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.10/3.10.5/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/homebrew/Cellar/python@3.10/3.10.5/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/__main__.py", line 34, in <module>
    main()
  File "/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/__main__.py", line 31, in main
    root(prog_name="detection_rules")
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/misc.py", line 444, in _wrapped
    return func(*args, **kwargs)
  File "/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/devtools.py", line 1045, in rule_survey
    counts = collector.count_from_rule(rules, start_time=start_time, end_time=end_time)
  File "/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py", line 291, in count_from_rule
    rule_results['search_count'] = self.count(query=rule.contents.data.query,
  File "/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py", line 273, in count
    results = self.search(query=query, language=language, index=index, start_time=start_time, end_time=end_time,
  File "/Users/stryker/workspace/Community/brokensound77/detection-rules/detection_rules/eswrap.py", line 183, in search
    results = self.client.eql.search(body=formatted_dsl, index=index_str, **kwargs)['hits']
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/elasticsearch/_sync/client/utils.py", line 414, in wrapped
    return api(*args, **kwargs)
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/elasticsearch/_sync/client/eql.py", line 303, in search
    return self.perform_request(  # type: ignore[return-value]
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/elasticsearch/_sync/client/_base.py", line 390, in perform_request
    return self._client.perform_request(
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/elasticsearch/_sync/client/_base.py", line 286, in perform_request
    meta, resp_body = self.transport.perform_request(
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/elastic_transport/_transport.py", line 329, in perform_request
    meta, raw_data = node.perform_request(
  File "/Users/stryker/.virtualenvs/detection_dev/lib/python3.10/site-packages/elastic_transport/_node/_http_urllib3.py", line 199, in perform_request
    raise err from None
elastic_transport.ConnectionTimeout: Connection timed out
python -m detection_rules dev test rule-survey  --es-user=elastic    -d  now  15.61s user 1.31s system 2% cpu 13:57.34 total

@Mikaayenson
Copy link
Contributor

Mikaayenson commented Sep 2, 2022

Update Sep 2 2022

@brokensound77 @terrancedejesus I looked a little closer at this and decided to time the functions causing the bottle neck. Ironically, I was able to get the execution to fully complete, which leads me to believe the issue was due to the amount of data within my stack. I haven't tested a bunch of RTAs on the stack lately so I suspect there was less data to process. The bottleneck appears to be the collector.count_from_rule call.

  • After looking at the timing numbers, I'm wondering if the survey time is skewed by the timing delay causing by the code?
  • FWIW as this is a dev internal command. We probably should move forward with this and just note in the README / blog / CLI help that like all dev commands, they are really for internal development purposes. Probably should also state what the search count numbers mean (-1, 0, +1).

Other random thoughts.

  • Note : I used the config instead of passing creds.
  • Note: I tested on a different computer and didn't see any of the output that looked like ElasticsearchWarning: this request accesses system indices:, which I think is unrelated to this issue. Rather its related to system indices
  • Note: Looking back, I tested this on a different later stack of 8.3 (where as the other stack that failed was 7.16.2)
  • Note: After the first run, I no longer saw the search_count with the 109 hits.
=================================================================================================================================================
 rule_id                                name                                                                          search_count   alert_count
=================================================================================================================================================
 a87a4e42-1d82-44bd-b0bf-d9b7f91fb89e   Web Application Suspicious Activity: POST Request Declined                               0
 75ee75d8-c180-481c-ba88-ee50129a6aef   Web Application Suspicious Activity: Unauthorized Method                                 0
 43303fd4-4839-4e48-b2b2-803ab060758d   Web Application Suspicious Activity: No User Agent                                       0
 d49cc73f-7a16-4def-89ce-9fc7127d7820   Web Application Suspicious Activity: sqlmap User Agent                                   0
 027ff9ea-85e7-42e3-99d2-bbb7069e02eb   Potential Cookies Theft via Browser Debugging                                            0
 3115bd2c-0baa-4df0-80ea-45e474b5ef93   Agent Spoofing - Mismatched Agent ID                                                     0
 493834ca-f861-414c-8602-150d5505b777   Agent Spoofing - Multiple Hosts Using Same Agent                                       109
 665e7a4f-c58e-4fc6-bc83-87a7572670ac   WebServer Access Logs Deleted                                                            0
 7bcbb3ac-e533-41ad-a612-d6c3bf666aba   Tampering of Bash Command-Line History                                                   0
 b627cd12-dac4-11ec-9582-f661ea17fbcd   Elastic Agent Service Terminated                                                         0
 b0046934-486e-462f-9487-0d4cf9e429c6   Timestomping using Touch Command                                                        -1
 870aecc0-cea4-4110-af3f-e02e9b373655   Security Software Discovery via Grep                                                    -1
 c85eb82c-d2c8-485c-a36f-534f914b7663   Virtual Machine Fingerprinting via Grep                                                 -1
 41824afb-d68c-4d0e-bfee-474dac1fa56e   EggShell Backdoor Execution                                                              0
 a1a0375f-22c2-48c0-81a4-7c2d11cc6856   Potential Reverse Shell Activity via Terminal                                            0
 8acb7614-1d92-4359-bfcf-478b6d9de150   Suspicious JAVA Child Process                                                            0
 c3f5e1d8-910e-43b4-8d44-d748e498ca86   Potential JAVA/JNDI Exploitation Attempt                                                -1
 9c260313-c811-4ec8-ab89-8f6530e0246c   Hosts File Modified                                                                      0
 58ac2aa5-6718-427c-a845-5f3ac5af00ba   Zoom Meeting with no Passcode                                                            0
 93f47b6f-5728-4004-ba00-625083b3dcb0   Modification of Standard Authentication Module or Configuration                          0
 e6c1a552-7776-44ad-ae0f-8746cc07773c   Bash Shell Profile Modification                                                          0
 2215b8bd-1759-4ffa-8ab8-55c8e6b32e7f   SSH Authorized Keys File Modification                                                    0
 76152ca1-71d0-4003-9e37-0983e12832da   Potential Privilege Escalation via Sudoers File Modification                             0
 8a1b0278-0f9a-487d-96bd-d4833298e87a   Setuid / Setgid Bit Set via chmod                                                       -1
 f37f3054-d40b-49ac-aa9b-a786c74c58b8   Sudo Heap-Based Buffer Overflow Attempt                                                  0
 931e25a5-0f5e-4ae0-ba0d-9e94eff7e3a4   Sudoers File Modification                                                                0
 699e9fdb-b77c-4c01-995c-1c15019b9c43   Threat Intel Filebeat Module (v8.x) Indicator Match                                      0
 0c9a14d9-d65d-486f-9b5b-91e4e6b22bd0   Threat Intel Indicator Match                                                             0
    collector = CollectEvents(elasticsearch_client)
    start = time.time()
    details = collector.search_from_rule(rules, start_time=start_time, end_time=end_time)
    end = time.time()
    print(f'Survey search took {end - start} seconds')


    collector = CollectEvents(elasticsearch_client)
    start = time.time()
    counts = collector.count_from_rule(rules, start_time=start_time, end_time=end_time)
    end = time.time()
    print(f'Survey count took {end - start} seconds')

First run with 10 second window

(detection_dev) ➜  detection-rules git:(cleanup-survey-code) ✗ python -m detection_rules dev test rule-survey -d now-10s now
Loaded config file: /Users/stryker/workspace/ElasticGitHub/community/bksound/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Running survey against 680 rules
Saving detailed dump to: /Users/stryker/workspace/ElasticGitHub/community/bksound/detection-rules/surveys/20220902T140101L.json
Survey search took 45.89954900741577 seconds
Survey count took 322.0308530330658 seconds
(detection_dev) ➜  detection-rules git:(cleanup-survey-code) ✗

Second run with 10 second window

(detection_dev) ➜  detection-rules git:(cleanup-survey-code) ✗ python -m detection_rules dev test rule-survey -d now-10s now
Loaded config file: /Users/stryker/workspace/ElasticGitHub/community/bksound/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Running survey against 680 rules
Saving detailed dump to: /Users/stryker/workspace/ElasticGitHub/community/bksound/detection-rules/surveys/20220902T140919L.json
Survey search took 40.028501987457275 seconds
Survey count took 336.776211977005 seconds
(detection_dev) ➜  detection-rules git:(cleanup-survey-code) ✗

Third run with 30 second window

(detection_dev) ➜  detection-rules git:(cleanup-survey-code) ✗ python -m detection_rules dev test rule-survey -d now-30s now
Loaded config file: /Users/stryker/workspace/ElasticGitHub/community/bksound/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Running survey against 680 rules
Saving detailed dump to: /Users/stryker/workspace/ElasticGitHub/community/bksound/detection-rules/surveys/20220902T141609L.json
Survey search took 43.72844910621643 seconds
Survey count took 316.44426012039185 seconds
(detection_dev) ➜  detection-rules git:(cleanup-survey-code) ✗

Final (fourth) run with 60 second window

(detection_dev) ➜  detection-rules git:(cleanup-survey-code) ✗ python -m detection_rules dev test rule-survey -d now-60s now
Loaded config file: /Users/stryker/workspace/ElasticGitHub/community/bksound/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Running survey against 680 rules
Saving detailed dump to: /Users/stryker/workspace/ElasticGitHub/community/bksound/detection-rules/surveys/20220902T142519L.json
Survey search took 39.9447717666626 seconds
Survey count took 331.9557840824127 seconds
(detection_dev) ➜  detection-rules git:(cleanup-survey-code) ✗

Copy link
Contributor

@Mikaayenson Mikaayenson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I finally identified the bottleneck with this bug. If may be good to adjust the default to a smaller list of indices instead of * as well.

detection_rules/eswrap.py Outdated Show resolved Hide resolved
@Mikaayenson Mikaayenson self-requested a review September 6, 2022 20:53
Copy link
Contributor

@Mikaayenson Mikaayenson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Can you add a note somewhere (readme, CLI.md, docstring or something that explains the meaning of search count numbers mean (-1, 0, +1) in the table generated.

Copy link
Contributor

@terrancedejesus terrancedejesus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work! LGTM

@brokensound77 brokensound77 merged commit 332ea40 into elastic:main Sep 6, 2022
protectionsmachine pushed a commit that referenced this pull request Sep 6, 2022
* Cleanup rule survey code

* default to only unique-ing on process name for lucene rules

* fix bug in kibana url parsing by removing redundant port from domain

* update search-alerts columns and nest fields

* fix rule.contents.data.index

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

(cherry picked from commit 332ea40)
protectionsmachine pushed a commit that referenced this pull request Sep 6, 2022
* Cleanup rule survey code

* default to only unique-ing on process name for lucene rules

* fix bug in kibana url parsing by removing redundant port from domain

* update search-alerts columns and nest fields

* fix rule.contents.data.index

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

(cherry picked from commit 332ea40)
protectionsmachine pushed a commit that referenced this pull request Sep 6, 2022
* Cleanup rule survey code

* default to only unique-ing on process name for lucene rules

* fix bug in kibana url parsing by removing redundant port from domain

* update search-alerts columns and nest fields

* fix rule.contents.data.index

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

(cherry picked from commit 332ea40)
@brokensound77 brokensound77 deleted the cleanup-survey-code branch September 7, 2022 02:56
protectionsmachine pushed a commit that referenced this pull request Sep 7, 2022
* Cleanup rule survey code

* default to only unique-ing on process name for lucene rules

* fix bug in kibana url parsing by removing redundant port from domain

* update search-alerts columns and nest fields

* fix rule.contents.data.index

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

(cherry picked from commit 332ea40)
protectionsmachine pushed a commit that referenced this pull request Sep 7, 2022
* Cleanup rule survey code

* default to only unique-ing on process name for lucene rules

* fix bug in kibana url parsing by removing redundant port from domain

* update search-alerts columns and nest fields

* fix rule.contents.data.index

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

(cherry picked from commit 332ea40)
protectionsmachine pushed a commit that referenced this pull request Sep 7, 2022
* Cleanup rule survey code

* default to only unique-ing on process name for lucene rules

* fix bug in kibana url parsing by removing redundant port from domain

* update search-alerts columns and nest fields

* fix rule.contents.data.index

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

(cherry picked from commit 332ea40)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport: auto bug Something isn't working python Internal python for the repository schema v8.4.0 v8.5.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants