ML-experimental-detections-20200506-3

Experimental rules: 0
Experimental ML jobs: 1
Other files: 1

---

DGA release:
date: 2021-05-06T22:40:00Z
For details reference: https://github.com/elastic/detection-rules/blob/main/docs/ML_DGA.md

ML-experimental-detections-20210804-6

changelog

detections added

URL Spoofing

• Experimental Rules
  • 47b1a804-4f65-40b0-a7ef-fdac3c00b00c: Added new rule for url_spoofing.prediction: phishing (model prediction) or abuseurl_label: 1 (threat intelligence enrichment)

---

Registry of experimental detections

Experimental detections

expand to view

• rules and dashboards can be imported via Kibana
• jobs and datafeeds can be imported using the CLI or Kibana devtools

Refer to the experimental-maching-learning docs for more details

detection ID	type	relative path
47b1a804-4f65-40b0-a7ef-fdac3c00b00c	rule	url_spoof/rule/url_spoof_ml_predicted_malicious_url.ndjson
problem_child_high_sum_by_parent	anomaly_detection	problem_child/anomaly_detection/problem_child_high_sum_by_parent.json
problem_child_high_sum_by_user	anomaly_detection	problem_child/anomaly_detection/problem_child_high_sum_by_user.json
problem_child_rare_process_by_parent	anomaly_detection	problem_child/anomaly_detection/problem_child_rare_process_by_parent.json
problem_child_rare_process_by_user	anomaly_detection	problem_child/anomaly_detection/problem_child_rare_process_by_user.json
problem_child_high_sum_by_host	anomaly_detection	problem_child/anomaly_detection/problem_child_high_sum_by_host.json
problem_child_rare_process_by_host	anomaly_detection	problem_child/anomaly_detection/problem_child_rare_process_by_host.json
problem_child_high_sum_by_parent	datafeed	problem_child/datafeed/problem_child_high_sum_by_parent.json
problem_child_high_sum_by_user	datafeed	problem_child/datafeed/problem_child_high_sum_by_user.json
problem_child_rare_process_by_parent	datafeed	problem_child/datafeed/problem_child_rare_process_by_parent.json
problem_child_rare_process_by_user	datafeed	problem_child/datafeed/problem_child_rare_process_by_user.json
problem_child_high_sum_by_host	datafeed	problem_child/datafeed/problem_child_high_sum_by_host.json
problem_child_rare_process_by_host	datafeed	problem_child/datafeed/problem_child_rare_process_by_host.json
9a2e372a-cbeb-4ad6-a288-017ef086324c	rule	problem_child/rule/problem_child_ml_high_probability_suspicious_windows_event.ndjson
a5cb4cd7-ba05-47e8-a815-f95c21719ded	rule	problem_child/rule/problem_child_ml_rare_suspicious_process_by_user.ndjson
9b98d945-2cce-45e5-aa84-4b021af0e153	rule	problem_child/rule/problem_child_ml_suspicious_process_cluster_by_parent.ndjson
ff590871-371b-468f-8cd8-2876b54c53bd	rule	problem_child/rule/problem_child_ml_suspicious_process_cluster_by_user.ndjson
ae7c2f69-0c51-4b02-ad54-d3d75023da8b	rule	problem_child/rule/problem_child_ml_rare_suspicious_process_by_parent.ndjson
34184d4e-ef61-477b-8d76-5c93448c29bf	rule	problem_child/rule/problem_child_ml_predicted_suspicious_windows_event.ndjson
415d6863-7676-401f-aa8d-62f59a28e849	rule	problem_child/rule/problem_child_ml_rare_suspicious_process_by_host.ndjson
86d57ec4-ace5-4456-8145-02e6f0cdd71a	rule	problem_child/rule/problem_child_ml_suspicious_process_cluster_by_host.ndjson
dga_high_sum_probability	anomaly_detection	dga/anomaly_detection/dga_high_sum_probability.json
dga_high_sum_probability	datafeed	dga/datafeed/dga_high_sum_probability.json
997ec71d-bddc-4513-b6f1-193f601fd420	rule	dga/rule/dga_command_and_control_high_sum_scores.ndjson
170b35d4-d944-4264-a8ca-3118ae2e1534	rule	dga/rule/dga_command_and_control_ml_sunburst_domain.ndjson
64116bb2-0f2c-4cf6-9df4-9973452b4d4b	rule	dga/rule/dga_command_and_control_ml_predicted_domain.ndjson
a020dadb-3da2-4252-91e9-b0fc148823e2	rule	dga/rule/dga_command_and_control_ml_probable_domain.ndjson
None	dashboard	dga/dashboard/dga_dashboard.ndjson

ML-experimental-detections-20210603-5

changelog

detections updated

problem child

• ML jobs
  • problem_child_high_sum_by_parent: Added an additional detector to detect anomalies on high_sum of blocklist_label
  • problem_child_high_sum_by_host: Added an additional detector to detect anomalies on high_sum of blocklist_label
  • problem_child_high_sum_by_user: Added an additional detector to detect anomalies on high_sum of blocklist_label
• Experimental Rules
  • 34184d4e-ef61-477b-8d76-5c93448c29bf: Added a check for blocklist_label:1
  • 9a2e372a-cbeb-4ad6-a288-017ef086324c: Added a check for blocklist_label:1

---

Registry of experimental detections

Experimental detections

expand to view

• rules and dashboards can be imported via Kibana
• jobs and datafeeds can be imported using the CLI or Kibana devtools

Refer to the experimental-maching-learning docs for more details

detection ID	type	relative path
problem_child_high_sum_by_parent	anomaly_detection	problem_child/anomaly_detection/problem_child_high_sum_by_parent.json
problem_child_high_sum_by_user	anomaly_detection	problem_child/anomaly_detection/problem_child_high_sum_by_user.json
problem_child_rare_process_by_parent	anomaly_detection	problem_child/anomaly_detection/problem_child_rare_process_by_parent.json
problem_child_rare_process_by_user	anomaly_detection	problem_child/anomaly_detection/problem_child_rare_process_by_user.json
problem_child_high_sum_by_host	anomaly_detection	problem_child/anomaly_detection/problem_child_high_sum_by_host.json
problem_child_rare_process_by_host	anomaly_detection	problem_child/anomaly_detection/problem_child_rare_process_by_host.json
problem_child_high_sum_by_parent	datafeed	problem_child/datafeed/problem_child_high_sum_by_parent.json
problem_child_high_sum_by_user	datafeed	problem_child/datafeed/problem_child_high_sum_by_user.json
problem_child_rare_process_by_parent	datafeed	problem_child/datafeed/problem_child_rare_process_by_parent.json
problem_child_rare_process_by_user	datafeed	problem_child/datafeed/problem_child_rare_process_by_user.json
problem_child_high_sum_by_host	datafeed	problem_child/datafeed/problem_child_high_sum_by_host.json
problem_child_rare_process_by_host	datafeed	problem_child/datafeed/problem_child_rare_process_by_host.json
9a2e372a-cbeb-4ad6-a288-017ef086324c	rule	problem_child/rule/problem_child_ml_high_probability_suspicious_windows_event.ndjson
a5cb4cd7-ba05-47e8-a815-f95c21719ded	rule	problem_child/rule/problem_child_ml_rare_suspicious_process_by_user.ndjson
9b98d945-2cce-45e5-aa84-4b021af0e153	rule	problem_child/rule/problem_child_ml_suspicious_process_cluster_by_parent.ndjson
ff590871-371b-468f-8cd8-2876b54c53bd	rule	problem_child/rule/problem_child_ml_suspicious_process_cluster_by_user.ndjson
ae7c2f69-0c51-4b02-ad54-d3d75023da8b	rule	problem_child/rule/problem_child_ml_rare_suspicious_process_by_parent.ndjson
34184d4e-ef61-477b-8d76-5c93448c29bf	rule	problem_child/rule/problem_child_ml_predicted_suspicious_windows_event.ndjson
415d6863-7676-401f-aa8d-62f59a28e849	rule	problem_child/rule/problem_child_ml_rare_suspicious_process_by_host.ndjson
86d57ec4-ace5-4456-8145-02e6f0cdd71a	rule	problem_child/rule/problem_child_ml_suspicious_process_cluster_by_host.ndjson
dga_high_sum_probability	anomaly_detection	dga/anomaly_detection/dga_high_sum_probability.json
dga_high_sum_probability	datafeed	dga/datafeed/dga_high_sum_probability.json
997ec71d-bddc-4513-b6f1-193f601fd420	rule	dga/rule/dga_command_and_control_high_sum_scores.ndjson
170b35d4-d944-4264-a8ca-3118ae2e1534	rule	dga/rule/dga_command_and_control_ml_sunburst_domain.ndjson
64116bb2-0f2c-4cf6-9df4-9973452b4d4b	rule	dga/rule/dga_command_and_control_ml_predicted_domain.ndjson
a020dadb-3da2-4252-91e9-b0fc148823e2	rule	dga/rule/dga_command_and_control_ml_probable_domain.ndjson
None	dashboard	dga/dashboard/dga_dashboard.ndjson

ML-experimental-detections-20210602-4

changelog

Rules are now stored as ndjson files rather than in toml format, to allow for importing via Kibana

detections added

problem child

• ML jobs
  • problem_child_high_sum_by_parent
  • problem_child_high_sum_by_host
  • problem_child_high_sum_by_user
  • problem_child_rare_process_by_parent
  • problem_child_rare_process_by_host
  • problem_child_rare_process_by_user
• Experimental Rules
  • 34184d4e-ef61-477b-8d76-5c93448c29bf: Search rule to detect on malicious activity predicted by the supervised ProblemChild model
  • 9a2e372a-cbeb-4ad6-a288-017ef086324c: Search rule to detect on malicious activity predicted by the supervised ProblemChild model with high probability
  • 9b98d945-2cce-45e5-aa84-4b021af0e153: ML rule to detect on malicious parent-child activity identified by an ML job
  • 86d57ec4-ace5-4456-8145-02e6f0cdd71a: ML rule to detect on malicious process activity from a particular host, identified by an ML job
  • ff590871-371b-468f-8cd8-2876b54c53bd: ML rule to detect on malicious process activity from a particular user, identified by an ML job
  • ae7c2f69-0c51-4b02-ad54-d3d75023da8b: ML rule to detect a rare process spawned by a parent process, identified by an ML job
  • 415d6863-7676-401f-aa8d-62f59a28e849: ML rule to detect a rare process spawned on a host, identified by an ML job
  • a5cb4cd7-ba05-47e8-a815-f95c21719ded: ML rule to detect a rare process spawned on a user, identified by an ML job

DGA

Updated file names and job ID references

---

Registry of experimental detections

Experimental detections

expand to view

• rules and dashboards can be imported via Kibana
• jobs and datafeeds can be imported using the CLI or Kibana devtools

Refer to the experimental-maching-learning docs for more details

detection ID	type	relative path
problem_child_high_sum_by_parent	anomaly_detection	problem_child/anomaly_detection/problem_child_high_sum_by_parent.json
problem_child_high_sum_by_user	anomaly_detection	problem_child/anomaly_detection/problem_child_high_sum_by_user.json
problem_child_rare_process_by_parent	anomaly_detection	problem_child/anomaly_detection/problem_child_rare_process_by_parent.json
problem_child_rare_process_by_user	anomaly_detection	problem_child/anomaly_detection/problem_child_rare_process_by_user.json
problem_child_high_sum_by_host	anomaly_detection	problem_child/anomaly_detection/problem_child_high_sum_by_host.json
problem_child_rare_process_by_host	anomaly_detection	problem_child/anomaly_detection/problem_child_rare_process_by_host.json
problem_child_high_sum_by_parent	datafeed	problem_child/datafeed/problem_child_high_sum_by_parent.json
problem_child_high_sum_by_user	datafeed	problem_child/datafeed/problem_child_high_sum_by_user.json
problem_child_rare_process_by_parent	datafeed	problem_child/datafeed/problem_child_rare_process_by_parent.json
problem_child_rare_process_by_user	datafeed	problem_child/datafeed/problem_child_rare_process_by_user.json
problem_child_high_sum_by_host	datafeed	problem_child/datafeed/problem_child_high_sum_by_host.json
problem_child_rare_process_by_host	datafeed	problem_child/datafeed/problem_child_rare_process_by_host.json
9a2e372a-cbeb-4ad6-a288-017ef086324c	rule	problem_child/rule/problem_child_ml_high_probability_suspicious_windows_event.ndjson
a5cb4cd7-ba05-47e8-a815-f95c21719ded	rule	problem_child/rule/problem_child_ml_rare_suspicious_process_by_user.ndjson
9b98d945-2cce-45e5-aa84-4b021af0e153	rule	problem_child/rule/problem_child_ml_suspicious_process_cluster_by_parent.ndjson
ff590871-371b-468f-8cd8-2876b54c53bd	rule	problem_child/rule/problem_child_ml_suspicious_process_cluster_by_user.ndjson
ae7c2f69-0c51-4b02-ad54-d3d75023da8b	rule	problem_child/rule/problem_child_ml_rare_suspicious_process_by_parent.ndjson
34184d4e-ef61-477b-8d76-5c93448c29bf	rule	problem_child/rule/problem_child_ml_predicted_suspicious_windows_event.ndjson
415d6863-7676-401f-aa8d-62f59a28e849	rule	problem_child/rule/problem_child_ml_rare_suspicious_process_by_host.ndjson
86d57ec4-ace5-4456-8145-02e6f0cdd71a	rule	problem_child/rule/problem_child_ml_suspicious_process_cluster_by_host.ndjson
dga_high_sum_probability	anomaly_detection	dga/anomaly_detection/dga_high_sum_probability.json
dga_high_sum_probability	datafeed	dga/datafeed/dga_high_sum_probability.json
997ec71d-bddc-4513-b6f1-193f601fd420	rule	dga/rule/dga_command_and_control_high_sum_scores.ndjson
170b35d4-d944-4264-a8ca-3118ae2e1534	rule	dga/rule/dga_command_and_control_ml_sunburst_domain.ndjson
64116bb2-0f2c-4cf6-9df4-9973452b4d4b	rule	dga/rule/dga_command_and_control_ml_predicted_domain.ndjson
a020dadb-3da2-4252-91e9-b0fc148823e2	rule	dga/rule/dga_command_and_control_ml_probable_domain.ndjson
None	dashboard	dga/dashboard/dga_dashboard.ndjson

ML-URLSpoof-20210804-1

model name: urlspoof_20210803_1.0
sha256: 4cbd8d82d382864d28147c5f80ac86108e774319bbe5d2c4c9f3c68d9f86e01e
for details, reference: https://github.com/elastic/detection-rules/tree/main/docs/experimental-machine-learning

---

changelog

This is the first release package for URL Spoofing. It consists of the following:

• Feature Extraction Scripts:

  • ml_urlspoof_char_continuity_script.json: Calculate the continuity of different parts of a the domain (i.e. number of consecutive characters before seeing a number)
  • ml_urlspoof_domain_entropy_script.json: Calculate the entropy of the URL domain
  • ml_urlspoof_keyword_extractor_script.json: Extract keywords of interest from certain features
  • ml_urlspoof_ngrams_extractor_script.json: Extract ngrams from certain features
  • ml_urlspoof_remove_features_script.json: Remove extra fields created for prediction purposes to avoid cluttering incoming documents - this will NOT remove any of your original fields in your documents
  • ml_urlspoof_tld_keyword_extractor_script.json: Extract top level domain related keywords of interest from certain features

• Model:

  • ml_urlspoof_model.json: Supervised model to classify URLs as malicious vs benign

• Inference Pipeline:

  • ml_urlspoof_inference_pipeline.json: Inference pipeline to make predictions on URLs using the URL Spoofing model and threat intelligence enrichments

• Training Pipeline:

  • ml_urlspoof_features_pipeline.json: Training pipeline used to train the URL Spoofing model - this is primarily for analysts looking for a starting point to train their own model

ML-ProblemChild-20210602-1

model name: problemchild_20210526_1.0
sha256: 4f4342b8559886f93702f571cdd320da7a176a28b2784a9d8c4497eeeca9b3bd
for details, reference: https://github.com/elastic/detection-rules/blob/main/docs/ML_DGA.md

---

Changelog

This is the first release package for ProblemChild. It consists of the following:

• Feature extraction scripts:

  • ML_ProblemChild_features_script.json: Extract required features irrespective of agent being used
  • ML_ProblemChild_normalize_ppath_script.json: Extract a normalized path feature
  • ML_ProblemChild_ngram_extractor_script.json: Extract ngrams from certain features

• Model:

  • ML_ProblemChild_model.json: Supervised model to classify incoming events as malicious vs benign

• Blocklist script:

  • ML_ProblemChild_blocklist_script.json: Blocklist script to override model verdict

• Inference pipeline:

  • ML_ProblemChild_inference_pipeline.json: Inference pipeline to make predictions on events using the ProblemChild model/blocklist

• Ingest pipeline:

  • ML_ProblemChild_ingest_pipeline.json: Ingest pipeline to run ML_ProblemChild_inference_pipeline only on Windows process events

ML-HostRiskScore-20210728-1

for details, reference: https://github.com/elastic/detection-rules/tree/main/docs/experimental-machine-learning

---

Changelog

This is the first release package for Host Risk Score. It consists of the following:

• Scripts, ingest pipelines and transforms used to calculate and update risk score across all hosts in your environment
• dashboards.ndjson contains all the assets required for two dashboards- "Current Risky Hosts", which shows the Top 20 currently suspicious hosts in your environment and "Host Risk Drilldown" which shows a more detailed breakdown of various types of activity taking place on hosts in your environment

ML-DGA-20210629-5

model name: dga_1611725_2.0
sha256: 29748b8ce3b3aed528079044ab0af881477beeb86c904bc939ea04a7f087b046
for details, reference: https://github.com/elastic/detection-rules/tree/main/docs/experimental-machine-learning

---

changelog

• casing for all files changed to lowercase

ML-DGA-20210604-4

model name: dga_1611725_2.0
sha256: 29748b8ce3b3aed528079044ab0af881477beeb86c904bc939ea04a7f087b046
for details, reference: https://github.com/elastic/detection-rules/tree/main/docs/experimental-machine-learning

---

changelog

• fix the reference to the inference pipeline name to accurately reflect the name change to ML_DGA_inference_pipeline

ML-DGA-20210602-3

model name: dga_1611725_2.0
sha256: 29748b8ce3b3aed528079044ab0af881477beeb86c904bc939ea04a7f087b046
for details, reference: https://github.com/elastic/detection-rules/blob/main/docs/ML_DGA.md

---

changelog

No changes to model or functionality, just to filenames and references to script IDs to be compatible with updates in
the CLI.

Note: previous release will no longer work with the changes
to the CLI. To use these previous DGA releases, use the CLI from the 7.12 branch