Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/rule_execution_logic/threat_match·ts - detection engine api security and spaces enabled - rule execution logic Threat match type rules terms and match should have the same alerts with pagination #155304

kibanamachine · 2023-04-19T17:12:17Z

A test failed on a tracked branch

Error: expected [ { 'kibana.version': '8.8.0-SNAPSHOT',
    'kibana.alert.rule.category': 'Indicator Match Rule',
    'kibana.alert.rule.consumer': 'siem',
    'kibana.alert.rule.producer': 'siem',
    'kibana.alert.rule.revision': 0,
    'kibana.alert.rule.rule_type_id': 'siem.indicatorRule',
    'kibana.space_ids': [ 'default' ],
    'kibana.alert.rule.tags': [],
    agent: 
     { ephemeral_id: '1b4978a0-48be-49b1-ac96-323425b389ab',
       hostname: 'zeek-sensor-amsterdam',
       id: 'e52588e6-7aa3-4c89-a2c4-d6bc5c286db1',
       type: 'auditbeat',
       version: '8.0.0' },
    auditd: 
     { data: [Object],
       message_type: 'user_err',
       result: 'fail',
       sequence: 2267,
       session: 'unset',
       summary: [Object] },
    cloud: 
     { instance: [Object],
       provider: 'digitalocean',
       region: 'ams3' },
    ecs: { version: '1.0.0-beta2' },
    host: 
     { architecture: 'x86_64',
       containerized: false,
       hostname: 'zeek-sensor-amsterdam',
       id: '2ce8b1e7d69e4a1d9c6bcddc473da9d9',
       name: 'zeek-sensor-amsterdam',
       os: [Object] },
    network: { direction: 'incoming' },
    process: { executable: '/usr/sbin/sshd', pid: 32739 },
    service: { type: 'auditd' },
    source: { ip: '46.101.47.213' },
    user: { audit: [Object], id: '0', name: 'root' },
    threat: { enrichments: [Object] },
    'event.action': 'error',
    'event.category': 'user-login',
    'event.module': 'auditd',
    'event.kind': 'signal',
    'kibana.alert.original_time': '2019-02-19T20:42:05.202Z',
    'kibana.alert.ancestors': [ [Object] ],
    'kibana.alert.status': 'active',
    'kibana.alert.workflow_status': 'open',
    'kibana.alert.depth': 1,
    'kibana.alert.severity': 'high',
    'kibana.alert.risk_score': 55,
    'kibana.alert.rule.actions': [],
    'kibana.alert.rule.author': [],
    'kibana.alert.rule.created_by': 'elastic',
    'kibana.alert.rule.description': 'Detecting root and admin users',
    'kibana.alert.rule.enabled': true,
    'kibana.alert.rule.exceptions_list': [],
    'kibana.alert.rule.false_positives': [],
    'kibana.alert.rule.from': '1900-01-01T00:00:00.000Z',
    'kibana.alert.rule.immutable': false,
    'kibana.alert.rule.interval': '5m',
    'kibana.alert.rule.indices': [ 'auditbeat-*' ],
    'kibana.alert.rule.max_signals': 100,
    'kibana.alert.rule.references': [],
    'kibana.alert.rule.risk_score_mapping': [],
    'kibana.alert.rule.severity_mapping': [],
    'kibana.alert.rule.threat': [],
    'kibana.alert.rule.to': 'now',
    'kibana.alert.rule.type': 'threat_match',
    'kibana.alert.rule.updated_by': 'elastic',
    'kibana.alert.rule.version': 1,
    'kibana.alert.rule.risk_score': 55,
    'kibana.alert.rule.severity': 'high',
    'kibana.alert.original_event.action': 'error',
    'kibana.alert.original_event.category': 'user-login',
    'kibana.alert.original_event.module': 'auditd' },
  { 'kibana.version': '8.8.0-SNAPSHOT',
    'kibana.alert.rule.category': 'Indicator Match Rule',
    'kibana.alert.rule.consumer': 'siem',
    'kibana.alert.rule.producer': 'siem',
    'kibana.alert.rule.revision': 0,
    'kibana.alert.rule.rule_type_id': 'siem.indicatorRule',
    'kibana.space_ids': [ 'default' ],
    'kibana.alert.rule.tags': [],
    agent: 
     { ephemeral_id: '1b4978a0-48be-49b1-ac96-323425b389ab',
       hostname: 'zeek-sensor-amsterdam',
       id: 'e52588e6-7aa3-4c89-a2c4-d6bc5c286db1',
       type: 'auditbeat',
       version: '8.0.0' },
    auditd: 
     { data: [Object],
       message_type: 'user_login',
       result: 'fail',
       sequence: 2266,
       session: 'unset',
       summary: [Object] },
    cloud: 
     { instance: [Object],
       provider: 'digitalocean',
       region: 'ams3' },
    ecs: { version: '1.0.0-beta2' },
    host: 
     { architecture: 'x86_64',
       containerized: false,
       hostname: 'zeek-sensor-amsterdam',
       id: '2ce8b1e7d69e4a1d9c6bcddc473da9d9',
       name: 'zeek-sensor-amsterdam',
       os: [Object] },
    network: { direction: 'incoming' },
    process: { executable: '/usr/sbin/sshd', pid: 32739 },
    service: { type: 'auditd' },
    source: { ip: '46.101.47.213' },
    user: { audit: [Object], id: '0', name: 'root' },
    threat: { enrichments: [Object] },
    'event.action': 'logged-in',
    'event.category': 'user-login',
    'event.module': 'auditd',
    'event.kind': 'signal',
    'kibana.alert.original_time': '2019-02-19T20:42:05.194Z',
    'kibana.alert.ancestors': [ [Object] ],
    'kibana.alert.status': 'active',
    'kibana.alert.workflow_status': 'open',
    'kibana.alert.depth': 1,
    'kibana.alert.severity': 'high',
    'kibana.alert.risk_score': 55,
    'kibana.alert.rule.actions': [],
    'kibana.alert.rule.author': [],
    'kibana.alert.rule.created_by': 'elastic',
    'kibana.alert.rule.description': 'Detecting root and admin users',
    'kibana.alert.rule.enabled': true,
    'kibana.alert.rule.exceptions_list': [],
    'kibana.alert.rule.false_positives': [],
    'kibana.alert.rule.from': '1900-01-01T00:00:00.000Z',
    'kibana.alert.rule.immutable': false,
    'kibana.alert.rule.interval': '5m',
    'kibana.alert.rule.indices': [ 'auditbeat-*' ],
    'kibana.alert.rule.max_signals': 100,
    'kibana.alert.rule.references': [],
    'kibana.alert.rule.risk_score_mapping': [],
    'kibana.alert.rule.severity_mapping': [],
    'kibana.alert.rule.threat': [],
    'kibana.alert.rule.to': 'now',
    'kibana.alert.rule.type': 'threat_match',
    'kibana.alert.rule.updated_by': 'elastic',
    'kibana.alert.rule.version': 1,
    'kibana.alert.rule.risk_score': 55,
    'kibana.alert.rule.severity': 'high',
    'kibana.alert.original_event.action': 'logged-in',
    'kibana.alert.original_event.category': 'user-login',
    'kibana.alert.original_event.module': 'auditd' },
  { 'kibana.version': '8.8.0-SNAPSHOT',
    'kibana.alert.rule.category': 'Indicator Match Rule',
    'kibana.alert.rule.consumer': 'siem',
    'kibana.alert.rule.producer': 'siem',
    'kibana.alert.rule.revision': 0,
    'kibana.alert.rule.rule_type_id': 'siem.indicatorRule',
    'kibana.space_ids': [ 'default' ],
    'kibana.alert.rule.tags': [],
    agent: 
     { ephemeral_id: '1b4978a0-48be-49b1-ac96-323425b389ab',
       hostname: 'zeek-sensor-amsterdam',
       id: 'e52588e6-7aa3-4c89-a2c4-d6bc5c286db1',
       type: 'auditbeat',
       version: '8.0.0' },
    auditd: 
     { data: [Object],
       message_type: 'user_login',
       result: 'fail',
       sequence: 2265,
       session: 'unset',
       summary: [Object] },
    cloud: 
     { instance: [Object],
       provider: 'digitalocean',
       region: 'ams3' },
    ecs: { version: '1.0.0-beta2' },
    host: 
     { architecture: 'x86_64',
       containerized: false,
       hostname: 'zeek-sensor-amsterdam',
       id: '2ce8b1e7d69e4a1d9c6bcddc473da9d9',
       name: 'zeek-sensor-amsterdam',
       os: [Object] },
    network: { direction: 'incoming' },
    process: { executable: '/usr/sbin/sshd', pid: 32739 },
    service: { type: 'auditd' },
    source: { ip: '46.101.47.213' },
    user: { audit: [Object], id: '0', name: 'root' },
    threat: { enrichments: [Object] },
    'event.action': 'logged-in',
    'event.category': 'user-login',
    'event.module': 'auditd',
    'event.kind': 'signal',
    'kibana.alert.original_time': '2019-02-19T20:42:05.190Z',
    'kibana.alert.ancestors': [ [Object] ],
    'kibana.alert.status': 'active',
    'kibana.alert.workflow_status': 'open',
    'kibana.alert.depth': 1,
    'kibana.alert.severity': 'high',
    'kibana.alert.risk_score': 55,
    'kibana.alert.rule.actions': [],
    'kibana.alert.rule.author': [],
    'kibana.alert.rule.created_by': 'elastic',
    'kibana.alert.rule.description': 'Detecting root and admin users',
    'kibana.alert.rule.enabled': true,
    'kibana.alert.rule.exceptions_list': [],
    'kibana.alert.rule.false_positives': [],
    'kibana.alert.rule.from': '1900-01-01T00:00:00.000Z',
    'kibana.alert.rule.immutable': false,
    'kibana.alert.rule.interval': '5m',
    'kibana.alert.rule.indices': [ 'auditbeat-*' ],
    'kibana.alert.rule.max_signals': 100,
    'kibana.alert.rule.references': [],
[report_failure] output truncated to 8192 characters

First failure: CI Build - main

The text was updated successfully, but these errors were encountered:

elasticmachine · 2023-04-19T17:12:41Z

Pinging @elastic/security-solution (Team: SecuritySolution)

elasticmachine · 2023-04-20T06:27:22Z

Pinging @elastic/security-detections-response (Team:Detections and Resp)

kibanamachine · 2023-04-20T08:02:54Z

New failure: CI Build - main

jbudz · 2023-04-27T15:56:17Z

/skip

kibanamachine · 2023-04-27T23:56:41Z

New failure: CI Build - 8.8

kibanamachine · 2023-04-28T12:14:33Z

New failure: CI Build - main

jbudz · 2023-04-28T12:30:08Z

Skipped

main: 24927cb
8.8: d050720

rylnd · 2023-05-17T18:47:26Z

For posterity: this test was introduced to cover the changes in #144511.

As an aside: the failure message for this test is not helpful as the diff is too large to output. Perhaps it would make sense to compare these alerts in series, so that the failure shows two single objects? Just a thought.

yctercero · 2023-05-18T16:47:32Z

Merged in the PR unskipping these as running them multiple times locally and on the flakey test runner did not reveal any failures.

Lets keep in mind @rylnd 's suggestion if this pops up again.

PR: #156489

rylnd · 2023-05-18T17:17:58Z

If anyone happens to have reproduced this locally and was able to actually observe full failure message, please share here! Do we think this was simply a false negative?

kibanamachine · 2023-05-22T11:55:34Z

New failure: CI Build - main

kibanamachine · 2023-05-22T12:06:38Z

New failure: CI Build - main

mistic · 2023-05-22T14:02:07Z

Skipped.

main: 39dad65

## Unskip IM rule tests It's a tricky test fail, which not reproduces locally, also in the [PR] CI can be green(#156489) but eventually [can failed on main](#155304) In this attempt, I change the test to narrow amount of alerts for a rule which has 2 potential benefits: 1. if there is a resources problem this rule will execute faster 2. We can read the error message if it fails on the maiт. --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

yctercero · 2023-06-27T15:05:01Z

Unskipped here #160094

kibanamachine · 2023-06-30T12:53:41Z

New failure: CI Build - main

MadameSheema · 2023-07-03T08:29:35Z

Hey @yctercero may you please take a look at this new failure when you have the chance? Thanks!

yctercero · 2023-07-03T21:51:26Z

@MadameSheema the detections team is working 8.10 to address all open flakey and skipped tests. Will add this one to the list.

nkhristinin · 2023-07-05T07:59:21Z

There another PR which should address it

kibanamachine · 2023-07-05T17:00:34Z

New failure: CI Build - main

WafaaNasr · 2023-07-13T09:30:06Z

Running it through flaky runner https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2625

WafaaNasr · 2023-07-13T10:34:29Z

After successfully running this test through the Flaky Test Runner for 100 iterations without any failures, it has been determined that the test is not truly flaky or failing. As a result, the ticket can be closed.

kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Apr 19, 2023

botelastic bot added the needs-team Issues missing a team label label Apr 19, 2023

kibanamachine added the Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. label Apr 19, 2023

botelastic bot removed the needs-team Issues missing a team label label Apr 19, 2023

MadameSheema added the Team:Detections and Resp Security Detection Response Team label Apr 20, 2023

MadameSheema added the Team:Security Solution Platform Security Solution Platform Team label Apr 20, 2023

MadameSheema assigned nkhristinin Apr 20, 2023

kibanamachine added the blocker label Apr 27, 2023

jbudz added a commit that referenced this issue Apr 28, 2023

skip flaky suite (#155304)

24927cb

jbudz added a commit that referenced this issue Apr 28, 2023

skip flaky suite (#155304)

d050720

jbudz added skipped-test v8.8.0 v8.9.0 labels Apr 28, 2023

yctercero added Team:Detection Engine Security Solution Detection Engine Area and removed Team:Security Solution Platform Security Solution Platform Team labels May 14, 2023

vitaliidm mentioned this issue May 18, 2023

[Security Solution][Detection Engine] fixes wildcard in field names issue for query language in detection rules #157981

Merged

2 tasks

yctercero closed this as completed May 18, 2023

kibanamachine removed skipped-test blocker v8.8.0 v8.9.0 labels May 18, 2023

kibanamachine reopened this May 22, 2023

mistic added a commit that referenced this issue May 22, 2023

skip flaky suite (#155304)

39dad65

mistic added v8.9.0 blocker and removed fixed labels May 22, 2023

MadameSheema added the skipped-test label May 23, 2023

delanni pushed a commit to delanni/kibana that referenced this issue May 25, 2023

skip flaky suite (elastic#155304)

26fc2ea

nkhristinin mentioned this issue Jun 21, 2023

Unksip #160094

Merged

yctercero closed this as completed Jun 27, 2023

kibanamachine removed blocker skipped-test v8.9.0 labels Jun 27, 2023

kibanamachine reopened this Jun 30, 2023

WafaaNasr self-assigned this Jul 13, 2023

WafaaNasr mentioned this issue Jul 13, 2023

[Security Solution] [Tests] Determine true flakes / failures #161531

Closed

WafaaNasr closed this as completed Jul 13, 2023

elasticmachine unassigned WafaaNasr Oct 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kibanamachine commented Apr 19, 2023 •

edited

Loading

elasticmachine commented Apr 19, 2023

elasticmachine commented Apr 20, 2023

kibanamachine commented Apr 20, 2023

jbudz commented Apr 27, 2023

kibanamachine commented Apr 27, 2023

kibanamachine commented Apr 28, 2023

jbudz commented Apr 28, 2023

rylnd commented May 17, 2023

yctercero commented May 18, 2023

rylnd commented May 18, 2023

kibanamachine commented May 22, 2023

kibanamachine commented May 22, 2023

mistic commented May 22, 2023

yctercero commented Jun 27, 2023

kibanamachine commented Jun 30, 2023

MadameSheema commented Jul 3, 2023

yctercero commented Jul 3, 2023

nkhristinin commented Jul 5, 2023

kibanamachine commented Jul 5, 2023

WafaaNasr commented Jul 13, 2023 •

edited

Loading

WafaaNasr commented Jul 13, 2023

Comments

kibanamachine commented Apr 19, 2023 • edited Loading

elasticmachine commented Apr 19, 2023

elasticmachine commented Apr 20, 2023

kibanamachine commented Apr 20, 2023

jbudz commented Apr 27, 2023

kibanamachine commented Apr 27, 2023

kibanamachine commented Apr 28, 2023

jbudz commented Apr 28, 2023

rylnd commented May 17, 2023

yctercero commented May 18, 2023

rylnd commented May 18, 2023

kibanamachine commented May 22, 2023

kibanamachine commented May 22, 2023

mistic commented May 22, 2023

yctercero commented Jun 27, 2023

kibanamachine commented Jun 30, 2023

MadameSheema commented Jul 3, 2023

yctercero commented Jul 3, 2023

nkhristinin commented Jul 5, 2023

kibanamachine commented Jul 5, 2023

WafaaNasr commented Jul 13, 2023 • edited Loading

WafaaNasr commented Jul 13, 2023

kibanamachine commented Apr 19, 2023 •

edited

Loading

WafaaNasr commented Jul 13, 2023 •

edited

Loading