[7.7] Sets min_doc_count=0 for parent histogram of derivative aggregation (#61390) #61896

tylersmalley · 2020-03-30T21:28:34Z

Backports the following commits to 7.7:

Sets min_doc_count=0 for parent histogram of derivative aggregation (Sets min_doc_count=0 for parent histogram of derivative aggregation #61390)

…lastic#61390) From the docs: The specified metric must be numeric and the enclosing histogram must have min_doc_count set to 0 (default for histogram aggregations). https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-derivative-aggregation.html#search-aggregations-pipeline-derivative-aggregation This is now enforced, and currently failing the promotion of the current Elasticsearch build Related to elastic#61387 Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>

tylersmalley · 2020-03-30T23:41:14Z

@elasticmachine merge upstream

tylersmalley · 2020-03-31T16:13:42Z

Error caused by #60865

tylersmalley · 2020-03-31T16:13:49Z

@elasticmachine merge upstream

kibanamachine · 2020-03-31T18:08:01Z

💛 Build succeeded, but was flaky

continuous-integration/kibana-ci/pull-request
Commit: 2c37bdf
Flaky suites:
- xpack-kibana-ciGroup1

Test Failures

Kibana Pipeline / kibana-xpack-agent / X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/tests/create_rules·ts.detection engine api security and spaces enabled create_rules creating rules should create a single Machine Learning rule

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 3 times on tracked branches: https://github.com/elastic/kibana/issues/61995

[00:00:00]       │
[00:00:00]         └-: detection engine api security and spaces enabled
[00:00:00]           └-> "before all" hook
[00:00:09]           └-: create_rules
[00:00:09]             └-> "before all" hook
[00:00:09]             └-: creating rules
[00:00:09]               └-> "before all" hook
[00:00:09]               └-> should create a single rule with a rule_id
[00:00:09]                 └-> "before each" hook: global before each
[00:00:09]                 └-> "before each" hook
[00:00:09]                   │ info [o.e.x.i.a.TransportPutLifecycleAction] [kibana-ci-immutable-ubuntu-18-tests-xl-1585671272611645973] adding index lifecycle policy [.siem-signals-default]
[00:00:10]                   │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585671272611645973] adding template [.siem-signals-default] for index patterns [.siem-signals-default-*]
[00:00:10]                   │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585671272611645973] [.siem-signals-default-000001] creating index, cause [api], templates [.siem-signals-default], shards [1]/[1], mappings [_doc]
[00:00:10]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585671272611645973] moving index [.siem-signals-default-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [.siem-signals-default]
[00:00:10]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585671272611645973] moving index [.siem-signals-default-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [.siem-signals-default]
[00:00:10]                 │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585671272611645973] moving index [.siem-signals-default-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [.siem-signals-default]
[00:00:14]                 │ info [o.e.x.s.a.AuthenticationService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585671272611645973] Authentication using apikey failed - api key has been invalidated
[00:00:14]                 │ proc [kibana]   log   [17:23:58.723] [error][plugins][siem] An error occurred during rule execution:
[00:00:14]                 │ proc [kibana] message: "[security_exception] missing authentication credentials for REST request [/auditbeat-*/_search?allow_no_indices=true&size=100&ignore_unavailable=true], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } }"
[00:00:14]                 │ proc [kibana] name: "Simple Rule Query"
[00:00:14]                 │ proc [kibana] id: "c62d76f9-4966-4a43-99e2-1a8abd762bab"
[00:00:14]                 │ proc [kibana] rule id: "rule-1"
[00:00:14]                 │ proc [kibana] signals index: ".siem-signals-default"
[00:00:14]                 │ info [o.e.x.s.a.AuthenticationService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585671272611645973] Authentication using apikey failed - api key has been invalidated
[00:00:14]                 │ proc [kibana]   log   [17:23:58.728] [error][alerting][alerting][plugins][plugins] Executing Alert "c62d76f9-4966-4a43-99e2-1a8abd762bab" has resulted in Error: [security_exception] missing authentication credentials for REST request [/_security/user/_has_privileges], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } }
[00:00:14]                 └- ✓ pass  (4.5s) "detection engine api security and spaces enabled create_rules creating rules should create a single rule with a rule_id"
[00:00:14]               └-> "after each" hook
[00:00:14]                 │ info [o.e.c.m.MetaDataDeleteIndexService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585671272611645973] [.siem-signals-default-000001/3_gvEDuwS1azEYeaUE5kiw] deleting index
[00:00:14]                 │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585671272611645973] removing template [.siem-signals-default]
[00:00:15]               └-> should create a single rule without an input index
[00:00:15]                 └-> "before each" hook: global before each
[00:00:15]                 └-> "before each" hook
[00:00:15]                   │ info [o.e.x.i.a.TransportPutLifecycleAction] [kibana-ci-immutable-ubuntu-18-tests-xl-1585671272611645973] adding index lifecycle policy [.siem-signals-default]
[00:00:15]                   │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585671272611645973] adding template [.siem-signals-default] for index patterns [.siem-signals-default-*]
[00:00:15]                   │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585671272611645973] [.siem-signals-default-000001] creating index, cause [api], templates [.siem-signals-default], shards [1]/[1], mappings [_doc]
[00:00:15]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585671272611645973] moving index [.siem-signals-default-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [.siem-signals-default]
[00:00:15]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585671272611645973] moving index [.siem-signals-default-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [.siem-signals-default]
[00:00:15]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585671272611645973] moving index [.siem-signals-default-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [.siem-signals-default]
[00:00:19]                 └- ✓ pass  (4.4s) "detection engine api security and spaces enabled create_rules creating rules should create a single rule without an input index"
[00:00:19]               └-> "after each" hook
[00:00:19]                 │ info [o.e.c.m.MetaDataDeleteIndexService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585671272611645973] [.siem-signals-default-000001/wwmcso-0TXmiY_dM60WCNQ] deleting index
[00:00:19]                 │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585671272611645973] removing template [.siem-signals-default]
[00:00:20]               └-> should create a single rule without a rule_id
[00:00:20]                 └-> "before each" hook: global before each
[00:00:20]                 └-> "before each" hook
[00:00:20]                   │ info [o.e.x.s.a.AuthenticationService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585671272611645973] Authentication using apikey failed - api key has been invalidated
[00:00:20]                   │ proc [kibana]   log   [17:24:04.194] [error][plugins][siem] An error occurred during rule execution:
[00:00:20]                   │ proc [kibana] message: "[security_exception] missing authentication credentials for REST request [/_security/user/_has_privileges], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } }"
[00:00:20]                   │ proc [kibana] name: "Simple Rule Query"
[00:00:20]                   │ proc [kibana] id: "c936601a-b41e-4c5a-9ab5-409e23e3e5b4"
[00:00:20]                   │ proc [kibana] rule id: "rule-1"
[00:00:20]                   │ proc [kibana] signals index: ".siem-signals-default"
[00:00:20]                   │ info [o.e.x.s.a.AuthenticationService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585671272611645973] Authentication using apikey failed - api key has been invalidated
[00:00:20]                   │ proc [kibana]   log   [17:24:04.200] [error][alerting][alerting][plugins][plugins] Executing Alert "c936601a-b41e-4c5a-9ab5-409e23e3e5b4" has resulted in Error: [security_exception] missing authentication credentials for REST request [/_security/user/_has_privileges], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } }
[00:00:20]                   │ info [o.e.x.i.a.TransportPutLifecycleAction] [kibana-ci-immutable-ubuntu-18-tests-xl-1585671272611645973] adding index lifecycle policy [.siem-signals-default]
[00:00:20]                   │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585671272611645973] adding template [.siem-signals-default] for index patterns [.siem-signals-default-*]
[00:00:20]                   │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585671272611645973] [.siem-signals-default-000001] creating index, cause [api], templates [.siem-signals-default], shards [1]/[1], mappings [_doc]
[00:00:20]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585671272611645973] moving index [.siem-signals-default-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [.siem-signals-default]
[00:00:20]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585671272611645973] moving index [.siem-signals-default-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [.siem-signals-default]
[00:00:20]                 │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585671272611645973] moving index [.siem-signals-default-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [.siem-signals-default]
[00:00:24]                 └- ✓ pass  (4.5s) "detection engine api security and spaces enabled create_rules creating rules should create a single rule without a rule_id"
[00:00:24]               └-> "after each" hook
[00:00:24]                 │ info [o.e.c.m.MetaDataDeleteIndexService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585671272611645973] [.siem-signals-default-000001/UYqbzHRcR3SemhOOrNCT4g] deleting index
[00:00:24]                 │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585671272611645973] removing template [.siem-signals-default]
[00:00:25]               └-> should create a single Machine Learning rule
[00:00:25]                 └-> "before each" hook: global before each
[00:00:25]                 └-> "before each" hook
[00:00:25]                   │ info [o.e.x.i.a.TransportPutLifecycleAction] [kibana-ci-immutable-ubuntu-18-tests-xl-1585671272611645973] adding index lifecycle policy [.siem-signals-default]
[00:00:25]                   │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585671272611645973] adding template [.siem-signals-default] for index patterns [.siem-signals-default-*]
[00:00:25]                   │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-ubuntu-18-tests-xl-1585671272611645973] [.siem-signals-default-000001] creating index, cause [api], templates [.siem-signals-default], shards [1]/[1], mappings [_doc]
[00:00:25]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585671272611645973] moving index [.siem-signals-default-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [.siem-signals-default]
[00:00:25]                   │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585671272611645973] moving index [.siem-signals-default-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [.siem-signals-default]
[00:00:25]                 │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xl-1585671272611645973] moving index [.siem-signals-default-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [.siem-signals-default]
[00:00:29]                 │ proc [kibana]   log   [17:24:13.971] [warning][plugins][siem] Machine learning job is not started:
[00:00:29]                 │ proc [kibana] job id: "some_job_id"
[00:00:29]                 │ proc [kibana] job status: "undefined"
[00:00:29]                 │ proc [kibana] datafeed status: "undefined"
[00:00:29]                 │ proc [kibana] name: "Simple ML Rule"
[00:00:29]                 │ proc [kibana] id: "1cc359a8-490f-4bb9-af16-8865abd321ba"
[00:00:29]                 │ proc [kibana] rule id: "rule-1"
[00:00:29]                 │ proc [kibana] signals index: ".siem-signals-default"
[00:00:30]                 └- ✖ fail: "detection engine api security and spaces enabled create_rules creating rules should create a single Machine Learning rule"
[00:00:30]                 │

Stack Trace

{ Error: expected { created_by: 'elastic',
  description: 'Simple Machine Learning Rule',
  enabled: true,
  false_positives: [],
  from: 'now-6m',
  immutable: false,
  interval: '5m',
  rule_id: 'rule-1',
  output_index: '.siem-signals-default',
  max_signals: 100,
  risk_score: 1,
  name: 'Simple ML Rule',
  references: [],
  severity: 'high',
  updated_by: 'elastic',
  tags: [],
  to: 'now',
  type: 'machine_learning',
  threat: [],
  version: 1,
  lists: [],
  actions: [],
  throttle: 'no_actions',
  last_failure_at: '2020-03-31T17:24:14.020Z',
  last_failure_message: 'Machine learning job is not started:\njob id: "some_job_id"\njob status: "undefined"\ndatafeed status: "undefined"\nname: "Simple ML Rule"\nid: "1cc359a8-490f-4bb9-af16-8865abd321ba"\nrule id: "rule-1"\nsignals index: ".siem-signals-default"',
  anomaly_threshold: 44,
  machine_learning_job_id: 'some_job_id' } to sort of equal { actions: [],
  created_by: 'elastic',
  description: 'Simple Machine Learning Rule',
  enabled: true,
  false_positives: [],
  from: 'now-6m',
  immutable: false,
  interval: '5m',
  rule_id: 'rule-1',
  output_index: '.siem-signals-default',
  max_signals: 100,
  risk_score: 1,
  name: 'Simple ML Rule',
  references: [],
  severity: 'high',
  updated_by: 'elastic',
  tags: [],
  to: 'now',
  type: 'machine_learning',
  threat: [],
  throttle: 'no_actions',
  lists: [],
  version: 1,
  anomaly_threshold: 44,
  machine_learning_job_id: 'some_job_id' }
    at Assertion.assert (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:100:11)
    at Assertion.eql (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:244:8)
    at Context.it (test/detection_engine_api_integration/security_and_spaces/tests/create_rules.ts:101:34)
  actual:
   '{\n  "actions": []\n  "anomaly_threshold": 44\n  "created_by": "elastic"\n  "description": "Simple Machine Learning Rule"\n  "enabled": true\n  "false_positives": []\n  "from": "now-6m"\n  "immutable": false\n  "interval": "5m"\n  "last_failure_at": "2020-03-31T17:24:14.020Z"\n  "last_failure_message": "Machine learning job is not started:\\njob id: \\"some_job_id\\"\\njob status: \\"undefined\\"\\ndatafeed status: \\"undefined\\"\\nname: \\"Simple ML Rule\\"\\nid: \\"1cc359a8-490f-4bb9-af16-8865abd321ba\\"\\nrule id: \\"rule-1\\"\\nsignals index: \\".siem-signals-default\\""\n  "lists": []\n  "machine_learning_job_id": "some_job_id"\n  "max_signals": 100\n  "name": "Simple ML Rule"\n  "output_index": ".siem-signals-default"\n  "references": []\n  "risk_score": 1\n  "rule_id": "rule-1"\n  "severity": "high"\n  "tags": []\n  "threat": []\n  "throttle": "no_actions"\n  "to": "now"\n  "type": "machine_learning"\n  "updated_by": "elastic"\n  "version": 1\n}',
  expected:
   '{\n  "actions": []\n  "anomaly_threshold": 44\n  "created_by": "elastic"\n  "description": "Simple Machine Learning Rule"\n  "enabled": true\n  "false_positives": []\n  "from": "now-6m"\n  "immutable": false\n  "interval": "5m"\n  "lists": []\n  "machine_learning_job_id": "some_job_id"\n  "max_signals": 100\n  "name": "Simple ML Rule"\n  "output_index": ".siem-signals-default"\n  "references": []\n  "risk_score": 1\n  "rule_id": "rule-1"\n  "severity": "high"\n  "tags": []\n  "threat": []\n  "throttle": "no_actions"\n  "to": "now"\n  "type": "machine_learning"\n  "updated_by": "elastic"\n  "version": 1\n}',
  showDiff: true }

History

💔 Build #37356 failed b1945d3
💔 Build #37314 failed 0383e70

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

tylersmalley added the backport label Mar 30, 2020

Merge branch '7.7' into backport/7.7/pr-61390

b1945d3

Merge branch '7.7' into backport/7.7/pr-61390

2c37bdf

tylersmalley merged commit 4af5b7c into elastic:7.7 Mar 31, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[7.7] Sets min_doc_count=0 for parent histogram of derivative aggregation (#61390) #61896

[7.7] Sets min_doc_count=0 for parent histogram of derivative aggregation (#61390) #61896

tylersmalley commented Mar 30, 2020

tylersmalley commented Mar 30, 2020

tylersmalley commented Mar 31, 2020

tylersmalley commented Mar 31, 2020

kibanamachine commented Mar 31, 2020

Standard Out

Stack Trace

[7.7] Sets min_doc_count=0 for parent histogram of derivative aggregation (#61390) #61896

[7.7] Sets min_doc_count=0 for parent histogram of derivative aggregation (#61390) #61896

Conversation

tylersmalley commented Mar 30, 2020

tylersmalley commented Mar 30, 2020

tylersmalley commented Mar 31, 2020

tylersmalley commented Mar 31, 2020

kibanamachine commented Mar 31, 2020

💛 Build succeeded, but was flaky

Test Failures

Standard Out

Stack Trace

History