[test-failed]: Chrome X-Pack UI Functional Tests1.x-pack/test/functional/apps/reporting_management/report_delete_pagination·ts - reporting management app Delete reports "before all" hook for "Confirm single report deletion works"

[liza-mae]

Version: 7.8.0
Class: Chrome X-Pack UI Functional Tests1.x-pack/test/functional/apps/reporting_management/report_delete_pagination·ts
Stack Trace:
{ TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="kibanaChrome"])
Wait timed out after 102212ms
at /var/lib/jenkins/workspace/elastic+estf-cloud-kibana-tests/JOB/xpackGrp1/TASK/saas_run_kibana_tests/node/linux-immutable/ci/cloud/common/build/kibana/node_modules/selenium-webdriver/lib/webdriver.js:842:17
at process._tickCallback (internal/process/next_tick.js:68:7) name: 'TimeoutError', remoteStacktrace: '' }
Platform: cloud
Build Num: 324

[elasticmachine]

Pinging @elastic/kibana-test-triage (failed-test)

[elasticmachine]

Pinging @elastic/kibana-reporting-services (Team:Reporting Services)

[tsullivan]

Hi, I ran this test in the flaky test runner for Cloud SaaS, and found the following server logs. It looks like Elasticsearch becomes unresponsive. It's hard to tell how far into the test this succeeds, but it looks like not very far.

"@timestamp",message
"June 30th 2020, 20:57:06.000","Stopping all plugins."
"June 30th 2020, 20:57:05.000","Failed to poll for work: Service Unavailable :: {""path"":""/.kibana_task_manager/_update_by_query"",""query"":{""ignore_unavailable"":true,""refresh"":true,""max_docs"":10,""conflicts"":""proceed""},""body"":""{\""query\"":{\""bool\"":{\""must\"":[{\""term\"":{\""type\"":\""task\""}},{\""bool\"":{\""must\"":[{\""bool\"":{\""must\"":[{\""bool\"":{\""should\"":[{\""bool\"":{\""must\"":[{\""term\"":{\""task.status\"":\""idle\""}},{\""range\"":{\""task.runAt\"":{\""lte\"":\""now\""}}}]}},{\""bool\"":{\""must\"":[{\""bool\"":{\""should\"":[{\""term\"":{\""task.status\"":\""running\""}},{\""term\"":{\""task.status\"":\""claiming\""}}]}},{\""range\"":{\""task.retryAt\"":{\""lte\"":\""now\""}}}]}}]}},{\""bool\"":{\""should\"":[{\""exists\"":{\""field\"":\""task.schedule\""}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""vis_telemetry\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":3}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""lens_telemetry\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":3}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""actions:.email\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":1}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""actions:.index\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":1}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""actions:.pagerduty\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":1}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""actions:.server-log\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":1}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""actions:.slack\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":1}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""actions:.webhook\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":1}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""actions:.servicenow\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":1}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""actions:.jira\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":1}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""alerting:.index-threshold\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":3}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""alerting:siem.signals\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":3}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""alerting:siem.notifications\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":3}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""alerting:metrics.alert.threshold\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":3}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""alerting:metrics.alert.inventory.threshold\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":3}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""alerting:logs.alert.document.count\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":3}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""alerting:apm.transaction_duration\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":3}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""alerting:apm.error_rate\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":3}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""alerting:xpack.uptime.alerts.monitorStatus\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":3}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""alerting:xpack.uptime.alerts.tls\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":3}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""actions_telemetry\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":3}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""alerting_telemetry\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":3}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""apm-telemetry-task\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":3}}}]}}]}}]}}],\""filter\"":[{\""bool\"":{\""must_not\"":[{\""bool\"":{\""should\"":[{\""term\"":{\""task.status\"":\""running\""}},{\""term\"":{\""task.status\"":\""claiming\""}}],\""must\"":{\""range\"":{\""task.retryAt\"":{\""gt\"":\""now\""}}}}}]}}]}}]}},\""sort\"":[\""_score\"",{\""_script\"":{\""type\"":\""number\"",\""order\"":\""asc\"",\""script\"":{\""lang\"":\""painless\"",\""source\"":\""\\nif (doc['task.retryAt'].size()!=0) {\\n  return doc['task.retryAt'].value.toInstant().toEpochMilli();\\n}\\nif (doc['task.runAt'].size()!=0) {\\n  return doc['task.runAt'].value.toInstant().toEpochMilli();\\n}\\n    \""}}}],\""seq_no_primary_term\"":true,\""script\"":{\""source\"":\""ctx._source.task.ownerId=params.ownerId; ctx._source.task.status=params.status; ctx._source.task.retryAt=params.retryAt;\"",\""lang\"":\""painless\"",\""params\"":{\""ownerId\"":\""kibana:5b2de169-2785-441b-ae8c-186a1936b17d\"",\""status\"":\""claiming\"",\""retryAt\"":\""2020-06-30T20:57:35.220Z\""}}}"",""statusCode"":503,""response"":""{\""ok\"":false,\""message\"":\""Resource is under maintenance / recovery.\""}\n""}"
"June 30th 2020, 20:57:04.000","License information could not be obtained from Elasticsearch due to Service Unavailable :: {""path"":""/_xpack"",""statusCode"":503,""response"":""{\""ok\"":false,\""message\"":\""Resource is under maintenance / recovery.\""}\n""} error"
"June 30th 2020, 20:57:04.000","GET / 503 20ms - 9.0B"
"June 30th 2020, 20:57:04.000","License information could not be obtained from Elasticsearch due to Service Unavailable :: {""path"":""/_xpack"",""statusCode"":503,""response"":""{\""ok\"":false,\""message\"":\""Resource is under maintenance / recovery.\""}\n""} error"
"June 30th 2020, 20:57:03.000","You cannot use crossClusterReplication because license information is not available at this time."
"June 30th 2020, 20:57:03.000","Status changed from red to red - Service Unavailable"
"June 30th 2020, 20:57:03.000","Status changed from green to red - Unable to retrieve version information from Elasticsearch nodes."
"June 30th 2020, 20:57:03.000","You cannot use Remote Clusters because license information is not available at this time."
"June 30th 2020, 20:57:03.000","Monitoring status upload endpoint is not enabled in Elasticsearch:Monitoring stats collection is stopped"
"June 30th 2020, 20:57:03.000","Status changed from red to red - Service Unavailable"
"June 30th 2020, 20:57:03.000","Unable to retrieve version information from Elasticsearch nodes."
"June 30th 2020, 20:57:03.000","Status changed from green to red - Unable to retrieve version information from Elasticsearch nodes."
"June 30th 2020, 20:57:03.000","You cannot use index_management because license information is not available at this time."
"June 30th 2020, 20:57:03.000","You cannot use snapshot_restore because license information is not available at this time."
"June 30th 2020, 20:57:03.000","Status changed from red to red - Service Unavailable"
"June 30th 2020, 20:57:03.000","kc2e7s2o000w334cc054kt42 - job querying failed: Error: Service Unavailable
    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)
    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)
    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)
    at IncomingMessage.emit (events.js:203:15)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)"
"June 30th 2020, 20:57:03.000","GET /api/status 200 82ms - 9.0B"
"June 30th 2020, 20:57:03.000","Status changed from green to red - Unable to retrieve version information from Elasticsearch nodes."
"June 30th 2020, 20:57:03.000","Status changed from green to red - Unable to retrieve version information from Elasticsearch nodes."
"June 30th 2020, 20:57:03.000","Status changed from green to red - Unable to retrieve version information from Elasticsearch nodes."
"June 30th 2020, 20:57:03.000","Status changed from green to red - Unable to retrieve version information from Elasticsearch nodes."
"June 30th 2020, 20:57:03.000","You cannot use transform because license information is not available at this time."
"June 30th 2020, 20:57:03.000","You cannot use rollup because license information is not available at this time."
"June 30th 2020, 20:57:03.000","You cannot use watcher because license information is not available at this time."
"June 30th 2020, 20:57:03.000","Status changed from red to red - Service Unavailable"
"June 30th 2020, 20:57:03.000","License information could not be obtained from Elasticsearch due to Service Unavailable :: {""path"":""/_xpack"",""statusCode"":503,""response"":""{\""ok\"":false,\""message\"":\""Resource is under maintenance / recovery.\""}\n""} error"
"June 30th 2020, 20:57:03.000","Status changed from red to red - Service Unavailable"
"June 30th 2020, 20:57:03.000","You cannot use searchprofiler because license information is not available at this time."
"June 30th 2020, 20:57:03.000","You cannot use painlessLab because license information is not available at this time."
"June 30th 2020, 20:57:03.000","You cannot use index_lifecycle_management because license information is not available at this time."
"June 30th 2020, 20:57:03.000","You cannot use ingest_pipelines because license information is not available at this time."
"June 30th 2020, 20:57:02.000","Failed to poll for work: Service Unavailable :: {""path"":""/.kibana_task_manager/_update_by_query"",""query"":{""ignore_unavailable"":true,""refresh"":true,""max_docs"":10,""conflicts"":""proceed""},""body"":""{\""query\"":{\""bool\"":{\""must\"":[{\""term\"":{\""type\"":\""task\""}},{\""bool\"":{\""must\"":[{\""bool\"":{\""must\"":[{\""bool\"":{\""should\"":[{\""bool\"":{\""must\"":[{\""term\"":{\""task.status\"":\""idle\""}},{\""range\"":{\""task.runAt\"":{\""lte\"":\""now\""}}}]}},{\""bool\"":{\""must\"":[{\""bool\"":{\""should\"":[{\""term\"":{\""task.status\"":\""running\""}},{\""term\"":{\""task.status\"":\""claiming\""}}]}},{\""range\"":{\""task.retryAt\"":{\""lte\"":\""now\""}}}]}}]}},{\""bool\"":{\""should\"":[{\""exists\"":{\""field\"":\""task.schedule\""}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""vis_telemetry\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":3}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""lens_telemetry\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":3}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""actions:.email\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":1}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""actions:.index\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":1}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""actions:.pagerduty\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":1}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""actions:.server-log\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":1}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""actions:.slack\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":1}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""actions:.webhook\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":1}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""actions:.servicenow\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":1}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""actions:.jira\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":1}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""alerting:.index-threshold\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":3}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""alerting:siem.signals\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":3}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""alerting:siem.notifications\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":3}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""alerting:metrics.alert.threshold\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":3}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""alerting:metrics.alert.inventory.threshold\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":3}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""alerting:logs.alert.document.count\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":3}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""alerting:apm.transaction_duration\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":3}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""alerting:apm.error_rate\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":3}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""alerting:xpack.uptime.alerts.monitorStatus\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":3}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""alerting:xpack.uptime.alerts.tls\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":3}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""actions_telemetry\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":3}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""alerting_telemetry\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":3}}}]}},{\""bool\"":{\""must\"":[{\""term\"":{\""task.taskType\"":\""apm-telemetry-task\""}},{\""range\"":{\""task.attempts\"":{\""lt\"":3}}}]}}]}}]}}],\""filter\"":[{\""bool\"":{\""must_not\"":[{\""bool\"":{\""should\"":[{\""term\"":{\""task.status\"":\""running\""}},{\""term\"":{\""task.status\"":\""claiming\""}}],\""must\"":{\""range\"":{\""task.retryAt\"":{\""gt\"":\""now\""}}}}}]}}]}}]}},\""sort\"":[\""_score\"",{\""_script\"":{\""type\"":\""number\"",\""order\"":\""asc\"",\""script\"":{\""lang\"":\""painless\"",\""source\"":\""\\nif (doc['task.retryAt'].size()!=0) {\\n  return doc['task.retryAt'].value.toInstant().toEpochMilli();\\n}\\nif (doc['task.runAt'].size()!=0) {\\n  return doc['task.runAt'].value.toInstant().toEpochMilli();\\n}\\n    \""}}}],\""seq_no_primary_term\"":true,\""script\"":{\""source\"":\""ctx._source.task.ownerId=params.ownerId; ctx._source.task.status=params.status; ctx._source.task.retryAt=params.retryAt;\"",\""lang\"":\""painless\"",\""params\"":{\""ownerId\"":\""kibana:5b2de169-2785-441b-ae8c-186a1936b17d\"",\""status\"":\""claiming\"",\""retryAt\"":\""2020-06-30T20:57:32.219Z\""}}}"",""statusCode"":503,""response"":""{\""ok\"":false,\""message\"":\""Resource is under maintenance / recovery.\""}\n""}"

[liza-mae]

@tsullivan I believe the error you are seeing is different than the one I reported in this issue.

{ Error: Request Timeout after 30000ms
at /var/lib/jenkins/workspace/elastic+estf-cloud-kibana-flaky-test-runner/JOB/flakyRun1/TASK/saas_run_kibana_tests/node/linux-immutable/ci/cloud/common/build/kibana/node_modules/elasticsearch/src/lib/transport.js:397:9
at Timeout. (/var/lib/jenkins/workspace/elastic+estf-cloud-kibana-flaky-test-runner/JOB/flakyRun1/TASK/saas_run_kibana_tests/node/linux-immutable/ci/cloud/common/build/kibana/node_modules/elasticsearch/src/lib/transport.js:429:7)
status: undefined,
displayName: 'RequestTimeout',
message: 'Request Timeout after 30000ms',
body: undefined }

[liza-mae]

The problem reported in this issue appears to be caused by the test order execution and is not specific to cloud, we should be able to reproduce via the test order. Attached is the screenshot.

@tsullivan the other error you reported still needs to be investigated, but it is a separate issue.

[tsullivan]

The problem reported in this issue appears to be caused by the test order execution and is not specific to cloud, we should be able to reproduce via the test order. Attached is the screenshot.

I see. In my testing, I referred to a single test suite that only includes this test and 1 other one. That explains why I didn't hit the problem of test order of execution.

[rashmivkulkarni]

I ran this test in my local and on cloud 7.9 snapshot , runs fine in a loop. It must be the test order of execution as Tim rightly points out .

[liza-mae]

We would need to change the order in the functional/config.js if we feel transform test needs to be run last.

kibana/x-pack/test/functional/config.js

Line 58 in 0e008e3

resolve(__dirname, './apps/transform'),

kibana/x-pack/test/functional/config.js

Line 59 in 0e008e3

resolve(__dirname, './apps/reporting_management'),

[LeeDr]

I figured out the problem. The transform tests just need to logout() in the after method in the index file.

I have the fix implemented locally and I'll create a PR for it.

[liza-mae]

Fix was not back ported to 7.8, can it be back ported to that branch?

[LeeDr]

on it's way #70973