Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kibana 7.8.1. does not start. Stuck on migration notice while the cluster is newly created. #74225

Closed
kad-poelw opened this issue Aug 4, 2020 · 3 comments
Closed

Kibana 7.8.1. does not start. Stuck on migration notice while the cluster is newly created. #74225

kad-poelw opened this issue Aug 4, 2020 · 3 comments
Labels
Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc

Comments

@kad-poelw
Copy link

Kibana version:
7.8.1
Elasticsearch version:
7.8.1
Server OS version:
Azure Kubernetes Service 1.16.8 - Ubuntu 16.04
Browser version:
FF 79.0 64bit
Browser OS version:
Windows 10 2004 (19041.329)
Original install method (e.g. download page, yum, from source, etc.):
Kustomize deployment of ELK stack on Azure Kubernetes instance. All 7.8.1 images are default, except Elasticsearch which has the repository-azure installed as extra component.

Describe the bug:
When deploying an entire new and empty ES cluster via kustomize, we get the following behaviour: Kibana is started together with Logstash and Elasticsearch in one go. Kibana logs will show it is not able to contact the Elasticsearch cluster yet as it's still spinning up. After Kibana is able to contact the ES cluster, it creates both the .kibana_1 and the .kibana_task_manager_1 index. a few steps later it tells us Another Kibana instance appears to be migrating the index. Manual removal of the mentioned index files and deleting the pod (resulting in the replicasets.apps to restart a new one) makes the Kibana pod work as intended.

Steps to reproduce:

  1. Deploy new ELK stack based on 7.8.1.
  2. Monitor the logs
  3. Notice the issue about Another Kibana instance appears to be migrating the index

Expected behavior:
Kibana will start in one go without human intervention.

Screenshots (if relevant):
N/A

Errors in browser console (if relevant):
N/A

Provide logs and/or server output (if relevant):

{"type":"log","@timestamp":"2020-08-04T08:23:26Z","tags":["warning","plugins-discovery"],"pid":6,"message":"Expect plugin \"id\" in camelCase, but found: apm_oss"}
{"type":"log","@timestamp":"2020-08-04T08:23:26Z","tags":["warning","plugins-discovery"],"pid":6,"message":"Expect plugin \"id\" in camelCase, but found: triggers_actions_ui"}
{"type":"log","@timestamp":"2020-08-04T08:24:30Z","tags":["info","plugins-service"],"pid":6,"message":"Plugin \"visTypeXy\" is disabled."}
{"type":"log","@timestamp":"2020-08-04T08:24:30Z","tags":["info","plugins-service"],"pid":6,"message":"Plugin \"endpoint\" is disabled."}
{"type":"log","@timestamp":"2020-08-04T08:24:30Z","tags":["info","plugins-service"],"pid":6,"message":"Plugin \"ingestManager\" is disabled."}
{"type":"log","@timestamp":"2020-08-04T08:24:30Z","tags":["info","plugins-service"],"pid":6,"message":"Plugin \"lists\" is disabled."}
{"type":"log","@timestamp":"2020-08-04T08:24:36Z","tags":["warning","config","deprecation"],"pid":6,"message":"\"xpack.monitoring.ui.container.elasticsearch.enabled\" is deprecated and has been replaced by \"monitoring.ui.container.elasticsearch.enabled\""}
{"type":"log","@timestamp":"2020-08-04T08:24:36Z","tags":["warning","config","deprecation"],"pid":6,"message":"\"xpack.monitoring\" is deprecated and has been replaced by \"monitoring\". However both key are present, ignoring \"xpack.monitoring\""}
{"type":"log","@timestamp":"2020-08-04T08:24:36Z","tags":["warning","config","deprecation"],"pid":6,"message":"Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0.\""}
{"type":"log","@timestamp":"2020-08-04T08:24:36Z","tags":["info","plugins-system"],"pid":6,"message":"Setting up [94] plugins: [taskManager,licensing,observability,eventLog,encryptedSavedObjects,code,usageCollection,ossTelemetry,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaLegacy,devTools,translations,uiActions,statusPage,share,newsfeed,mapsLegacy,mapsLegacyLicensing,kibanaReact,kibanaUtils,inspector,embeddable,advancedUiActions,embeddableEnhanced,drilldowns,indexPatternManagement,esUiShared,discover,charts,bfetch,expressions,data,home,cloud,console,consoleExtensions,apm_oss,searchprofiler,painlessLab,grokdebugger,management,upgradeAssistant,reporting,licenseManagement,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,watcher,advancedSettings,telemetryManagementSection,fileUpload,dataEnhanced,visualizations,visTypeVislib,visTypeVega,visTypeTimeseries,rollup,visTypeTimelion,features,security,snapshotRestore,transform,ingestPipelines,canvas,visTypeTagcloud,visTypeTable,visTypeMarkdown,visTypeMetric,inputControlVis,savedObjects,navigation,lens,graph,maps,visualize,dashboard,dashboardEnhanced,savedObjectsManagement,spaces,actions,case,alerting,alertingBuiltins,triggers_actions_ui,infra,monitoring,logstash,uptime,ml,siem,apm]"}
{"type":"log","@timestamp":"2020-08-04T08:24:36Z","tags":["warning","plugins","security","config"],"pid":6,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."}
{"type":"log","@timestamp":"2020-08-04T08:24:37Z","tags":["info","plugins","monitoring","monitoring"],"pid":6,"message":"config sourced from: production cluster"}
{"type":"log","@timestamp":"2020-08-04T08:24:37Z","tags":["warning","plugins","monitoring","monitoring"],"pid":6,"message":"X-Pack Monitoring Cluster Alerts will not be available: undefined"}
{"type":"log","@timestamp":"2020-08-04T08:24:37Z","tags":["error","elasticsearch","monitoring"],"pid":6,"message":"Request error, retrying\nGET http://elasticsearch:9200/_xpack => connect ECONNREFUSED 192.168.254.151:9200"}
{"type":"log","@timestamp":"2020-08-04T08:24:37Z","tags":["error","elasticsearch","data"],"pid":6,"message":"Request error, retrying\nGET http://elasticsearch:9200/_xpack => connect ECONNREFUSED 192.168.254.151:9200"}
{"type":"log","@timestamp":"2020-08-04T08:24:37Z","tags":["error","elasticsearch","monitoring"],"pid":6,"message":"Request error, retrying\nGET http://elasticsearch:9200/_xpack => connect ECONNREFUSED 192.168.254.151:9200"}
{"type":"log","@timestamp":"2020-08-04T08:24:37Z","tags":["warning","elasticsearch","data"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:24:37Z","tags":["warning","elasticsearch","data"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:24:37Z","tags":["warning","plugins","licensing"],"pid":6,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}
{"type":"log","@timestamp":"2020-08-04T08:24:37Z","tags":["info","plugins","searchprofiler"],"pid":6,"message":"You cannot use searchprofiler because license information is not available at this time."}
{"type":"log","@timestamp":"2020-08-04T08:24:37Z","tags":["info","plugins","painlessLab"],"pid":6,"message":"You cannot use painlessLab because license information is not available at this time."}
{"type":"log","@timestamp":"2020-08-04T08:24:37Z","tags":["info","plugins","indexManagement"],"pid":6,"message":"You cannot use index_management because license information is not available at this time."}
{"type":"log","@timestamp":"2020-08-04T08:24:37Z","tags":["info","plugins","remoteClusters"],"pid":6,"message":"You cannot use Remote Clusters because license information is not available at this time."}
{"type":"log","@timestamp":"2020-08-04T08:24:37Z","tags":["info","plugins","crossClusterReplication"],"pid":6,"message":"You cannot use crossClusterReplication because license information is not available at this time."}
{"type":"log","@timestamp":"2020-08-04T08:24:37Z","tags":["info","plugins","indexLifecycleManagement"],"pid":6,"message":"You cannot use index_lifecycle_management because license information is not available at this time."}
{"type":"log","@timestamp":"2020-08-04T08:24:37Z","tags":["info","plugins","watcher"],"pid":6,"message":"You cannot use watcher because license information is not available at this time."}
{"type":"log","@timestamp":"2020-08-04T08:24:37Z","tags":["info","plugins","rollup"],"pid":6,"message":"You cannot use rollup because license information is not available at this time."}
{"type":"log","@timestamp":"2020-08-04T08:24:37Z","tags":["info","plugins","snapshotRestore"],"pid":6,"message":"You cannot use snapshot_restore because license information is not available at this time."}
{"type":"log","@timestamp":"2020-08-04T08:24:37Z","tags":["info","plugins","transform"],"pid":6,"message":"You cannot use transform because license information is not available at this time."}
{"type":"log","@timestamp":"2020-08-04T08:24:37Z","tags":["info","plugins","ingestPipelines"],"pid":6,"message":"You cannot use ingest_pipelines because license information is not available at this time."}
{"type":"log","@timestamp":"2020-08-04T08:24:37Z","tags":["warning","elasticsearch","monitoring"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:24:37Z","tags":["warning","elasticsearch","monitoring"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:24:37Z","tags":["warning","plugins","licensing"],"pid":6,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}
{"type":"log","@timestamp":"2020-08-04T08:24:37Z","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":6,"message":"Monitoring status upload endpoint is not enabled in Elasticsearch:Monitoring stats collection is stopped"}
{"type":"log","@timestamp":"2020-08-04T08:24:37Z","tags":["info","savedobjects-service"],"pid":6,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2020-08-04T08:24:37Z","tags":["warning","elasticsearch","monitoring"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:24:37Z","tags":["warning","elasticsearch","monitoring"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:24:37Z","tags":["warning","plugins","licensing"],"pid":6,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}
{"type":"log","@timestamp":"2020-08-04T08:24:37Z","tags":["error","elasticsearch","admin"],"pid":6,"message":"Request error, retrying\nGET http://elasticsearch:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => connect ECONNREFUSED 192.168.254.151:9200"}
{"type":"log","@timestamp":"2020-08-04T08:24:37Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:24:37Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:24:37Z","tags":["error","savedobjects-service"],"pid":6,"message":"Unable to retrieve version information from Elasticsearch nodes."}
{"type":"log","@timestamp":"2020-08-04T08:24:39Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:24:39Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:24:42Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:24:42Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:24:44Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:24:44Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:24:47Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:24:47Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:24:49Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:24:49Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:24:52Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:24:52Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:24:54Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:24:55Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:24:57Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:24:57Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:25:00Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:25:00Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:25:02Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:25:02Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:25:05Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:25:05Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:25:07Z","tags":["warning","elasticsearch","monitoring"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:25:07Z","tags":["warning","elasticsearch","monitoring"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:25:07Z","tags":["warning","plugins","licensing"],"pid":6,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}
{"type":"log","@timestamp":"2020-08-04T08:25:07Z","tags":["warning","elasticsearch","data"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:25:07Z","tags":["warning","elasticsearch","data"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:25:07Z","tags":["warning","plugins","licensing"],"pid":6,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}
{"type":"log","@timestamp":"2020-08-04T08:25:07Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:25:07Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:25:10Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:25:10Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:25:12Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:25:12Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:25:15Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:25:15Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:25:17Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:25:17Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:25:20Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:25:20Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:25:22Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:25:22Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:25:25Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:25:25Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:25:27Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:25:27Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:25:30Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:25:30Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:25:32Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:25:32Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:25:35Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:25:35Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:25:37Z","tags":["warning","elasticsearch","monitoring"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:25:37Z","tags":["warning","elasticsearch","monitoring"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:25:37Z","tags":["warning","plugins","licensing"],"pid":6,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}
{"type":"log","@timestamp":"2020-08-04T08:25:37Z","tags":["warning","elasticsearch","data"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:25:37Z","tags":["warning","elasticsearch","data"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:25:37Z","tags":["warning","plugins","licensing"],"pid":6,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}
{"type":"log","@timestamp":"2020-08-04T08:25:37Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:25:37Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:25:40Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:25:40Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:25:42Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:25:42Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:25:45Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:25:45Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:25:47Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:25:47Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:25:50Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:25:50Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:25:52Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:25:52Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:25:55Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:25:55Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:25:57Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:25:57Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:26:00Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:26:00Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:26:02Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:26:02Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:26:05Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:26:05Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:26:07Z","tags":["warning","elasticsearch","monitoring"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:26:07Z","tags":["warning","elasticsearch","monitoring"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:26:07Z","tags":["warning","plugins","licensing"],"pid":6,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}
{"type":"log","@timestamp":"2020-08-04T08:26:07Z","tags":["warning","elasticsearch","data"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:26:07Z","tags":["warning","elasticsearch","data"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:26:07Z","tags":["warning","plugins","licensing"],"pid":6,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}
{"type":"log","@timestamp":"2020-08-04T08:26:07Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:26:07Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:26:10Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:26:10Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:26:12Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:26:12Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:26:15Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:26:15Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:26:17Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:26:17Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:26:20Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:26:20Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:26:22Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:26:22Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:26:25Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:26:25Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:26:27Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:26:27Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:26:30Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:26:30Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:26:32Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:26:32Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:26:35Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:26:35Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:26:37Z","tags":["warning","elasticsearch","monitoring"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:26:37Z","tags":["warning","elasticsearch","monitoring"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:26:37Z","tags":["warning","plugins","licensing"],"pid":6,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}
{"type":"log","@timestamp":"2020-08-04T08:26:37Z","tags":["warning","elasticsearch","data"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:26:37Z","tags":["warning","elasticsearch","data"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:26:37Z","tags":["warning","plugins","licensing"],"pid":6,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}
{"type":"log","@timestamp":"2020-08-04T08:26:37Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:26:37Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:26:40Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:26:40Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:26:42Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:26:42Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:26:45Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:26:45Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:26:47Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:26:47Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:26:50Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:26:50Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:26:52Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:26:52Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:26:55Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:26:55Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:26:57Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:26:57Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:27:00Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:27:00Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:27:02Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:27:02Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:27:05Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:27:05Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:27:07Z","tags":["warning","elasticsearch","monitoring"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:27:07Z","tags":["warning","elasticsearch","monitoring"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:27:07Z","tags":["warning","plugins","licensing"],"pid":6,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}
{"type":"log","@timestamp":"2020-08-04T08:27:07Z","tags":["warning","elasticsearch","data"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:27:07Z","tags":["warning","elasticsearch","data"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:27:07Z","tags":["warning","plugins","licensing"],"pid":6,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}
{"type":"log","@timestamp":"2020-08-04T08:27:07Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:27:07Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:27:10Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:27:10Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:27:12Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:27:12Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:27:15Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:27:15Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:27:17Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:27:17Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:27:20Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:27:20Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:27:23Z","tags":["info","savedobjects-service"],"pid":6,"message":"Starting saved objects migrations"}
{"type":"log","@timestamp":"2020-08-04T08:27:25Z","tags":["error","elasticsearch","admin"],"pid":6,"message":"Request error, retrying\nGET http://elasticsearch:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => connect ECONNREFUSED 192.168.254.151:9200"}
{"type":"log","@timestamp":"2020-08-04T08:27:25Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:27:25Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:27:25Z","tags":["error","savedobjects-service"],"pid":6,"message":"Unable to retrieve version information from Elasticsearch nodes."}
{"type":"log","@timestamp":"2020-08-04T08:27:27Z","tags":["error","elasticsearch","admin"],"pid":6,"message":"Request error, retrying\nGET http://elasticsearch:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => connect ECONNREFUSED 192.168.254.151:9200"}
{"type":"log","@timestamp":"2020-08-04T08:27:27Z","tags":["error","elasticsearch","admin"],"pid":6,"message":"Request error, retrying\nGET http://elasticsearch:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => connect ECONNREFUSED 192.168.254.151:9200"}
{"type":"log","@timestamp":"2020-08-04T08:27:37Z","tags":["warning","elasticsearch","data"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2020-08-04T08:27:37Z","tags":["warning","elasticsearch","data"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-08-04T08:27:37Z","tags":["warning","plugins","licensing"],"pid":6,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}
{"type":"log","@timestamp":"2020-08-04T08:27:53Z","tags":["warning","savedobjects-service"],"pid":6,"message":"Unable to connect to Elasticsearch. Error: [master_not_discovered_exception] null"}
{"type":"log","@timestamp":"2020-08-04T08:27:53Z","tags":["warning","savedobjects-service"],"pid":6,"message":"Unable to connect to Elasticsearch. Error: Request Timeout after 30000ms"}
{"type":"log","@timestamp":"2020-08-04T08:28:04Z","tags":["info","savedobjects-service"],"pid":6,"message":"Creating index .kibana_task_manager_1."}
{"type":"log","@timestamp":"2020-08-04T08:28:04Z","tags":["info","savedobjects-service"],"pid":6,"message":"Creating index .kibana_1."}
{"type":"log","@timestamp":"2020-08-04T08:28:07Z","tags":["info","plugins","searchprofiler"],"pid":6,"message":"You cannot use searchprofiler because license information is not available at this time."}
{"type":"log","@timestamp":"2020-08-04T08:28:07Z","tags":["info","plugins","painlessLab"],"pid":6,"message":"You cannot use painlessLab because license information is not available at this time."}
{"type":"log","@timestamp":"2020-08-04T08:28:07Z","tags":["info","plugins","indexManagement"],"pid":6,"message":"You cannot use index_management because license information is not available at this time."}
{"type":"log","@timestamp":"2020-08-04T08:28:07Z","tags":["info","plugins","remoteClusters"],"pid":6,"message":"You cannot use Remote Clusters because license information is not available at this time."}
{"type":"log","@timestamp":"2020-08-04T08:28:07Z","tags":["info","plugins","crossClusterReplication"],"pid":6,"message":"You cannot use crossClusterReplication because license information is not available at this time."}
{"type":"log","@timestamp":"2020-08-04T08:28:07Z","tags":["info","plugins","indexLifecycleManagement"],"pid":6,"message":"You cannot use index_lifecycle_management because license information is not available at this time."}
{"type":"log","@timestamp":"2020-08-04T08:28:07Z","tags":["info","plugins","watcher"],"pid":6,"message":"You cannot use watcher because license information is not available at this time."}
{"type":"log","@timestamp":"2020-08-04T08:28:07Z","tags":["info","plugins","rollup"],"pid":6,"message":"You cannot use rollup because license information is not available at this time."}
{"type":"log","@timestamp":"2020-08-04T08:28:07Z","tags":["info","plugins","snapshotRestore"],"pid":6,"message":"You cannot use snapshot_restore because license information is not available at this time."}
{"type":"log","@timestamp":"2020-08-04T08:28:07Z","tags":["info","plugins","transform"],"pid":6,"message":"You cannot use transform because license information is not available at this time."}
{"type":"log","@timestamp":"2020-08-04T08:28:07Z","tags":["info","plugins","ingestPipelines"],"pid":6,"message":"You cannot use ingest_pipelines because license information is not available at this time."}
{"type":"log","@timestamp":"2020-08-04T08:28:07Z","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":6,"message":"Monitoring status upload endpoint is not enabled in Elasticsearch:Monitoring stats collection is stopped"}
{"type":"log","@timestamp":"2020-08-04T08:28:38Z","tags":["warning","savedobjects-service"],"pid":6,"message":"Unable to connect to Elasticsearch. Error: [resource_already_exists_exception] index [.kibana_task_manager_1/kV8p4G3bQwqSYDbixAAeow] already exists, with { index_uuid=\"kV8p4G3bQwqSYDbixAAeow\" & index=\".kibana_task_manager_1\" }"}
{"type":"log","@timestamp":"2020-08-04T08:28:38Z","tags":["warning","savedobjects-service"],"pid":6,"message":"Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_task_manager_1 and restarting Kibana."}
{"type":"log","@timestamp":"2020-08-04T08:28:38Z","tags":["warning","savedobjects-service"],"pid":6,"message":"Unable to connect to Elasticsearch. Error: [resource_already_exists_exception] index [.kibana_1/ucCET2NGT22qbvgFarfJMw] already exists, with { index_uuid=\"ucCET2NGT22qbvgFarfJMw\" & index=\".kibana_1\" }"}
{"type":"log","@timestamp":"2020-08-04T08:28:38Z","tags":["warning","savedobjects-service"],"pid":6,"message":"Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_1 and restarting Kibana."}
{"type":"log","@timestamp":"2020-08-04T08:29:07Z","tags":["info","plugins","crossClusterReplication"],"pid":6,"message":"Your basic license does not support crossClusterReplication. Please upgrade your license."}
{"type":"log","@timestamp":"2020-08-04T08:29:07Z","tags":["info","plugins","watcher"],"pid":6,"message":"Your basic license does not support watcher. Please upgrade your license."}
{"type":"log","@timestamp":"2020-08-04T08:29:07Z","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":6,"message":"Starting monitoring stats collection"}

Any additional context:
@kad-poelw
Copy link
Author

kad-poelw commented Aug 12, 2020 •
edited
Loading

Is it vacation time or is the kibana's issue list not tracked by dev?

@flash1293 flash1293 added the Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc label Aug 20, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-platform (Team:Platform)

@rudolf
Copy link
Contributor

rudolf commented Oct 20, 2020

Error: Request Timeout after 30000ms

Kibana creates a migration "lock" by creating a new index. When that request reaches a timeout, but Elasticsearch successfully created the index Kibana no longer knows whether it has acquired the lock or if it was another Kibana instance. As you mentioned this requires manually deleting the indices to resolve.

This is a flaw in our migration algorithm. We've designed a new algorithm in #66056 and are tracking the work to implement it in #75780

@rudolf rudolf closed this as completed Oct 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@rudolf @flash1293 @elasticmachine @kad-poelw