Skip to content

Latest commit

 

History

History
150 lines (116 loc) · 6.93 KB

cs_troubleshoot_health.md

File metadata and controls

150 lines (116 loc) · 6.93 KB
Raw
copyright lastupdated
years
2014, 2018
2018-04-10

{:new_window: target="_blank"} {:shortdesc: .shortdesc} {:screen: .screen} {:pre: .pre} {:table: .aria-labeledby="caption"} {:codeblock: .codeblock} {:tip: .tip} {:download: .download} {:tsSymptoms: .tsSymptoms} {:tsCauses: .tsCauses} {:tsResolve: .tsResolve}

Troubleshooting logging and monitoring

{: #cs_troubleshoot_health}

As you use {{site.data.keyword.containerlong}}, consider these techniques for troubleshooting issues with logging and monitoring. {: shortdesc}

If you have a more general issue, try out cluster debugging. {: tip}

Logs do not appear

{: #cs_no_logs}

{: tsSymptoms} When you access the Kibana dashboard, your logs do not display.

{: tsResolve} Review the following reasons why your cluster logs are not appearing and the corresponding troubleshooting steps:

Why it's happening How to fix it
No logging configuration is set up. In order for logs to be sent, you must create a logging configuration. To do so, see Configuring cluster logging.
The cluster is not in a Normal state. To check the state of your cluster, see Debugging clusters.
The log storage quota has been hit. To increase your log storage limits, see the {{site.data.keyword.loganalysislong_notm}} documentation.
If you specified a space at cluster creation, the account owner does not have Manager, Developer, or Auditor permissions to that space. To change access permissions for the account owner:
  1. To find out who the account owner for the cluster is, run bx cs api-key-info <cluster_name_or_ID>.
  2. To grant that account owner Manager, Developer, or Auditor {{site.data.keyword.containershort_notm}} access permissions to the space, see Managing cluster access.
  3. To refresh the logging token after permissions have been changed, run bx cs logging-config-refresh <cluster_name_or_ID>.
You have an application logging config with a symlink in your app path.

In order for logs to be sent, you must use an absolute path in your logging configuration or the logs cannot be read. If your path is mounted to your worker node, it might have created a symlink.

Example: If the specified path is /usr/local/spark/work/app-0546/0/stderr but the logs actually go to /usr/local/spark-1.0-hadoop-1.2/work/app-0546/0/stderr, then the logs cannot be read.

To test changes you made during troubleshooting, you can deploy Noisy, a sample pod that produces several log events, onto a worker node in your cluster.

  1. Target your CLI to a cluster where you want to start producing logs.

  2. Create the deploy-noisy.yaml configuration file.

    apiVersion: v1
kind: Pod
metadata:
  name: noisy
spec:
  containers:
  - name: noisy
    image: ubuntu:16.04
    command: ["/bin/sh"]
    args: ["-c", "while true; do sleep 10; echo 'Hello world!'; done"]
    imagePullPolicy: "Always"

    {: codeblock}

  3. Run the configuration file in the cluster's context.

    kubectl apply -f <filepath_to_noisy>

    {:pre}

  4. After a few minutes, you can view your logs in the Kibana dashboard. To access the Kibana dashboard, go to one of the following URLs and select the {{site.data.keyword.Bluemix_notm}} account where you created the cluster. If you specified a space at cluster creation, go to that space instead.


Kubernetes dashboard does not display utilization graphs

{: #cs_dashboard_graphs}

{: tsSymptoms} When you access the Kubernetes dashboard, utilization graphs do not display.

{: tsCauses} Sometimes after a cluster update or worker node reboot the kube-dashboard pod does not update.

{: tsResolve} Delete the kube-dashboard pod to force a restart. The pod is re-created with RBAC policies to access heapster for utilization information.

kubectl delete pod -n kube-system $(kubectl get pod -n kube-system --selector=k8s-app=kubernetes-dashboard -o jsonpath='{.items..metadata.name}')

{: pre}


Getting help and support

{: #ts_getting_help}

Still having issues with your cluster? {: shortdesc}

  • To see whether {{site.data.keyword.Bluemix_notm}} is available, check the {{site.data.keyword.Bluemix_notm}} status page External link icon.

  • Post a question in the {{site.data.keyword.containershort_notm}} Slack. External link icon If you are not using an IBM ID for your {{site.data.keyword.Bluemix_notm}} account, request an invitation to this Slack. {: tip}

  • Review the forums to see whether other users ran into the same issue. When you use the forums to ask a question, tag your question so that it is seen by the {{site.data.keyword.Bluemix_notm}} development teams.

    • If you have technical questions about developing or deploying clusters or apps with {{site.data.keyword.containershort_notm}}, post your question on Stack Overflow External link icon and tag your question with ibm-cloud, kubernetes, and containers.
    • For questions about the service and getting started instructions, use the IBM developerWorks dW Answers External link icon forum. Include the ibm-cloud and containers tags. See Getting help for more details about using the forums.

  • Contact IBM Support by opening a ticket. For information about opening an IBM support ticket, or about support levels and ticket severities, see Contacting support.

{:tip} When reporting an issue, include your cluster ID. To get your cluster ID, run bx cs clusters.