Faild to deploy tidb when running import script of importing grafana dashboards

[aeoluswing]

Please answer these questions before submitting your issue. Thanks!

What did you do?
If possible, provide a recipe for reproducing the error.
After preparing steps(include deploy),run ansible-playbook start.yml to start the tidb cluster.

What did you expect to see?
Start successfully.

What did you see instead?
Some error when importing grafana dashboards - run import script.This error can reproducing.

What version of TiDB are you using (tidb-server -V or run select tidb_version(); on TiDB)?
tidb version: v3.0.0 - v.3.0.9

This problem happened when i update tidb version from 2.x to 3.0.0,and upgrading could not resolve,by the way,2.x won't lead this problem before.

[liubo0127]

Looks like the Grafana service is abnormal, please checks the Grafana service. In addition, which version (tag) of tidb-ansible is used?

[aeoluswing]

tidb-ansible version 2.7.11

[liubo0127]

Please post the tidb-ansible version, not the ansible version. And please confirm that Grafana service is normal (you can check it through the webpage: http://grafana_host:3000).

[aeoluswing]

@liubo0127 sorry,tidb-ansible tag v3.0.9.In fact,i've tried multiple versions later than v3.0.0 and got the same result.
Grafana service was not available.ps command was used to look into the server which monitor was launched but no grafana process was found.

[liubo0127]

Please check the Grafana log (in the {{deploy_dir}}/log directory of the Grafana machine), there should be some error messages.

[aeoluswing]

@liubo0127 The log(named Grafana.log or Grafana.log.*) under {{deploy_dir}}/log shows some resources not found(http error code 404).

After redeploying the cluster(rerun start.yml),the same error which was mentioned before occurred.

I tried the webpage again but this time it was available.I found the grafana process and the service was bound at port 3000.

At the webpage there was a failure during getting data source which named test-cluster.

It seemed there was a failed initialization about the template.

[liubo0127]

The Grafana configuration has a Datasource (pointing to Prometheus), and monitoring is querying data from Prometheus. You can open the Datasource in the http://ip:3000/datasources page, and click Save & Test to check if it is connected.

If there is not Datasource in http://ip:3000/datasources, maybe some error occurred during import Datasource(one step in start.yml in tidb-ansible).

[aeoluswing]

@liubo0127 Indeed,there is no datasource in Datasource page.As you said,initialization process failed.

[liubo0127]

Please re-execute ansible-playbook start.yml and post the error message.

[aeoluswing]

@liubo0127 I have re-executed the start.yml but got the same error.

[load] from <performance_read.json>:performance_read
[import] <Tidb-Cluster-Performance-Read> to [Tidb-Cluster]    ............ ERROR: Unprocessable Entity

Unprocessable Entity indicates that the server understands the content type of the request entity, and the syntax of the request entity is correct, but it was unable to process the contained instructions.

Is there any update of performance_read.json?

[liubo0127]

I think of a possible reason, when upgrading to version 3.0, ansible-playbook rolling_update_monitor.yml may not be executed, which may cause the error.

Please execute ansible-playbook rolling_update_monitor.yml to upgrade the monitor.

[aeoluswing]

sorry for late reply.After rolling_update_monitor,i got the same error.Maybe this is not the error on this point,because either update or start performance_read.json will encounter unprocessable entity

[liubo0127]

If there is no Datasource named test-cluster in Grafana, add it manually and then execute rolling_update_monitor.yml.

[aeoluswing]

@liubo0127 As your said,this problem was resolved by editting datasource in grafana.The default prometheus datasource is named Prometheus and located at http://localhost:9090.
In the older version(earlier than version 3.0),the tidb-ansible scripts will extract the cluster_name and prometheus_server from inventory.ini and initialize grafana automatically.
This is wrong it seems that and bewildering.

After the modification,reexecute ansible-playbook rolling_update_monitor.yml --tags=grafana and finally,at the appearance of the all dashboards i guess it works within expectations.

This issue will be closed soon.I am truly appreciate your helping me to solved this problem.

[liubo0127]

This should not report an error, the Datasource will be automatically configured under normal circumstances. I will check the code again.