Implement heartbeat metrics #183

roidelapluie · 2017-02-20T13:57:44Z

pt-heartbeat measures actual replication lab on a mysql server. It does
not rely on data stored in SHOW SLAVE STATUS, as it can be unreliable.

pt-heartbeat website:
https://www.percona.com/doc/percona-toolkit/2.2/pt-heartbeat.html

Signed-off-by: Julien Pivotto roidelapluie@inuits.eu

brian-brazil · 2017-02-20T14:03:05Z

How many different heartbeat implementations are there out there?

roidelapluie · 2017-02-20T14:05:29Z

pt-heartbeat has been there for a very long time; it is widely used.

grypyrg · 2017-02-20T18:14:17Z

@brian-brazil : pt-heartbeat is the most widely used (if not only) alternative tool to measure replication lag properly.
(Of course, there are many companies that do heartbeats using a custom built solution)

Given these 2 new metrics, can we easily get the time difference with them?
Maybe we should also calculate in mysqld_exporter itself already how much the difference is and write it to a different metric to make it easier to report the replication lag.

SuperQ · 2017-02-20T18:18:42Z

Could be interesting.

In what way is SHOW SLAVE STATUS unreliable?

grypyrg · 2017-02-20T18:29:10Z

SHOW SLAVE STATUS is unreliable when:

parallel replication is used (slave_parallel_workers>0). several metrics are only updated in regular intervals (https://dev.mysql.com/doc/refman/5.7/en/replication-options-slave.html#option_mysqld_slave-checkpoint-group & https://dev.mysql.com/doc/refman/5.7/en/replication-options-slave.html#option_mysqld_slave-checkpoint-period
when IO thread is not catching up with the master (due to limited bandwith), and the sql thread has caught up with the IO thread, seconds_behind_master will reported as 0
in multi-tiered architectures, replication lag is always the timestamp from where the query was originally executed. with pt-heartbeat you can run the heartbeat on the intermediate server and get replication lag for that tier only
when timezones are different between servers you can have for example +3600 or -3600 report seconds replication lag.
https://bugs.mysql.com/bug.php?id=80084
...

roidelapluie · 2017-02-21T05:36:24Z

Fixed:

added _seconds suffix as recommended in Prometheus documentation.
added the actual lag time
rounded to integers. Makes no sense to go below the second as pt-heartbeat only updates the table once per second.

SuperQ · 2017-02-21T08:03:53Z

@grypyrg Awesome, thanks for documenting those issues here.

SuperQ · 2017-02-21T08:13:10Z

Even with second-only precision hearbeats, it's still worth keeping the float data as is valid. Especially if pt-heartbeat attempts to write on-the-second and not just sleep(1)

Maybe for compatibility, we should implement a pt-hearbeat compatible writer and include it in the mysqld_exporter repo. We could also implement it to write out millisecond precision timestamps.

It's really too bad that pt-heartbeat uses a varchar for the timestamp instead of a datetime(6).

SuperQ · 2017-02-21T08:24:01Z

collector/pt_heartbeat.go

+	ptHeartbeat = "pt_heartbeat"
+	// pt-heartbeat query
+	// the second column allows gets the server timestamp at the exact same time the query is run
+	ptHeartbeatQuery = "SELECT UNIX_TIMESTAMP(ts), UNIX_TIMESTAMP(CONCAT(DATE(NOW()), ' ', CURTIME(6))), server_id from %s"


This can be simplified with UNIX_TIMESTAMP(NOW(6)). I tested on 5.5 and 5.6.

roidelapluie · 2017-02-21T08:26:02Z

@SuperQ pt-heartbeat'i very initial version dates from https://sourceforge.net/p/maatkit/code/857/ 2007-09-05. The latest version is the result of almost 10 years of maturing. Older releases were using timestamps, and that got changed to VARCHAR later. I could not track back the reason.

I do agree that a Go version of that tool might be handy, but is out of scope of this pull request.

SuperQ · 2017-02-21T08:27:12Z

collector/pt_heartbeat.go

+		"Timestamp of the current server.",
+		[]string{"server_id"}, nil,
+	)
+	PtHeartbeatDelayDesc = prometheus.NewDesc(


This metric isn't necessary (and against Prometheus design) since you can do this in PromQL.

@grypyrg That delay can be made at the prometheus level using https://prometheus.io/docs/querying/rules/ . Once this patch gets in, maybe we can document an example recording rule for this on a blogpost.

SuperQ · 2017-02-21T08:27:55Z

README.md

@@ -66,6 +66,8 @@ collect.perf_schema.indexiowaits                       | 5.6           | Collect
 collect.perf_schema.tableiowaits                       | 5.6           | Collect metrics from performance_schema.table_io_waits_summary_by_table.
 collect.perf_schema.tablelocks                         | 5.6           | Collect metrics from performance_schema.table_lock_waits_summary_by_table.
 collect.slave_status                                   | 5.1           | Collect from SHOW SLAVE STATUS (Enabled by default)
+collect.pt-heartbeat                                   | 5.1           | Collect from pt-heartbeat


I would prefer to call this heartbeat, as there could be other implementations.

renamed everywhere

SuperQ · 2017-02-21T08:28:54Z

Overall, I like this idea, some comments in-line.

roidelapluie · 2017-02-21T08:32:41Z

Should I rename ptHeartbeat to heartbeat everywhere or just in the CLI flags?

roidelapluie · 2017-02-21T08:43:52Z

Updated after latest comments.

@SuperQ do you want more documentation for this in README?

SuperQ · 2017-02-21T08:44:13Z

@roidelapluie Totally agree, re-implementing is out of scope for this feature. I was mostly just thinking out loud.

Yes, please remove pt from everything. But add a note and a link to pt-heartbeat in the README that "this is a supported implementation of heartbeat checking".

roidelapluie · 2017-02-21T09:01:22Z

collector/heartbeat.go

+	// timestamps. %s will be replaced by the table name.
+	// The second column allows gets the server timestamp at the exact same
+	// time the query is run.
+	heartbeatQuery = "SELECT UNIX_TIMESTAMP(ts), UNIX_TIMESTAMP(NOW(6)), server_id from `%s`"


@SuperQ how can we prevent this to turn into sql injection?

Apparently not much golang/go#18478

split in two: database and table, for better escaping

grypyrg · 2017-02-21T09:03:39Z

@roidelapluie, @SuperQ : pt-heartbeat does support subsecond checking. See https://www.percona.com/doc/percona-toolkit/2.1/pt-heartbeat.html#cmdoption-pt-heartbeat--interval

The minimum (fastest) interval is 0.01, and the maximum precision is two decimal places, 
so 0.015 will be rounded to 0.02.

grypyrg · 2017-02-21T09:11:53Z

And maybe fetch another column to precalculate the seconds_lag

SELECT UNIX_TIMESTAMP(ts) AS executed_time,
       UNIX_TIMESTAMP(NOW(6)) AS server_time,
       UNIX_TIMESTAMP(NOW(6)) - UNIX_TIMESTAMP(ts) AS seconds_lag,
       server_id
FROM `%s`

Example output (to also show it's subsecond):

+-------------------+-------------------+-------------+-----------+
| executed_time     | server_time       | seconds_lag | server_id |
+-------------------+-------------------+-------------+-----------+
| 1487668183.001080 | 1487668278.442121 |   95.441041 |         1 |
+-------------------+-------------------+-------------+-----------+

roidelapluie · 2017-02-21T09:23:43Z

Updated with all the comments taken in consideration; also split database and table parameters so we can better escape them.

SuperQ · 2017-02-21T09:49:44Z

Maybe make a parameter for the column?

roidelapluie · 2017-02-21T09:52:21Z

Two parameters then. Is that complexity needed for an initial release?

SuperQ · 2017-02-21T10:50:42Z

No, not necssary, it was just an idea. I think it's fine to keep the columns hard-coded.

SuperQ · 2017-02-21T10:57:03Z

collector/heartbeat.go

+	}
+	defer heartbeatRows.Close()
+
+	var now sql.RawBytes


This should be:

var ( now, ts sql.RawBytes serverId int )

SuperQ

LGTM besides one Go nit.

roidelapluie · 2017-02-21T11:06:36Z

Go nit fixed.

pt-heartbeat measures actual replication lab on a mysql server. It does not rely on data stored in SHOW SLAVE STATUS, as it can be unreliable. pt-heartbeat website: https://www.percona.com/doc/percona-toolkit/2.2/pt-heartbeat.html Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

roidelapluie · 2017-02-21T11:24:35Z

Wording fixed in README.md and mysqld_exporter.go, so for me it is good to merge.

SuperQ · 2017-02-21T11:29:56Z

Yup, looks fine, thanks.

grypyrg · 2017-02-21T11:43:22Z

Thanks @roidelapluie !

roidelapluie · 2017-02-21T12:38:20Z

For reference, promsql query:

mysql_heartbeat_now_timestamp_seconds - on (server_id) mysql_heartbeat_stored_timestamp_seconds

SuperQ · 2017-02-21T12:49:10Z

Normally Prometheus will be able to match up the labels and you don't need to specify the on (server_id).

roidelapluie · 2017-02-21T12:52:38Z

Oh thanks. Nice trick.

Heartbeat[0] metrics allow for similar alerting symantics to "seconds behind master" metrics. * Add a recording rule to provide a similar gauge of seconds behind. * Add an alert with the same style. [0]: #183

* [FEATURE] Add read/write query response time #166 * [FEATURE] Add Galera gcache size metric #169 * [FEATURE] Add MariaDB multi source replication support #178 * [FEATURE] Implement heartbeat metrics #183 * [FEATURE] Add basic file_summary_by_instance metrics #189 * [BUGFIX] Workaround MySQL bug 79533 #173

roidelapluie changed the title ~~Implement pt-heartebat metrics~~ Implement pt-heartbeat metrics Feb 20, 2017

roidelapluie force-pushed the pth branch from f3a4605 to cc99c15 Compare February 20, 2017 13:58

SuperQ closed this Feb 20, 2017

SuperQ reopened this Feb 20, 2017

roidelapluie force-pushed the pth branch 3 times, most recently from 4bd2443 to 956bd82 Compare February 20, 2017 21:30

SuperQ reviewed Feb 21, 2017

View reviewed changes

roidelapluie force-pushed the pth branch from 956bd82 to 7c0937a Compare February 21, 2017 08:40

roidelapluie force-pushed the pth branch 2 times, most recently from c5685e4 to dab4a26 Compare February 21, 2017 08:58

roidelapluie commented Feb 21, 2017

View reviewed changes

roidelapluie force-pushed the pth branch from dab4a26 to 32f5999 Compare February 21, 2017 09:01

roidelapluie force-pushed the pth branch from 32f5999 to ec17330 Compare February 21, 2017 09:22

roidelapluie changed the title ~~Implement pt-heartbeat metrics~~ Implement heartbeat metrics Feb 21, 2017

SuperQ reviewed Feb 21, 2017

View reviewed changes

SuperQ approved these changes Feb 21, 2017

View reviewed changes

roidelapluie force-pushed the pth branch from ec17330 to 2847b6e Compare February 21, 2017 11:06

roidelapluie force-pushed the pth branch from 2847b6e to b0a2f39 Compare February 21, 2017 11:23

SuperQ merged commit 5bf756d into prometheus:master Feb 21, 2017

SuperQ mentioned this pull request Feb 21, 2017

Add an example of heartbeat metrics for slave lag #184

Merged

grypyrg mentioned this pull request Feb 25, 2017

Implement heartbeat graph in Replication dashboard percona/grafana-dashboards#14

Closed

SuperQ mentioned this pull request Mar 22, 2017

Release v0.10.0 #191

Merged

arvenil mentioned this pull request Jun 13, 2017

PMM-1011: merge upstream changes percona/mysqld_exporter#2

Merged

koorgoo mentioned this pull request Jun 20, 2020

Add heartbeat lag metric #479

Closed

Implement heartbeat metrics #183

Implement heartbeat metrics #183

Conversation

roidelapluie commented Feb 20, 2017

brian-brazil commented Feb 20, 2017

roidelapluie commented Feb 20, 2017

grypyrg commented Feb 20, 2017

SuperQ commented Feb 20, 2017 • edited Loading

grypyrg commented Feb 20, 2017

roidelapluie commented Feb 21, 2017

SuperQ commented Feb 21, 2017

SuperQ commented Feb 21, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

roidelapluie commented Feb 21, 2017

Choose a reason for hiding this comment

roidelapluie Feb 21, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SuperQ commented Feb 21, 2017

roidelapluie commented Feb 21, 2017

roidelapluie commented Feb 21, 2017 • edited Loading

SuperQ commented Feb 21, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grypyrg commented Feb 21, 2017 • edited Loading

grypyrg commented Feb 21, 2017

roidelapluie commented Feb 21, 2017 • edited Loading

SuperQ commented Feb 21, 2017

roidelapluie commented Feb 21, 2017

SuperQ commented Feb 21, 2017

Choose a reason for hiding this comment

SuperQ left a comment

Choose a reason for hiding this comment

roidelapluie commented Feb 21, 2017

roidelapluie commented Feb 21, 2017

SuperQ commented Feb 21, 2017

grypyrg commented Feb 21, 2017

roidelapluie commented Feb 21, 2017 • edited Loading

SuperQ commented Feb 21, 2017

roidelapluie commented Feb 21, 2017

SuperQ commented Feb 20, 2017 •

edited

Loading

roidelapluie Feb 21, 2017 •

edited

Loading

roidelapluie commented Feb 21, 2017 •

edited

Loading

grypyrg commented Feb 21, 2017 •

edited

Loading

roidelapluie commented Feb 21, 2017 •

edited

Loading

roidelapluie commented Feb 21, 2017 •

edited

Loading