Use Heatmap panel to represent Tablets status

[tzach]

Real time status of tablets, similar to tablet-mon.py but as part of a dashboard.

[tzach]

@tgrabiec FYI

[amnonh]

@tzach beside being a cool representation, can you describe a scenario that this is useful?

[tzach]

It would be beneficial to see, for a specific table, how it is distributed between the nodes.
See tablet-mon.py an in-terminal tool already in use.

[tgrabiec]

Sending detailed tablet metrics may be too much. We disable per-table stats by default, and per-tablet stats will be heavier than that.

We can present aggregate metrics for migrating tablets per shard.

[amnonh]

I was thinking, if we can show the number of tablets per shard/node we can show bars that would show the distribution.
Naturally, showing all tablets in a real cluster is too much

[tzach]

Sending detailed tablet metrics may be too much. We disable per-table stats by default, and per-tablet stats will be heavier than that.

Even if too much for production, it would be helpful for debuging and demos.

[tgrabiec]

@tzach So what would you like the heat map to show exactly?

[tgrabiec]

I feel that the thing provided by grafana is too limited in its presentation and update frequency. What tablet-mon is generally useful for is getting an intuition about the behavior of the system in real time, by observing movement of parts, and for that you need (1) high frequency of updates (2) animation. Grafana has neither, it's good at presenting coarse-grained statistics.

[tgrabiec]

For example, with tablet-mon I was able to immediately identify that there are stalls in tablet phase transitions which shouldn't be there. That would be lost in the noise in grafana. I think the two tools serve different purposes.

[amnonh]

closed by #2170