Monitoring 3.9.x - Allow toggling tooltip selects

[fee-mendes]

System information

• Scylla version (you are using): 2021.1.7
• Are you willing to contribute it (Yes/No): Yes

Describe the feature and the current behavior/state.
It looks like that starting with Scylla Monitoring 3.9.x the tooltip selector defaults to single rather than all. This was the default set-up as of <3.8.x. It isn't clear whether this was an intended change or whether this was caused by the upgrade to Grafana 8.

The only workaround I know for now is to update all Grafana JSONs to specify multi rather than single within the tooltip selector, as the following one-liner:

find . -type f -exec sed -i 's/"mode": "single"/"mode": "multi"/g' {} \;

Is it possible to add a toggler? I believe that a single tooltip selector has its own added value, but we can't remove the multi selector altogether without having a way to switch back to it eventually.

Who will benefit with this feature?
Allowing to toggle between different tooltip selectors will help to support a variety of use cases. In case this is not possible, multi should be the default as:

• Users are already used to it
• Analyzing cross-node/cross-shard metrics may become problematic

cc @amnonh