[Dashboards++] Usage counters enhancements

[gsoldevila]

Introduction

The Dashboards++ initiative proposes using usage-counters as a means to easily add telemetry to monitor dashboard utilization. In the same lines, this document proposes further enhancements to the usage counters logic, which will benefit not only this particular use case, but usage-counters in general.

Proposed changes

Dedicated index

The first thing we propose is to use a dedicated index to store all the usage-counters saved objects. Seeing that usage-counters are going to be used more broadly, having a dedicated index to store them will help:

• Reducing doc count and potential impact in the other system indices.
• Isolating docs of this type easily.
• Reducing risks / costs for migrations.
• Adding more capacity for SO storage (at the moment each index has 1 single shard).

The .kibana split, introduced in 8.8.0, started breaking down the .kibana index in multiple SO indices.
This creates an opportunity to introduce new indices for certain SO types. Thus, we propose creating a .kibana_usage_counters index, and storing all usage-counters objects there.

There’s a small challenge though: we cannot relocate existing documents to that index, as it would require a reindex migration, and that is currently not possible due to serverless constraints (no downtime).

Luckily, we have a couple of workarounds for that:
We can deprecate usage-counters SO type and use an alternative name, e.g. usage-counter. The drawback is that we might lose half to 1 day worth of counted events during the upgrade.
This seems like an acceptable trade off (see related thread).

Namespace type: single

At the moment, usage-counters have a namespace type agnostic. In practice, this means that we are not storing any value for the namespaces property. Whilst this might be convenient for some general use cases, Dashboards++ requires per-namespace counting, e.g. how many visualizations did dashboard X have on workspace Y on the last 30 days?

Now, this could be achieved by storing the namespace as one of the 2 different properties that shape the counters’ ids: counterType, counterName. But that means:

• We lose part of the flexibility for counter definitions.
• We’re not taking advantage of the namespaces property that already exists for SOs.
• The namespace management is done by the caller.
• We can’t benefit from the RBAC policies, which could prevent users from accessing other Spaces’ usage counters.

By changing the namespace type to single, we can leverage the namespace property and allow API users doing per-Space counting. Doing this change would usually require a reindexing migration, but switching to a dedicated index gives us an opportunity to make that change without reindexing.

Counters that are currently workspace agnostic can be stored in the default namespace. Internally, the Saved Objects Repository omits the namespace: string property of the SO when using the default namespace. For usage counters that are Space-aware, the API can provide a means to allow storing the desired namespace.

Finally, in order to keep iso-functional, for Space-aware counters the kibana_usage_collection plugin will aggregate countings for each namespace.
Thus, from the Stack Telemetry standpoint, counters will be global.

Tasks

• Moving usage-counters to a dedicated index
• Exposing an API to allow searching for persisted counters
• Dashboard views POC

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[rayafratkina]

This work is done - closing the issue

[willem-dhaese]

@rayafratkina Is there a way to view dashboard usage with these counters in gui?