Licensed feature usage API

[kobelb]

Currently, we have no way to know when a licensed feature was actually used. Kibana routinely checks whether or not a license enables a specific feature for a number of reasons, but it doesn't differentiate between checking for the sake of hiding something in the UI versus a check prior to a feature actually being consumed.

The Elasticsearch team will be adding a similar API, so we should likely standardize on a format to make consumption easier.

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[pgayvallet]

@kobelb A few questions on that one:

• I think that license checking and licensed feature consumption / usages should be distinct APIs. Are we on the same page?

• Is this a need exclusive to the server side, or does this need a public counterpart

• Should this be exposed from core, from the licensing plugin or from a new xpack plugin?

• What is the retention period of these metrics? Is on memory storage enough, or do we need to persist that into a SO?

The Elasticsearch team will be adding a similar API, so we should likely standardize on a format to make consumption easier.

Is there any documents on that ? Who can I reach to get more info about this standardization?

[kobelb]

I think that license checking and licensed feature consumption / usages should be distinct APIs. Are we on the same page?

Yup!

Is this a need exclusive to the server side, or does this need a public counterpart

For the time being, it's just a server-side concern.

Should this be exposed from core, from the licensing plugin or from a new xpack plugin?

I was originally thinking that we'd add it to the existing licensing plugin, but I'm amenable if you think it makes more sense for it to be a completely separate plugin. License checking and licensed feature usage seem conceptually similar and somewhat dissimilar. I ultimately defer to your judgement, and the judgement of the other platform team members. I would like for us to consider how this decision impacts the likelihood of a developer forgetting to use the feature usage API correctly, and only consuming the license checking.

What is the retention period of these metrics? Is on memory storage enough, or do we need to persist that into a SO?

Storing this only in-memory gets us less retention, especially when processes are stopped/started. However, it's the only storage that we have at the moment that end-users can't tamper with.

Storing it as a saved-object, which is ultimately persisted to the .kibana_* indices, is resilient to process restarts. However, end-users will be able to tamper with this storage if they have write privileges to the .kibana_* indices, which superusers currently have.

@rjernst I believe that you're working on the Elasticsearch counterpart to this. Do you happen to have any details regarding how Elasticsearch is persisting its feature usage? Is this being stored in a persistent manner, or is it ephemeral in memory? Also, do you happen to have any examples of the Elastisearch feature usage API output?

[rjernst]

Storing this only in-memory gets us less retention
Is this being stored in a persistent manner, or is it ephemeral in memory?

Elasticsearch won't be persisting this; it will be up to cloud to capture this information at regular intervals. It is completely in memory, and node local, so cloud will capture this from every node and coalesce as necessary.

do you happen to have any examples of the Elastisearch feature usage API output?

Not yet. We first need some lower level changes in how licensed features are checked in ES, which is what I'm working on now. But at a conceptual level it is mapping an arbitrary key for a feature to the last time that feature was used. We may want to expand this in the future to have eg the last N uses (again, all kept in memory), but at its simplest it would be something like this:

{
  "security_fls" : "2020-04-03T17:23:00Z"
}

[kobelb]

Awesome, thanks @rjernst! If you wouldn't mind keeping us in the loop as you further flesh out the ES API, it'd be much appreciated.

[pgayvallet]

I was originally thinking that we'd add it to the existing licensing plugin, but I'm amenable if you think it makes more sense for it to be a completely separate plugin. License checking and licensed feature usage seem conceptually similar and somewhat dissimilar

Adding it to the licensing plugin makes sense to me I think. @joshdover @restrry WDYT, do you see any reason to add this API elsewhere?

@kobelb

For the time being, it's just a server-side concern.

Do you have any example of the kind of feature we'll want to monitor with this new API?

@rjernst

But at a conceptual level it is mapping an arbitrary key for a feature to the last time that feature was used
We may want to expand this in the future to have eg the last N uses

So atm the API output is basically a Map<FeatureId, LastUsedTimestamp>? we don't even want a structure returning the number of uses on the last period of a predefined duration?

[joshdover]

Adding it to the licensing plugin makes sense to me I think. @joshdover @restrry WDYT, do you see any reason to add this API elsewhere?

Nope, I was expecting this to be part of the licensing plugin as well.

Do you have any example of the kind of feature we'll want to monitor with this new API?

There's only a handful right now (I want to say 3?), but one example I know of is the "Service Map" feature in APM.

[kobelb]

Do you have any example of the kind of feature we'll want to monitor with this new API?

The ones I'm aware of:

1. APM Service Maps (which @joshdover already brought up)
2. Roles using sub-feature privileges
3. Alerting will have certain actions, and potentially alerts, that only work with a Gold+ license

[rjernst]

We may want to expand this in the future to have eg the last N uses

So atm the API output is basically a Map<FeatureId, LastUsedTimestamp>? we don't even want a structure returning the number of uses on the last period of a predefined duration?

We decided to keep it simple at first, where cloud would handle persistence of this information, and thus choose the frequency with which to check the state. While we may in the future add a "last N uses", I don't think we would want a predefined duration, as that could get memory intensive for a frequently used feature.