Improve usage stats collection - first- and third-party requests

[jportner]

#81907 and #85621 added usage stats collection for various APIs. We attempt to differentiate whether a consumer is "first-party" (the Kibana client) or "third-party" (some other integration) by checking for the presence of kbn-version and referer headers. Originally we also checked for origin too, but discovered that not all first-party requests contain this header.

As discussed in #85706 (comment):

• The vast majority of our docs instruct consumers to use kbn-xsrf, not kbn-version (which is more strict); however, our Reporting docs currently state to use kbn-version.
• Currently, we only collect usage data for Saved Objects APIs, Saved Objects Management APIs, and the Copy saved objects APIs.
• We could add a specific header to differentiate between first- and third-party requests, though this may present its own issues:
  • We really don't have any guarantees that other integrations won't attempt to use this header (though they gain nothing by doing so -- it would just mean that our usage data collection is slightly skewed). However, consumers of these experimental APIs might see Kibana using this header and wrongly assume that they need to use it too.
  • Adding a new header might present problems for users with reverse proxies that may strip headers not in an allow-list; this would cause false negative for first-party request detection

Additionally, we added a regex in Core to check for a space identifier in the base path (#85706 (comment)). We could consider changing that to an explicit check.

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[stacey-gammon]

Since plugins aren't supposed to access other plugin's HTTP APIs directly, what about exposing the API via two paths, one which uses internal, and having the internal plugin leverage the internal path?

[jportner]

Since plugins aren't supposed to access other plugin's HTTP APIs directly, what about exposing the API via two paths, one which uses internal, and having the internal plugin leverage the internal path?

Sure we could do that, it does sound like potentially a lot of work just to improve our usage data though.

FWIW, since I opened this issue, all of our docs have been updated to instruct consumers to use kbn-xsrf.

[pgayvallet]

it does sound like potentially a lot of work just to improve our usage data though.

++ On that. Note that it would also requires changes in all FTR api integration suites to run the suites against both endpoints

[mshustov]

Since plugins aren't supposed to access other plugin's HTTP APIs directly,

What problem do we want to solve? If we want to prevent pluginA from accessing pluginB endpoints, Core can extend http client-side service to include a plugin-specific header (ie, x-kbn-plugin-origin). With that, the server-side http service can reject a request originated from another plugin.
It won't break public API providing server-side http service doesn't reject incoming requests without this special header.

[stacey-gammon]

What problem do we want to solve?

If the plugin is calling these APIs from the server-side, it won't work properly (from my understanding it will fail but in a non-obvious way). If the plugin is attempting to access from the client-side, they lose out on the benefit of type checking. I have a PR out to document these guidelines here: https://github.com/elastic/kibana/pull/113313/files#diff-3c4e9b75014f3ada216e36cecccffe18d918455747274f9bd8a9d6121d24d793R35

it does sound like potentially a lot of work just to improve our usage data though.

Yea, I agree. I was thinking about the APIs that we want to make internal anyway, and deprecate/remove the public versions. In that case, I think it makes sense to maintain both for a period of time to support a slow migration.

[pgayvallet]

I don't think we may want this anymore, closing