[WIP][APM] APM client

[cauemarcondes]

closes #67397

[kibanamachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request
• Commit: ec886b5
• Pipeline Steps (look for red circles / failed steps)
• Interpreting CI Failures

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[sorenlouv]

I think you need to use RequestHandlerContext like we do in

kibana/x-pack/plugins/apm/server/lib/helpers/es_client.ts

Lines 87 to 116 in fa8187b

	async function getParamsForSearchRequest(
	context: APMRequestHandlerContext,
	params: ESSearchRequest,
	apmOptions?: APMOptions
	) {
	const { uiSettings } = context.core;
	const [indices, includeFrozen] = await Promise.all([
	getApmIndices({
	savedObjectsClient: context.core.savedObjects.client,
	config: context.config,
	}),
	uiSettings.client.get('search:includeFrozen'),
	]);
	// Get indices for legacy data filter (only those which apply)
	const apmIndices = Object.values(
	pickKeys(
	indices,
	'apm_oss.sourcemapIndices',
	'apm_oss.errorIndices',
	'apm_oss.onboardingIndices',
	'apm_oss.spanIndices',
	'apm_oss.transactionIndices',
	'apm_oss.metricsIndices'
	)
	);
	return {
	...addFilterForLegacyData(apmIndices, params, apmOptions), // filter out pre-7.0 data
	ignore_throttled: !includeFrozen, // whether to query frozen indices or not
	};

.

I think for querying public apm indices (not system indices) the current user should be used. For querying internal indices like .apm-agent-configuration or .apm-custom-links the internal user should be used.
Thinking more about this, it should probably be separate clients.

So the following client should always use the current user, and should not be used for querying system indices:

apmClient.search({
  apmIndices: ['transaction', 'span'],
  size: 100,
  query: {...},
  aggs: {}
})

The following client should always use the internal user, but only be used to querying system indices

apmInternalClient.search({
  apmIndices: ['.apm-agent-configuration', '.apm-custom-links'],
  size: 100,
  query: {...},
  aggs: {}
})

[sorenlouv]

The "normal" apmClient should filter out legacy data addFilterForLegacyData and handle frozen indices. apmInternalClient should not.

[sorenlouv]

Currently we have one client that all requests are made with. I think we should go away from that thinking and instead have multiple smaller clients, eg apmClient, apmInternalClient, mlClient (until they ship their own) etc.

[sorenlouv]

It would be great if you can update es_client to reflect this change.

[sorenlouv]

I think you can simplify apmClient a lot since it should only have a search method (no delete, index etc).