Un-revert "Siem query rule - reduce field_caps usage"

[lukasolson]

Summary

#184890 was reverted in #186196 because it contained a bug with alerts created using Lucene queries. The bug was fixed in #186217. This PR un-reverts the original changes and preserves the fix. It also adds unit tests to cover the failed cases.

Checklist

• Unit or functional tests were updated or added to match the most common scenarios

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[mattkime]

Changes lgtm, the additional unit tests are a nice touch. I'm relying on @elastic/security-detection-engine to verify the functionality of their tests.

We should also add a functional test for a lucene query in discover but that's best done as a follow up.

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 0e9c318

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
data	511	512	+1

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
data	2585	2590	+5

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
data	420.0KB	420.2KB	+129.0B

Unknown metric groups

API count

id	before	after	diff
data	3194	3199	+5

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @lukasolson

[dej611]

nit: if the let usage is only due to the concat call at the end, maybe that can be replaced with push(...queryFields) and make this a const

Suggested change

	let fields = dataView.timeFieldName ? [dataView.timeFieldName] : [];
	if (sort) {
	const sortArr = Array.isArray(sort) ? sort : [sort];
	fields.push(...sortArr.flatMap((s) => Object.keys(s)));
	}
	for (const query of request.query) {
	if (query.query && query.language === 'kuery') {
	const nodes = fromKueryExpression(query.query);
	const queryFields = getKqlFieldNames(nodes);
	fields = fields.concat(queryFields);
	}
	}
	const fields = dataView.timeFieldName ? [dataView.timeFieldName] : [];
	if (sort) {
	const sortArr = Array.isArray(sort) ? sort : [sort];
	fields.push(...sortArr.flatMap((s) => Object.keys(s)));
	}
	for (const query of request.query) {
	if (query.query && query.language === 'kuery') {
	const nodes = fromKueryExpression(query.query);
	const queryFields = getKqlFieldNames(nodes);
	fields.push(...queryFields);
	}
	}

[lukasolson]

Although this nit makes sense, I prefer to leave this PR as close to its original counterpart (#184890) as possible

[dej611]

Left a minor code review, not a blocker for the PR.
Approve.

[dhurley14]

Tested locally with one of the original offending pre-built rules Cobalt Strike Command and Control Beacon and that generated an alert. This LGTM! Glad the fix was a simple one-liner.