[Fleet] removed restriction to use remote es as integration data

[juliaElastic]

Summary

Closes #173237

Removed restriction to allow using remote es output as integration data output.

Steps to verify:

Send system integration data to remote es

• Create a remote es output, verify that the output is allowed to be set as default for agent integrations
• Create an agent policy with system integration and set the remote es output as integration data output
• Enroll an agent to the agent policy
• Check the remote kibana - Discover, verify that system metrics are coming in from the agent
• Install system package on the remote cluster to see dashboards, mappings, etc.

Send nginx integration data to remote es

• Add nginx integration to the agent policy
• Create a dummy nginx log file in /var/tmp/nginx/access.log and add some dummy data to it
• Verify that the data from the nginx log file appears in the remote kibana Discover in logs-* data view.

Back to default output

• Change the agent policy integration output back to default
• Verify that the system integration data is ingested in the main cluster.
• Verify that the API key is invalidated in the remote cluster

System dashboard on remote cluster populated:

Checklist

• Unit or functional tests were updated or added to match the most common scenarios

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[apmmachine]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
• /oblt-deploy-serverless : Deploy a serverless Kibana instance using the Observability test environments.
• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[nchaulet]

@juliaElastic should we prevent remote ES to be used with fleet-server (and maybe synthetics too) as we do for kafka and logasth output?

[juliaElastic]

@juliaElastic should we prevent remote ES to be used with fleet-server (and maybe synthetics too) as we do for kafka and logasth output?

Actually I just ran into this limitation when I added APM to the agent policy: #149873
I suppose there is no reason to restrict APM, right? only fleet-server and synthetics

[nchaulet]

I suppose there is no reason to restrict APM, right? only fleet-server and synthetics

Yes there is probably no reason, but it may be worth checking with the APM team same for endpoint not sure he really make sense to allow endpoint to send data to an external ES

[juliaElastic]

I suppose there is no reason to restrict APM, right? only fleet-server and synthetics

Yes there is probably no reason, but it may be worth checking with the APM team same for endpoint not sure he really make sense to allow endpoint to send data to an external ES

Asked both teams on slack to confirm.
Also raised an es pr to add missing privileges to write to traces-* and .logs-endpoint data streams with fleet-server-remote service account: https://github.com/elastic/elasticsearch/pull/103445/files
We can change/revert this if not needed.

[juliaElastic]

It seems that endpoint doesn't send any integration data by agents, so it is not relevant to remote es output. I think it would be a no-op if an Agent policy with remote es has endpoint integration. So it doesn't seem needed to restrict using remote es with an agent policy where endpoint integration is added.
The logs-endpoint indices are not used by agent either, I'll remove those from the es pr.

[juliaElastic]

I'm trying to test with APM data being sent to remote es, having trouble to generate apm traces locally to pick up by agent. Any ideas how to do that locally? Tried to follow this guide: https://www.elastic.co/guide/en/observability/master/traces-get-started.html

[nchaulet]

I'm trying to test with APM data being sent to remote es, having trouble to generate apm traces locally to pick up by agent. Any ideas how to do that locally? Tried to follow this guide: https://www.elastic.co/guide/en/observability/master/traces-get-started.html

I think one easy way could be to configure kibana or fleet server to use your local APM server

[juliaElastic]

I managed to test APM data being sent to remote es, though seeing some errors as APM team confirmed:

Steps:

1. create agent policy with remote es output as integration data output
2. enroll an agent
3. Add APM integration to agent policy
4. Add to fleet-server integration policy advanced config

server.instrumentation.enabled: true
server.instrumentation.hosts: ["http://localhost:8200"]

5. Create data view in remote es output with *apm* index pattern
6. Seeing data coming in, with errors around sync sourcemaps metadata

As discussed with Nima and APM team, there are some features in APM that would have to be worked on to support remote ES output, until then we are going to disable using remote output on a policy with APM integration.

some of the APM Server features piggyback on the ES output config: for synchronising source maps and agent config to APM Server, and for reporting agent config usage metrics to display in Kibana.

[juliaElastic]

Added back the check to disallow remote es as data output if agent policy has APM integration:

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 7e7585b

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
fleet	1.2MB	1.2MB	-58.0B

History

• 💚 Build #183852 succeeded bbcfe60
• 💚 Build #183673 succeeded 457df3d
• 💛 Build #183633 was flaky 6178650
• 💛 Build #183564 was flaky efa5a6a

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @juliaElastic

[nchaulet]

LGTM 🚀

[kibanamachine]

💔 All backports failed

Status	Branch	Result
❌	8.12	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 173353

Questions ?

Please refer to the Backport tool documentation