[Fleet] Enable agent.monitoring.http settings on agent policy UI

[juliaElastic]

Summary

Related to https://github.com/elastic/ingest-dev/issues/2471

With elastic/elastic-agent#4499 merged, it became possible to reload monitoring settings changes in a running agent, so enabling these settings on the UI.

To verify:

• Create an agent policy and enroll an agent with latest 8.15-SNAPSHOT
• edit agent policy, and change monitoring settings, e.g. change port
• verify that the metrics endpoint is running on the new port

@cmacknz Does it make sense to allow changing the host name? It seems to me that monitoring can only work in localhost.
Another question, how can we verify that the buffer.enabled setting is applied correctly?

15:07:40.054
elastic_agent
[elastic_agent][error] Failed creating a server: failed to create api server: listen tcp 192.168.178.217:6791: bind: cannot assign requested address

Also, I'm not sure if switching off the enabled flag did anything, seeing this again in the logs:

15:13:15.167
elastic_agent
[elastic_agent][info] Starting server
15:13:15.168
elastic_agent
[elastic_agent][info] Starting stats endpoint
15:13:15.168
elastic_agent
[elastic_agent][debug] Server started
15:13:15.168
elastic_agent
[elastic_agent][info] Metrics endpoint listening on: 127.0.0.1:6791 (configured: http://localhost:6791)

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[apmmachine]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
• /oblt-deploy-serverless : Deploy a serverless Kibana instance using the Observability test environments.
• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[juliaElastic]

Moving this back to draft as the reload changes are being reverted: elastic/elastic-agent#4583

[juliaElastic]

Should be ready to enable again: elastic/elastic-agent#4586

[juliaElastic]

/ci

[nchaulet]

@cmacknz Does it make sense to allow changing the host name? It seems to me that monitoring can only work in localhost.

I think it's the host we want to listen to, so. I guess it make sense to make this configurable if we want to listen on 0.0.0.0 for example. Should it default to null and not localhost?

[jen-huang]

Can we make the display name and description better? The current values don't convey much meaning, a user would need to go to the docs to understand what this setting does.

My suggestion:

HTTP monitoring endpoint

Enables a liveness HTTP endpoint that returns the overall health of Elastic Agent. Can be used by Kubernetes to restart the container, for example.

@kilfoyle Can you suggest something better? More background is in elastic/elastic-agent#390. Also, I think we need a section for this setting on this page or expand the existing Override the default monitoring port section, not sure if there's a docs issue yet.

[kilfoyle]

I like the text @jen-huang suggested. Just one small tweak so the second sentence is complete:

HTTP monitoring endpoint

Enables a liveness HTTP endpoint that returns the overall health of Elastic Agent. This can be used by Kubernetes to restart the container, for example.

Regarding the docs, @juliaElastic would you mind opening an issue for those? I think this Enable custom settings in an agent policy page might actually not be the best place for the changes, since those are all things you can do through the API because the setting isn't available in Fleet. Instead, I think we'd probably want a brief mention on the Monitoring Elastic Agents and a full description somewhere in the Fleet settings pages, but let me know if that doesn't seem right.

[cmacknz]

I think it's the host we want to listen to, so. I guess it make sense to make this configurable if we want to listen on 0.0.0.0 for example. Should it default to null and not localhost?

You want to default to localhost so that the server is not exposed off of the current host. 0.0.0.0 could for example expose the server on an internet facing network. If you set it to null you are just deferring to the agent's default, which should also be localhost for the same reason. Better to explicitly list what the default is.

[juliaElastic]

@kilfoyle Thanks for the suggestion, created an issue: elastic/ingest-docs#1073

[juliaElastic]

Updated the title and description.
Removed enabled and buffer.enabled from the config, as they don't seem to do anything.
@cmacknz Are these settings supposed to work?

[cmacknz]

@cmacknz Are these settings supposed to work?

In 8.15.0 they should work from Fleet, we have an integration test that confirms it is reloadable using the overrides API https://github.com/elastic/elastic-agent/blob/fdba118cf3cb5ccff1dc6c9f1b747c4d1471248e/testing/integration/monitoring_probe_reload_test.go#L108-L124

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 391b5aa

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
fleet	1.3MB	1.3MB	-10.0B

History

• 💚 Build #209706 succeeded 022d8b6
• 💔 Build #209456 failed d84eb53
• 💔 Build #209391 failed a1a314b
• 💔 Build #209172 failed 1f09d00
• 💛 Build #204042 was flaky 23a6d35

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @juliaElastic

[juliaElastic]

Tested enabled and buffer.enabled again with agent version 8.15-SNAPSHOT, but doesn't seem to affect anything (at least not logged out). I think I'll leave as is for now.

[elastic_agent][info] creating monitoring API with cfg api.Config{Enabled:true, Host:"http://0.0.0.0:6792", Port:6792, User:"", SecurityDescriptor:"", Timeout:5000000000}

In the agent diagnostics I'm seeing the changed config in computed-config.yaml
elastic-agent-diagnostics-2024-05-14T11-49-24Z-00.zip

    monitoring:
        enabled: true
        http:
            buffer:
                enabled: true
            enabled: false
            host: 0.0.0.0
            port: 6792

[rgarcia89]

@juliaElastic @cmacknz I am not seeing this monitoring enable switcher in my kibana v8.15.0.
Am I missing something?

[juliaElastic]

@rgarcia89 Those settings are not enabled yet, as I couldn't successfully test it. We can create an issue to enable and test again.

[rgarcia89]

Understood so they will come in the next version? Otherwise how can I enable the liveness probe for an elastic-agent managed daemonset now? I am currently using https://github.com/elastic/elastic-agent/blob/main/deploy/kubernetes/elastic-agent-managed/elastic-agent-managed-daemonset.yaml

Thanks

[juliaElastic]

@cmacknz Could you help with this? I'm not familiar with elastic-agent daemonsets.