Enable monitoring buffer for elastic-agent

[michel-laterman]

What does this PR do?

Enable the monitoring buffer for the elastic-agent and beats that it starts.
Add metrics collection (from buffers) into the diagnostics collect bundle if enabled.

Why is it important?

Allows us to gather metrics data for debugging if agent->ES communications are failing.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation PR
• I have made corresponding change to the default configuration files
• [] I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Start agent and run elastic-agent diagnostics collect. Metrics should not be included in the archive by default.
Enable metrics buffers by adding agent.monitoring.http.buffer.enabled: true then rerun elastic-agent diagnostics collect, metrics (json) files should be added to the archive

Related issues

• Relates Beats diagnostic tooling - Phase II elastic-agent#94

[mergify]

This pull request does not have a backport label. Could you fix it @michel-laterman? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-03-04T00:50:51.647+0000

• Duration: 16 min 2 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b agent-metrics-buffer upstream/agent-metrics-buffer
git merge upstream/main
git push upstream agent-metrics-buffer

[michel-laterman]

I've added the full buffer config here, but I'm not sure if we should. Currently the enabled flag is the only one that's passed (it's injected via command line args at the moment).

[ph]

So the other values would be noop, if this is the case we should not expose them.

[blakerouse]

I agree if its not exposed then it should be hidden.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[michel-laterman]

/package

[lykkin]

LGTM, just some questions for context:

beat metrics are collected in the diag command over grpc by way of an http endpoint. does that sound right?

are there any docs i can read up on the metrics libbeat exports this way?

[ph]

@michel-laterman Can we add a few tests to ensure the behavior?

[ph]

So the other values would be noop, if this is the case we should not expose them.

[blakerouse]

Looks good to me, minus the one comment.

[blakerouse]

I agree if its not exposed then it should be hidden.

[michel-laterman]

@ph, as we discussed I think testing for the diagnostics commands should be added to e2e tests, i've made an issue to track this here: elastic/e2e-testing#2204

[ph]

LGTM

[jlind23]

@michel-laterman will you be able to merge it before your time off?

[michel-laterman]

yep, as soon as the e2e passes i'll merge

[michel-laterman]

/test

[michel-laterman]

I'm going to force this through, all checks succeeded in c8339ad and I have only changed default config to remove unused values.